Max level shown:
1 starting March 17, 2025 at 10:34:22 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2758073
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.022526944
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.02346676
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Starting downstairs at port 8830
20 Make output file at "/tmp/dsc/downstairs-8820.txt"
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1198)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1199)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1200)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Mar 17 22:34:23.075 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Mar 17 22:34:28.142 INFO accepted connection, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
36 Mar 17 22:34:28.147 INFO request completed, latency_us: 1550, response_code: 200, uri: /regioninfo, method: GET, req_id: ce9ff52f-ea5b-44d8-86e2-b96d8cd7e4de, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
37 Mar 17 22:34:28.149 INFO request completed, latency_us: 95, response_code: 200, uri: /regioncount, method: GET, req_id: e68e715d-7b7c-420c-a106-c23c471a822c, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
382025-03-17T22:34:28.149ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-03-17T22:34:28.149ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Mar 17 22:34:28.150 INFO request completed, latency_us: 133, response_code: 200, uri: /port/cid/0, method: GET, req_id: d8284d97-d9e9-4a52-acdf-45b16af4a1f2, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
41 Mar 17 22:34:28.151 INFO request completed, latency_us: 97, response_code: 200, uri: /port/cid/1, method: GET, req_id: eb9487ac-397d-4f34-ab82-505862da0fd6, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
42 Mar 17 22:34:28.151 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/2, method: GET, req_id: 98c83c01-f177-4fbf-8976-aa8830f78dae, remote_addr: 127.0.0.1:63933, local_addr: 127.0.0.1:9998
432025-03-17T22:34:28.151ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-03-17T22:34:28.152ZINFOcrucible: Upstairs starts
452025-03-17T22:34:28.152ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "45feeae3dd1bdcd88dce11a80acb5415f57b38bc", git_commit_timestamp: "2025-03-17T22:11:58.000000000Z", git_branch: "renovate/opentelemetry-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-03-17T22:34:28.152ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-03-17T22:34:28.152ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-03-17T22:34:28.153ZINFOcrucible: Crucible 5859d175-50aa-427e-8003-fde1244e3a4b has session id: 10cfdc24-8436-4bf0-96a2-740034d5dcbc session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
492025-03-17T22:34:28.153ZINFOcrucible: Upstairs opts: Upstairs UUID: 5859d175-50aa-427e-8003-fde1244e3a4b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
502025-03-17T22:34:28.153ZINFOcrucible: Crucible stats registered with UUID: 5859d175-50aa-427e-8003-fde1244e3a4b session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
512025-03-17T22:34:28.153ZWARNcrucible: could not find Downstairs address for Nexus session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
522025-03-17T22:34:28.155ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-03-17T22:34:28.155ZINFOcrucible: The guest has requested activation with gen:1
542025-03-17T22:34:28.155ZINFOcrucible: 5859d175-50aa-427e-8003-fde1244e3a4b active request set session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
552025-03-17T22:34:28.155ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
562025-03-17T22:34:28.155ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
572025-03-17T22:34:28.155ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
582025-03-17T22:34:28.155ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
592025-03-17T22:34:28.155ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
602025-03-17T22:34:28.155ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
612025-03-17T22:34:28.155ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
622025-03-17T22:34:28.155ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
632025-03-17T22:34:28.155ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
642025-03-17T22:34:28.155ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54487) = io task client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
652025-03-17T22:34:28.155ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49063) = io task client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
662025-03-17T22:34:28.155ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64697) = io task client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
672025-03-17T22:34:28.157ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
682025-03-17T22:34:28.157ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
692025-03-17T22:34:28.157ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
702025-03-17T22:34:28.158ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
712025-03-17T22:34:28.158ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
722025-03-17T22:34:28.158ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
732025-03-17T22:34:28.159ZINFOcrucible: Waiting for more clients to be ready session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
742025-03-17T22:34:28.160ZINFOcrucible: Waiting for more clients to be ready session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
752025-03-17T22:34:28.160ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
762025-03-17T22:34:28.160ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
772025-03-17T22:34:28.160ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
782025-03-17T22:34:28.160ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
792025-03-17T22:34:28.160ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
802025-03-17T22:34:28.160ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
812025-03-17T22:34:28.160ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
822025-03-17T22:34:28.160ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
832025-03-17T22:34:28.160ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
842025-03-17T22:34:28.160ZINFOcrucible: Max found gen is 1 = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
852025-03-17T22:34:28.160ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
862025-03-17T22:34:28.160ZINFOcrucible: Next flush: 1 = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
872025-03-17T22:34:28.160ZINFOcrucible: All extents match = downstairs session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
882025-03-17T22:34:28.160ZINFOcrucible: No downstairs reconciliation required session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
892025-03-17T22:34:28.160ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
902025-03-17T22:34:28.160ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
912025-03-17T22:34:28.160ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
922025-03-17T22:34:28.160ZINFOcrucible: All required reconciliation work is completed session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
93 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Activate successful
94 Wait for a query_work_queue command to finish before sending IO
95 ,"time":"2025-03-17T22:34:28.160407459Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1204,"session_id":"10cfdc24-8436-4bf0-96a2-740034d5dcbc"}
962025-03-17T22:34:28.160ZINFOcrucible: 5859d175-50aa-427e-8003-fde1244e3a4b is now active with session: 10cfdc24-8436-4bf0-96a2-740034d5dcbc session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
972025-03-17T22:34:28.160ZINFOcrucible: Set Active after no reconciliation session_id = 10cfdc24-8436-4bf0-96a2-740034d5dcbc
982025-03-17T22:34:28.160ZINFOcrucible: The guest has finished waiting for activation with:1
992025-03-17T22:34:28.160ZINFOcrucible: Activated sub_volume 5859d175-50aa-427e-8003-fde1244e3a4b
100 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
101 Fill test
102 Read and Verify all blocks (0..5000 range:false)
103 Wrote out file "/tmp/test_replay_verify.log"
104 CLIENT: Tests done. All submitted work has been ACK'd
105 ----------------------------------------------------------------
106 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
107 States: Active Active Active
108 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
109 1021 Acked Read 256 Done Done Done false
110 1022 Acked Read 256 Done Done Done false
111 1023 Acked Read 256 Done Done Done false
112 1024 Acked Read 256 Done Done Done false
113 1025 Acked Read 256 Done Done Done false
114 1026 Acked Read 256 Done Done Done false
115 1027 Acked Read 256 Done Done Done false
116 1028 Acked Read 256 Done Done Done false
117 1029 Acked Read 256 Done Done Done false
118 1030 Acked Read 256 Done Done Done false
119 1031 Acked Read 256 Done Done Done false
120 1032 Acked Read 256 Done Done Done false
121 1033 Acked Read 256 Done Done Done false
122 1034 Acked Read 256 Done Done Done false
123 1035 Acked Read 256 Done Done Done false
124 1036 Acked Read 256 Done Done Done false
125 1037 Acked Read 256 Done Done Done false
126 1038 Acked Read 136 Done Done Done false
127 1039 Acked Read 256 Done Done Done false
128 1040 Acked Read 256 Done Done Done false
129 STATES DS:0 DS:1 DS:2 TOTAL
130 Sent 0 0 0 0
131 Done 20 20 20 60
132 Skipped 0 0 0 0
133 Error 0 0 0 0
134 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
135 Downstairs last five completed: 1020 1019 1018 1017 1016
136 Upstairs last five completed: 1040 1039 1038 1037 1036
137 CLIENT: Up:0 ds:20 act:3
138 ----------------------------------------------------------------
139 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
140 Downstairs last five completed: 1041 1040 1039 1038 1037
141 Upstairs last five completed: 1041 1040 1039 1038 1037
142 CLIENT: Up:0 ds:0 act:3
143 CLIENT: All crucible jobs finished, exiting program
144 Replay loop starts now March 17, 2025 at 10:34:32 PM UTC
145 Mar 17 22:34:32.791 INFO accepted connection, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
146 Mar 17 22:34:32.792 INFO request completed, latency_us: 171, response_code: 200, uri: /regioninfo, method: GET, req_id: 76c677ad-ff69-4218-bb5c-bda15e7fcec3, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
1472025-03-17T22:34:32.792ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Mar 17 22:34:32.793 INFO request completed, latency_us: 87, response_code: 200, uri: /regioncount, method: GET, req_id: e5c07806-bffc-4a5a-9b3c-c90b158d2607, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
1492025-03-17T22:34:32.793ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Mar 17 22:34:32.793 INFO request completed, latency_us: 112, response_code: 200, uri: /port/cid/0, method: GET, req_id: ff8e4318-6d54-4983-ae10-0a5957cae114, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
151 Mar 17 22:34:32.794 INFO request completed, latency_us: 99, response_code: 200, uri: /port/cid/1, method: GET, req_id: e75f2591-7e52-4b14-aa5f-2eb3fa210356, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
152 Mar 17 22:34:32.794 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/2, method: GET, req_id: 707a67a6-a4ef-4793-8450-e7e76a6f0d0e, remote_addr: 127.0.0.1:40297, local_addr: 127.0.0.1:9998
1532025-03-17T22:34:32.794ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-03-17T22:34:32.795ZINFOcrucible: Upstairs starts
1552025-03-17T22:34:32.795ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "45feeae3dd1bdcd88dce11a80acb5415f57b38bc", git_commit_timestamp: "2025-03-17T22:11:58.000000000Z", git_branch: "renovate/opentelemetry-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-03-17T22:34:32.795ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-03-17T22:34:32.795ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-03-17T22:34:32.795ZINFOcrucible: Crucible 7b8e3684-e6c8-40c1-987e-31ba678a8163 has session id: 34ae9076-d4ea-416d-9168-6f1a5fc19802 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1592025-03-17T22:34:32.795ZINFOcrucible: Upstairs opts: Upstairs UUID: 7b8e3684-e6c8-40c1-987e-31ba678a8163, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1602025-03-17T22:34:32.795ZINFOcrucible: Crucible stats registered with UUID: 7b8e3684-e6c8-40c1-987e-31ba678a8163 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1612025-03-17T22:34:32.795ZWARNcrucible: could not find Downstairs address for Nexus session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1622025-03-17T22:34:32.796ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-03-17T22:34:32.796ZINFOcrucible: The guest has requested activation with gen:2
1642025-03-17T22:34:32.796ZINFOcrucible: 7b8e3684-e6c8-40c1-987e-31ba678a8163 active request set session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1652025-03-17T22:34:32.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1662025-03-17T22:34:32.796ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1672025-03-17T22:34:32.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1682025-03-17T22:34:32.796ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1692025-03-17T22:34:32.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1702025-03-17T22:34:32.796ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1712025-03-17T22:34:32.797ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1722025-03-17T22:34:32.797ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1732025-03-17T22:34:32.797ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1742025-03-17T22:34:32.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42428) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1752025-03-17T22:34:32.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55996) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1762025-03-17T22:34:32.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36697) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1772025-03-17T22:34:32.797ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1782025-03-17T22:34:32.797ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1792025-03-17T22:34:32.797ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1802025-03-17T22:34:32.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1812025-03-17T22:34:32.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1822025-03-17T22:34:32.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1832025-03-17T22:34:32.799ZINFOcrucible: Waiting for more clients to be ready session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1842025-03-17T22:34:32.799ZINFOcrucible: Waiting for more clients to be ready session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1852025-03-17T22:34:32.799ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1862025-03-17T22:34:32.799ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1872025-03-17T22:34:32.799ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1882025-03-17T22:34:32.799ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1892025-03-17T22:34:32.799ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1902025-03-17T22:34:32.799ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1912025-03-17T22:34:32.799ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1922025-03-17T22:34:32.799ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1932025-03-17T22:34:32.799ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1942025-03-17T22:34:32.799ZINFOcrucible: Max found gen is 2 = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1952025-03-17T22:34:32.799ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
196 {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:34:32.799593793Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"downstairs","":"downstairs","Activate successful
197 session_id"Wait for a query_work_queue command to finish before sending IO
198 :"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
1992025-03-17T22:34:32.799ZINFOcrucible: All extents match = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2002025-03-17T22:34:32.799ZINFOcrucible: No downstairs reconciliation required session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2012025-03-17T22:34:32.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2022025-03-17T22:34:32.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2032025-03-17T22:34:32.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2042025-03-17T22:34:32.799ZINFOcrucible: All required reconciliation work is completed session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2052025-03-17T22:34:32.799ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2062025-03-17T22:34:32.799ZINFOcrucible: 7b8e3684-e6c8-40c1-987e-31ba678a8163 is now active with session: 34ae9076-d4ea-416d-9168-6f1a5fc19802 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2072025-03-17T22:34:32.799ZINFOcrucible: Set Active after no reconciliation session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2082025-03-17T22:34:32.799ZINFOcrucible: The guest has finished waiting for activation with:2
2092025-03-17T22:34:32.799ZINFOcrucible: Activated sub_volume 7b8e3684-e6c8-40c1-987e-31ba678a8163
210 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
211 Loading write count information from file "/tmp/test_replay_verify.log"
212 Mar 17 22:34:32.833 INFO accepted connection, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
213 Main task has work to do, go find it
214 got dsc Stop(1)
215 stop 1
216 Mar 17 22:34:32.834 INFO request completed, latency_us: 160, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 05f33701-059e-46f8-8546-51bceb10d919, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
217 [8820] Got stop action so:false kr:false
2182025-03-17T22:34:32.834ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2192025-03-17T22:34:32.834ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2202025-03-17T22:34:32.834ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2212025-03-17T22:34:32.834ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
222 Mar 17 22:34:32.834 INFO request completed, latency_us: 164, response_code: 200, uri: /state/cid/1, method: GET, req_id: f326a376-ad4c-40e6-b3f6-355f4af8028b, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
223 [8820] Exited with: ExitStatus(unix_wait_status(9))
224 [8820][1] reports Exit
225 Mar 17 22:34:36.837 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2fadd6b9-6d9e-485a-af89-979f98c2bea3, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
226 001/300 Read block 4036 len 4096
227 002/300 Write block 1291 len 28672 data: 2 2 2 2 2 2 2
228 003/300 Write block 108 len 36864 data: 2 2 2 2 2 2 2 2 2
229 004/300 Read block 1964 len 28672
230 005/300 Read block 2780 len 32768
231 006/300 Write block 3181 len 36864 data: 2 2 2 2 2 2 2 2 2
232 007/300 Write block 1525 len 40960 data: 2 2 2 2 2 2 2 2 2 2
233 008/300 Write block 2161 len 40960 data: 2 2 2 2 2 2 2 2 2 2
234 009/300 Write block 3729 len 28672 data: 2 2 2 2 2 2 2
235 010/300 Write block 3178 len 36864 data: 2 2 2 3 3 3 3 3 3
236 011/300 Write block 1866 len 4096 data: 2
237 012/300 Read block 1932 len 16384
238 013/300 Read block 2551 len 16384
239 014/300 Read block 3505 len 28672
240 015/300 Read block 2648 len 36864
241 016/300 Read block 2124 len 28672
242 017/300 Read block 2907 len 12288
243 018/300 Flush
244 019/300 Write block 3051 len 28672 data: 2 2 2 2 2 2 2
245 020/300 Write block 4501 len 24576 data: 2 2 2 2 2 2
246 021/300 Flush
247 022/300 Read block 756 len 16384
248 023/300 Flush
249 024/300 Write block 3776 len 40960 data: 2 2 2 2 2 2 2 2 2 2
250 025/300 Write block 2777 len 8192 data: 2 2
251 026/300 Read block 2052 len 20480
252 027/300 Write block 4658 len 24576 data: 2 2 2 2 2 2
253 028/300 Write block 2371 len 24576 data: 2 2 2 2 2 2
254 029/300 Read block 4315 len 40960
255 030/300 Write block 254 len 8192 data: 2 2
256 031/300 Read block 1849 len 16384
257 032/300 Read block 4364 len 24576
258 033/300 Write block 2806 len 24576 data: 2 2 2 2 2 2
259 034/300 Flush
260 035/300 Read block 1621 len 4096
261 036/300 Read block 3978 len 40960
262 037/300 Read block 4479 len 32768
263 038/300 Read block 4092 len 28672
264 039/300 Write block 3133 len 8192 data: 2 2
265 040/300 Read block 3487 len 40960
266 041/300 Read block 1219 len 20480
267 042/300 Write block 818 len 8192 data: 2 2
268 043/300 Write block 273 len 16384 data: 2 2 2 2
269 044/300 Write block 3463 len 8192 data: 2 2
270 045/300 Flush
271 046/300 Read block 1331 len 28672
272 047/300 Flush
273 048/300 Read block 4425 len 4096
274 049/300 Flush
275 050/300 Write block 2026 len 36864 data: 2 2 2 2 2 2 2 2 2
276 051/300 Read block 1014 len 24576
277 052/300 Read block 2641 len 20480
278 053/300 Flush
279 054/300 Read block 4624 len 28672
280 055/300 Write block 4225 len 36864 data: 2 2 2 2 2 2 2 2 2
281 056/300 Write block 1504 len 24576 data: 2 2 2 2 2 2
282 057/300 Flush
283 058/300 Write block 414 len 32768 data: 2 2 2 2 2 2 2 2
284 059/300 Read block 4741 len 32768
285 060/300 Write block 1090 len 20480 data: 2 2 2 2 2
286 061/300 Write block 4247 len 28672 data: 2 2 2 2 2 2 2
287 062/300 Read block 1546 len 24576
288 063/300 Write block 4381 len 4096 data: 2
289 064/300 Read block 1917 len 20480
290 065/300 Write block 1083 len 24576 data: 2 2 2 2 2 2
291 066/300 Read block 1410 len 16384
292 067/300 Write block 4131 len 8192 data: 2 2
293 068/300 Write block 4846 len 36864 data: 2 2 2 2 2 2 2 2 2
294 069/300 Write block 415 len 4096 data: 3
295 070/300 Write block 23 len 8192 data: 2 2
296 071/300 Write block 3384 len 20480 data: 2 2 2 2 2
297 072/300 Write block 1870 len 36864 data: 2 2 2 2 2 2 2 2 2
298 073/300 Read block 2583 len 16384
299 074/300 Write block 4154 len 24576 data: 2 2 2 2 2 2
300 075/300 Read block 3804 len 36864
301 076/300 Read block 1611 len 32768
302 077/300 Flush
303 078/300 Read block 3540 len 20480
304 079/300 Read block 1794 len 8192
305 080/300 Read block 2995 len 12288
306 081/300 Read block 3830 len 20480
307 082/300 Write block 513 len 16384 data: 2 2 2 2
308 083/300 Write block 3069 len 28672 data: 2 2 2 2 2 2 2
309 084/300 Write block 1744 len 12288 data: 2 2 2
310 085/300 Write block 4822 len 28672 data: 2 2 2 2 2 2 2
311 086/300 Read block 4780 len 24576
312 087/300 Write block 748 len 4096 data: 2
313 088/300 Read block 4049 len 24576
314 089/300 Write block 1783 len 20480 data: 2 2 2 2 2
315 090/300 Write block 2707 len 4096 data: 2
316 091/300 Read block 1661 len 36864
317 092/300 Read block 1699 len 32768
318 093/300 Write block 648 len 28672 data: 2 2 2 2 2 2 2
319 094/300 Read block 1040 len 12288
320 095/300 Write block 969 len 28672 data: 2 2 2 2 2 2 2
321 096/300 Read block 2866 len 8192
322 097/300 Write block 4432 len 28672 data: 2 2 2 2 2 2 2
323 098/300 Write block 4564 len 24576 data: 2 2 2 2 2 2
324 099/300 Read block 3301 len 36864
325 100/300 Write block 155 len 8192 data: 2 2
326 101/300 Write block 1593 len 20480 data: 2 2 2 2 2
327 102/300 Read block 92 len 20480
328 103/300 Read block 3285 len 36864
329 104/300 Write block 3905 len 32768 data: 2 2 2 2 2 2 2 2
330 105/300 Write block 2073 len 20480 data: 2 2 2 2 2
331 106/300 Write block 3506 len 8192 data: 2 2
332 107/300 Read block 4528 len 8192
333 108/300 Flush
334 109/300 Flush
335 110/300 Write block 1990 len 4096 data: 2
336 111/300 Read block 4769 len 40960
337 112/300 Flush
338 113/300 Read block 1978 len 12288
339 114/300 Flush
340 115/300 Read block 2540 len 8192
341 116/300 Flush
342 117/300 Write block 1383 len 4096 data: 2
343 118/300 Flush
344 119/300 Write block 3485 len 20480 data: 2 2 2 2 2
345 120/300 Read block 3527 len 20480
346 121/300 Write block 446 len 20480 data: 2 2 2 2 2
347 122/300 Flush
348 123/300 Read block 4620 len 24576
349 124/300 Read block 194 len 32768
350 125/300 Read block 4379 len 4096
351 126/300 Write block 2493 len 16384 data: 2 2 2 2
352 127/300 Read block 1234 len 24576
353 128/300 Read block 2274 len 4096
354 129/300 Write block 4773 len 28672 data: 2 2 2 2 2 2 2
355 130/300 Read block 1990 len 16384
356 131/300 Flush
357 132/300 Read block 1516 len 40960
358 133/300 Write block 2307 len 32768 data: 2 2 2 2 2 2 2 2
359 134/300 Write block 1463 len 20480 data: 2 2 2 2 2
360 135/300 Read block 283 len 8192
361 136/300 Read block 2937 len 20480
362 137/300 Read block 4692 len 4096
363 138/300 Write block 3546 len 12288 data: 2 2 2
364 139/300 Read block 1246 len 32768
365 140/300 Read block 2692 len 8192
366 141/300 Write block 4709 len 28672 data: 2 2 2 2 2 2 2
367 142/300 Read block 3497 len 20480
368 143/300 Read block 4496 len 20480
369 144/300 Read block 4210 len 20480
370 145/300 Write block 2287 len 4096 data: 2
371 146/300 Write block 4091 len 28672 data: 2 2 2 2 2 2 2
372 147/300 Read block 2547 len 32768
373 148/300 Read block 4407 len 4096
374 149/300 Read block 3151 len 24576
375 150/300 Read block 374 len 20480
376 151/300 Read block 3002 len 24576
377 152/300 Read block 4581 len 40960
378 153/300 Read block 4493 len 20480
379 154/300 Read block 2461 len 24576
380 155/300 Flush
381 156/300 Flush
382 157/300 Flush
383 158/300 Read block 754 len 32768
384 159/300 Write block 3892 len 24576 data: 2 2 2 2 2 2
385 160/300 Write block 2911 len 28672 data: 2 2 2 2 2 2 2
386 161/300 Write block 3432 len 36864 data: 2 2 2 2 2 2 2 2 2
387 162/300 Read block 847 len 32768
388 163/300 Read block 2542 len 24576
389 164/300 Read block 3813 len 16384
390 165/300 Flush
391 166/300 Read block 2326 len 8192
392 167/300 Flush
393 168/300 Flush
394 169/300 Read block 3411 len 24576
395 170/300 Write block 1065 len 40960 data: 2 2 2 2 2 2 2 2 2 2
396 171/300 Write block 3136 len 4096 data: 2
397 172/300 Read block 768 len 20480
398 173/300 Write block 1525 len 4096 data: 3
399 174/300 Read block 1534 len 24576
400 175/300 Write block 4910 len 40960 data: 2 2 2 2 2 2 2 2 2 2
401 176/300 Write block 2048 len 32768 data: 2 2 2 2 2 2 2 2
402 177/300 Read block 1872 len 36864
403 178/300 Write block 3805 len 36864 data: 2 2 2 2 2 2 2 2 2
404 179/300 Read block 3599 len 16384
405 180/300 Write block 402 len 36864 data: 2 2 2 2 2 2 2 2 2
406 181/300 Read block 1027 len 36864
407 182/300 Read block 1825 len 12288
408 183/300 Flush
409 184/300 Write block 1829 len 4096 data: 2
410 185/300 Read block 1884 len 32768
411 186/300 Flush
412 187/300 Write block 2729 len 28672 data: 2 2 2 2 2 2 2
413 188/300 Read block 586 len 32768
414 189/300 Write block 478 len 28672 data: 2 2 2 2 2 2 2
415 190/300 Write block 335 len 40960 data: 2 2 2 2 2 2 2 2 2 2
416 191/300 Write block 1178 len 16384 data: 2 2 2 2
417 192/300 Write block 4373 len 36864 data: 2 2 2 2 2 2 2 2 3
418 193/300 Write block 1399 len 12288 data: 2 2 2
419 194/300 Read block 2186 len 4096
420 195/300 Read block 53 len 36864
421 196/300 Read block 1837 len 24576
422 197/300 Read block 1023 len 8192
423 198/300 Read block 3640 len 4096
424 199/300 Read block 3587 len 4096
425 200/300 Read block 4589 len 32768
426 201/300 Flush
427 202/300 Write block 2785 len 40960 data: 2 2 2 2 2 2 2 2 2 2
428 203/300 Read block 2721 len 36864
429 204/300 Write block 4633 len 32768 data: 2 2 2 2 2 2 2 2
430 205/300 Read block 1340 len 16384
431 206/300 Read block 2025 len 36864
432 207/300 Read block 1772 len 4096
433 208/300 Read block 2804 len 40960
434 209/300 Write block 4152 len 12288 data: 2 2 3
435 210/300 Write block 1601 len 40960 data: 2 2 2 2 2 2 2 2 2 2
436 211/300 Read block 1464 len 40960
437 212/300 Write block 4281 len 28672 data: 2 2 2 2 2 2 2
438 213/300 Read block 2165 len 32768
439 214/300 Write block 4351 len 16384 data: 2 2 2 2
440 215/300 Read block 4099 len 40960
441 216/300 Write block 4624 len 4096 data: 2
442 217/300 Write block 4928 len 8192 data: 2 2
443 218/300 Write block 4013 len 20480 data: 2 2 2 2 2
444 219/300 Read block 2357 len 40960
445 220/300 Write block 1271 len 32768 data: 2 2 2 2 2 2 2 2
446 221/300 Read block 3198 len 40960
447 222/300 Write block 1895 len 16384 data: 2 2 2 2
448 223/300 Read block 1271 len 24576
449 224/300 Read block 2931 len 32768
450 225/300 Write block 3304 len 8192 data: 2 2
451 226/300 Write block 2576 len 32768 data: 2 2 2 2 2 2 2 2
452 227/300 Write block 1204 len 20480 data: 2 2 2 2 2
453 228/300 Write block 4327 len 16384 data: 2 2 2 2
454 229/300 Write block 120 len 32768 data: 2 2 2 2 2 2 2 2
455 230/300 Write block 3209 len 40960 data: 2 2 2 2 2 2 2 2 2 2
456 231/300 Write block 2253 len 12288 data: 2 2 2
457 232/300 Read block 375 len 4096
458 233/300 Read block 2322 len 32768
459 234/300 Read block 833 len 12288
460 235/300 Read block 836 len 16384
461 236/300 Write block 3727 len 12288 data: 2 2 3
462 237/300 Read block 2881 len 28672
463 238/300 Write block 2568 len 4096 data: 2
464 239/300 Write block 4185 len 12288 data: 2 2 2
465 240/300 Read block 3077 len 12288
466 241/300 Read block 3246 len 24576
467 242/300 Read block 151 len 16384
468 243/300 Read block 516 len 12288
469 244/300 Write block 3367 len 40960 data: 2 2 2 2 2 2 2 2 2 2
470 245/300 Write block 3519 len 20480 data: 2 2 2 2 2
471 246/300 Flush
472 247/300 Write block 884 len 4096 data: 2
473 248/300 Read block 565 len 32768
474 249/300 Read block 4102 len 24576
475 250/300 Read block 515 len 20480
476 251/300 Read block 4495 len 12288
477 252/300 Read block 3466 len 40960
478 253/300 Read block 3455 len 4096
479 254/300 Read block 1010 len 40960
480 255/300 Read block 4508 len 12288
481 256/300 Flush
482 257/300 Write block 4406 len 20480 data: 2 2 2 2 2
483 258/300 Read block 3709 len 32768
484 259/300 Read block 853 len 32768
485 260/300 Read block 4528 len 40960
486 261/300 Write block 943 len 24576 data: 2 2 2 2 2 2
487 262/300 Write block 375 len 8192 data: 2 2
488 263/300 Read block 3491 len 36864
489 264/300 Read block 1197 len 40960
490 265/300 Write block 305 len 8192 data: 2 2
491 266/300 Write block 4234 len 36864 data: 2 2 2 2 2 2 2 2 2
492 267/300 Flush
493 268/300 Read block 439 len 32768
494 269/300 Read block 2304 len 24576
495 270/300 Read block 3138 len 24576
496 271/300 Write block 4923 len 36864 data: 2 2 2 2 2 3 3 2 2
497 272/300 Read block 3301 len 12288
498 273/300 Flush
499 274/300 Read block 1877 len 4096
500 275/300 Read block 4860 len 20480
501 276/300 Write block 239 len 16384 data: 2 2 2 2
502 277/300 Read block 3914 len 8192
503 278/300 Read block 2471 len 8192
504 279/300 Read block 2518 len 28672
505 280/300 Read block 2283 len 36864
506 281/300 Write block 275 len 32768 data: 3 3 2 2 2 2 2 2
507 282/300 Write block 1533 len 4096 data: 3
508 283/300 Write block 519 len 32768 data: 2 2 2 2 2 2 2 2
509 284/300 Read block 3757 len 40960
510 285/300 Write block 921 len 4096 data: 2
511 286/300 Write block 2370 len 4096 data: 2
512 287/300 Flush
513 288/300 Flush
514 289/300 Read block 4159 len 16384
515 290/300 Read block 464 len 40960
516 291/300 Read block 2738 len 40960
517 292/300 Write block 534 len 8192 data: 2 2
518 293/300 Flush
519 294/300 Read block 4848 len 12288
520 295/300 Write block 1007 len 4096 data: 2
521 296/300 Write block 3522 len 4096 data: 3
522 297/300 Flush
523 298/300 Read block 4900 len 20480
524 299/300 Read block 697 len 12288
525 300/300 Write block 4143 len 24576 data: 2 2 2 2 2 2
526 301/300 Read block 3940 len 36864
527 Main task has work to do, go find it
528 got dsc Start(1)
529 start 1
530 Mar 17 22:34:37.084 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/1, method: POST, req_id: f1a27137-5b30-4641-ace0-fa256f676034, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
531 Replay: started 1, returned:Ok(())
532 ----------------------------------------------------------------
533 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
534 States: Active Offline Active
535 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
536 1000 Acked Read 1 Done Sent Done false
537 1001 Acked Write 7 Done Sent Done false
538 1002 Acked Write 9 Done Sent Done false
539 1003 Acked Read 7 Done Sent Done false
540 1004 Acked Read 8 Done Sent Done false
541 1005 Acked Write 9 Done Sent Done false
542 1006 Acked Write 10 Done Sent Done false
543 1007 Acked Write 10 Done Sent Done false
544 1008 Acked Write 7 Done Sent Done false
545 1009 Acked Write 9 Done Sent Done false
546 1010 Acked Write 1 Done Sent Done false
547 1011 Acked Read 4 Done Sent Done false
548 1012 Acked Read 4 Done Sent Done false
549 1013 Acked Read 7 Done Sent Done false
550 1014 Acked Read 9 Done Sent Done false
551 1015 Acked Read 7 Done Sent Done false
552 1016 Acked Read 3 Done Sent Done false
553 1017 Acked Flush 0 Done Sent Done false
554 1018 Acked Write 7 Done Sent Done false
555 1019 Acked Write 6 Done Sent Done false
556 1020 Acked Flush 0 Done Sent Done false
557 1021 Acked Read 4 Done Sent Done false
558 1022 Acked Flush 0 Done Sent Done false
559 1023 Acked Write 10 Done Sent Done false
560 1024 Acked Write 2 Done Sent Done false
561 1025 Acked Read 5 Done Sent Done false
562 1026 Acked Write 6 Done Sent Done false
563 1027 Acked Write 6 Done Sent Done false
564 1028 Acked Read 10 Done Sent Done false
565 1029 Acked Write 2 Done Sent Done false
566 1030 Acked Read 4 Done Sent Done false
567 1031 Acked Read 6 Done Sent Done false
568 1032 Acked Write 6 Done Sent Done false
569 1033 Acked Flush 0 Done Sent Done false
570 1034 Acked Read 1 Done Sent Done false
571 1035 Acked Read 10 Done Sent Done false
572 1036 Acked Read 8 Done Sent Done false
573 1037 Acked Read 7 Done Sent Done false
574 1038 Acked Write 2 Done Sent Done false
575 1039 Acked Read 10 Done Sent Done false
576 1040 Acked Read 5 Done Sent Done false
577 1041 Acked Write 2 Done Sent Done false
578 1042 Acked Write 4 Done Sent Done false
579 1043 Acked Write 2 Done Sent Done false
580 1044 Acked Flush 0 Done Sent Done false
581 1045 Acked Read 7 Done Sent Done false
582 1046 Acked Flush 0 Done Sent Done false
583 1047 Acked Read 1 Done Sent Done false
584 1048 Acked Flush 0 Done Sent Done false
585 1049 Acked Write 9 Done Sent Done false
586 1050 Acked Read 6 Done Sent Done false
587 1051 Acked Read 5 Done Sent Done false
588 1052 Acked Flush 0 Done Sent Done false
589 1053 Acked Read 7 Done Sent Done false
590 1054 Acked Write 9 Done Sent Done false
591 1055 Acked Write 6 Done Sent Done false
592 1056 Acked Flush 0 Done Sent Done false
593 1057 Acked Write 8 Done Sent Done false
594 1058 Acked Read 8 Done Sent Done false
595 1059 Acked Write 5 Done Sent Done false
596 1060 Acked Write 7 Done Sent Done false
597 1061 Acked Read 6 Done Sent Done false
598 1062 Acked Write 1 Done Sent Done false
599 1063 Acked Read 5 Done Sent Done false
600 1064 Acked Write 6 Done Sent Done false
601 1065 Acked Read 4 Done Sent Done false
602 1066 Acked Write 2 Done Sent Done false
603 1067 Acked Write 9 Done Sent Done false
604 1068 Acked Write 1 Done Sent Done false
605 1069 Acked Write 2 Done Sent Done false
606 1070 Acked Write 5 Done Sent Done false
607 1071 Acked Write 9 Done Sent Done false
608 1072 Acked Read 4 Done Sent Done false
609 1073 Acked Write 6 Done Sent Done false
610 1074 Acked Read 9 Done Sent Done false
611 1075 Acked Read 8 Done Sent Done false
612 1076 Acked Flush 0 Done Sent Done false
613 1077 Acked Read 5 Done Sent Done false
614 1078 Acked Read 2 Done Sent Done false
615 1079 Acked Read 3 Done Sent Done false
616 1080 Acked Read 5 Done Sent Done false
617 1081 Acked Write 4 Done Sent Done false
618 1082 Acked Write 7 Done Sent Done false
619 1083 Acked Write 3 Done Sent Done false
620 1084 Acked Write 7 Done Sent Done false
621 1085 Acked Read 6 Done Sent Done false
622 1086 Acked Write 1 Done Sent Done false
623 1087 Acked Read 6 Done Sent Done false
624 1088 Acked Write 5 Done Sent Done false
625 1089 Acked Write 1 Done Sent Done false
626 1090 Acked Read 9 Done Sent Done false
627 1091 Acked Read 8 Done Sent Done false
628 1092 Acked Write 7 Done Sent Done false
629 1093 Acked Read 3 Done Sent Done false
630 1094 Acked Write 7 Done Sent Done false
631 1095 Acked Read 2 Done Sent Done false
632 1096 Acked Write 7 Done Sent Done false
633 1097 Acked Write 6 Done Sent Done false
634 1098 Acked Read 9 Done Sent Done false
635 1099 Acked Write 2 Done Sent Done false
636 1100 Acked Write 5 Done Sent Done false
637 1101 Acked Read 5 Done Sent Done false
638 1102 Acked Read 9 Done Sent Done false
639 1103 Acked Write 8 Done Sent Done false
640 1104 Acked Write 5 Done Sent Done false
641 1105 Acked Write 2 Done Sent Done false
642 1106 Acked Read 2 Done Sent Done false
643 1107 Acked Flush 0 Done Sent Done false
644 1108 Acked Flush 0 Done Sent Done false
645 1109 Acked Write 1 Done Sent Done false
646 1110 Acked Read 10 Done Sent Done false
647 1111 Acked Flush 0 Done Sent Done false
648 1112 Acked Read 3 Done Sent Done false
649 1113 Acked Flush 0 Done Sent Done false
650 1114 Acked Read 2 Done Sent Done false
651 1115 Acked Flush 0 Done Sent Done false
652 1116 Acked Write 1 Done Sent Done false
653 1117 Acked Flush 0 Done Sent Done false
654 1118 Acked Write 5 Done Sent Done false
655 1119 Acked Read 5 Done Sent Done false
656 1120 Acked Write 5 Done Sent Done false
657 1121 Acked Flush 0 Done Sent Done false
658 1122 Acked Read 6 Done Sent Done false
659 1123 Acked Read 8 Done Sent Done false
660 1124 Acked Read 1 Done Sent Done false
661 1125 Acked Write 4 Done Sent Done false
662 1126 Acked Read 6 Done Sent Done false
663 1127 Acked Read 1 Done Sent Done false
664 1128 Acked Write 7 Done Sent Done false
665 1129 Acked Read 4 Done Sent Done false
666 1130 Acked Flush 0 Done Sent Done false
667 1131 Acked Read 10 Done Sent Done false
668 1132 Acked Write 8 Done Sent Done false
669 1133 Acked Write 5 Done Sent Done false
670 1134 Acked Read 2 Done Sent Done false
671 1135 Acked Read 5 Done Sent Done false
672 1136 Acked Read 1 Done Sent Done false
673 1137 Acked Write 3 Done Sent Done false
674 1138 Acked Read 8 Done Sent Done false
675 1139 Acked Read 2 Done Sent Done false
676 1140 Acked Write 7 Done Sent Done false
677 1141 Acked Read 5 Done Sent Done false
678 1142 Acked Read 5 Done Sent Done false
679 1143 Acked Read 5 Done Sent Done false
680 1144 Acked Write 1 Done Sent Done false
681 1145 Acked Write 7 Done Sent Done false
682 1146 Acked Read 8 Done Sent Done false
683 1147 Acked Read 1 Done Sent Done false
684 1148 Acked Read 6 Done Sent Done false
685 1149 Acked Read 5 Done Sent Done false
686 1150 Acked Read 6 Done Sent Done false
687 1151 Acked Read 10 Done Sent Done false
688 1152 Acked Read 5 Done Sent Done false
689 1153 Acked Read 6 Done Sent Done false
690 1154 Acked Flush 0 Done Sent Done false
691 1155 Acked Flush 0 Done Sent Done false
692 1156 Acked Flush 0 Done Sent Done false
693 1157 Acked Read 8 Done Sent Done false
694 1158 Acked Write 6 Done Sent Done false
695 1159 Acked Write 7 Done Sent Done false
696 1160 Acked Write 9 Done Sent Done false
697 1161 Acked Read 8 Done Sent Done false
698 1162 Acked Read 6 Done Sent Done false
699 1163 Acked Read 4 Done Sent Done false
700 1164 Acked Flush 0 Done Sent Done false
701 1165 Acked Read 2 Done Sent Done false
702 1166 Acked Flush 0 Done Sent Done false
703 1167 Acked Flush 0 Done Sent Done false
704 1168 Acked Read 6 Done Sent Done false
705 1169 Acked Write 10 Done Sent Done false
706 1170 Acked Write 1 Done Sent Done false
707 1171 Acked Read 5 Done Sent Done false
708 1172 Acked Write 1 Done Sent Done false
709 1173 Acked Read 6 Done Sent Done false
710 1174 Acked Write 10 Done Sent Done false
711 1175 Acked Write 8 Done Sent Done false
712 1176 Acked Read 9 Done Sent Done false
713 1177 Acked Write 9 Done Sent Done false
714 1178 Acked Read 4 Done Sent Done false
715 1179 Acked Write 9 Done Sent Done false
716 1180 Acked Read 9 Done Sent Done false
717 1181 Acked Read 3 Done Sent Done false
718 1182 Acked Flush 0 Done Sent Done false
719 1183 Acked Write 1 Done Sent Done false
720 1184 Acked Read 8 Done Sent Done false
721 1185 Acked Flush 0 Done Sent Done false
722 1186 Acked Write 7 Done Sent Done false
723 1187 Acked Read 8 Done Sent Done false
724 1188 Acked Write 7 Done Sent Done false
725 1189 Acked Write 10 Done Sent Done false
726 1190 Acked Write 4 Done Sent Done false
727 1191 Acked Write 9 Done Sent Done false
728 1192 Acked Write 3 Done Sent Done false
729 1193 Acked Read 1 Done Sent Done false
730 1194 Acked Read 9 Done Sent Done false
731 1195 Acked Read 6 Done Sent Done false
732 1196 Acked Read 2 Done Sent Done false
733 1197 Acked Read 1 Done Sent Done false
734 1198 Acked Read 1 Done Sent Done false
735 1199 Acked Read 8 Done Sent Done false
736 1200 Acked Flush 0 Done Sent Done false
737 1201 Acked Write 10 Done Sent Done false
738 1202 Acked Read 9 Done Sent Done false
739 1203 Acked Write 8 Done Sent Done false
740 1204 Acked Read 4 Done Sent Done false
741 1205 Acked Read 9 Done Sent Done false
742 1206 Acked Read 1 Done Sent Done false
743 1207 Acked Read 10 Done Sent Done false
744 1208 Acked Write 3 Done Sent Done false
745 1209 Acked Write 10 Done Sent Done false
746 1210 Acked Read 10 Done Sent Done false
747 1211 Acked Write 7 Done Sent Done false
748 1212 Acked Read 8 Done Sent Done false
749 1213 Acked Write 4 Done Sent Done false
750 1214 Acked Read 10 Done Sent Done false
751 1215 Acked Write 1 Done Sent Done false
752 1216 Acked Write 2 Done Sent Done false
753 1217 Acked Write 5 Done Sent Done false
754 1218 Acked Read 10 Done Sent Done false
755 1219 Acked Write 8 Done Sent Done false
756 1220 Acked Read 10 Done Sent Done false
757 1221 Acked Write 4 Done Sent Done false
758 1222 Acked Read 6 Done Sent Done false
759 1223 Acked Read 8 Done Sent Done false
760 1224 Acked Write 2 Done Sent Done false
761 1225 Acked Write 8 Done Sent Done false
762 1226 Acked Write 5 Done Sent Done false
763 1227 Acked Write 4 Done Sent Done false
764 1228 Acked Write 8 Done Sent Done false
765 1229 Acked Write 10 Done Sent Done false
766 1230 Acked Write 3 Done Sent Done false
767 1231 Acked Read 1 Done Sent Done false
768 1232 Acked Read 8 Done Sent Done false
769 1233 Acked Read 3 Done Sent Done false
770 1234 Acked Read 4 Done Sent Done false
771 1235 Acked Write 3 Done Sent Done false
772 1236 Acked Read 7 Done Sent Done false
773 1237 Acked Write 1 Done Sent Done false
774 1238 Acked Write 3 Done Sent Done false
775 1239 Acked Read 3 Done Sent Done false
776 1240 Acked Read 6 Done Sent Done false
777 1241 Acked Read 4 Done Sent Done false
778 1242 Acked Read 3 Done Sent Done false
779 1243 Acked Write 10 Done Sent Done false
780 1244 Acked Write 5 Done Sent Done false
781 1245 Acked Flush 0 Done Sent Done false
782 1246 Acked Write 1 Done Sent Done false
783 1247 Acked Read 8 Done Sent Done false
784 1248 Acked Read 6 Done Sent Done false
785 1249 Acked Read 5 Done Sent Done false
786 1250 Acked Read 3 Done Sent Done false
787 1251 Acked Read 10 Done Sent Done false
788 1252 Acked Read 1 Done Sent Done false
789 1253 Acked Read 10 Done Sent Done false
790 1254 Acked Read 3 Done Sent Done false
791 1255 Acked Flush 0 Done Sent Done false
792 1256 Acked Write 5 Done Sent Done false
793 1257 Acked Read 8 Done Sent Done false
794 1258 Acked Read 8 Done Sent Done false
795 1259 Acked Read 10 Done Sent Done false
796 1260 Acked Write 6 Done Sent Done false
797 1261 Acked Write 2 Done Sent Done false
798 1262 Acked Read 9 Done Sent Done false
799 1263 Acked Read 10 Done Sent Done false
800 1264 Acked Write 2 Done Sent Done false
801 1265 Acked Write 9 Done Sent Done false
802 1266 Acked Flush 0 Done Sent Done false
803 1267 Acked Read 8 Done Sent Done false
804 1268 Acked Read 6 Done Sent Done false
805 1269 Acked Read 6 Done Sent Done false
806 1270 Acked Write 9 Done Sent Done false
807 1271 Acked Read 3 Done Sent Done false
808 1272 Acked Flush 0 Done Sent Done false
809 1273 Acked Read 1 Done Sent Done false
810 1274 Acked Read 5 Done Sent Done false
811 1275 Acked Write 4 Done Sent Done false
812 1276 Acked Read 2 Done Sent Done false
813 1277 Acked Read 2 Done Sent Done false
814 1278 Acked Read 7 Done Sent Done false
815 1279 Acked Read 9 Done Sent Done false
816 1280 Acked Write 8 Done Sent Done false
817 1281 Acked Write 1 Done Sent Done false
818 1282 Acked Write 8 Done Sent Done false
819 1283 Acked Read 10 Done Sent Done false
820 1284 Acked Write 1 Done Sent Done false
821 1285 Acked Write 1 Done Sent Done false
822 1286 Acked Flush 0 Done Sent Done false
823 1287 Acked Flush 0 Done Sent Done false
824 1288 Acked Read 4 Done Sent Done false
825 1289 Acked Read 10 Done Sent Done false
826 1290 Acked Read 10 Done Sent Done false
827 1291 Acked Write 2 Done Sent Done false
828 1292 Acked Flush 0 Done Sent Done false
829 1293 Acked Read 3 Done Sent Done false
830 1294 Acked Write 1 Done Sent Done false
831 1295 Acked Write 1 Done Sent Done false
832 1296 Acked Flush 0 Done Sent Done false
833 1297 Acked Read 5 Done Sent Done false
834 1298 Acked Read 3 Done Sent Done false
835 1299 Acked Write 6 Done Sent Done false
836 1300 Acked Read 9 Done Sent Done false
837 STATES DS:0 DS:1 DS:2 TOTAL
838 Sent 0 301 0 301
839 Done 301 0 301 602
840 Skipped 0 0 0 0
841 Error 0 0 0 0
842 Last Flush: Some(JobId(1296)) None Some(JobId(1296))
843 Downstairs last five completed:
844 Upstairs last five completed: 1300 1299 1298 1297 1296
845 CLIENT: Up:0 ds:301 act:2
8462025-03-17T22:34:37.586ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8472025-03-17T22:34:38.305ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
848 [8820] got start action
849 [8820] I am going to restart
850 Starting downstairs at port 8820
851 Make output file at "/tmp/dsc/downstairs-8820.txt"
852 [8820][1] reports Starting
853 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1209)
854 [8820][1] reports Running
8552025-03-17T22:34:39.306ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8562025-03-17T22:34:40.307ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
857 ----------------------------------------------------------------
858 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
859 States: Active Offline Active
860 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
861 1000 Acked Read 1 Done Sent Done false
862 1001 Acked Write 7 Done Sent Done false
863 1002 Acked Write 9 Done Sent Done false
864 1003 Acked Read 7 Done Sent Done false
865 1004 Acked Read 8 Done Sent Done false
866 1005 Acked Write 9 Done Sent Done false
867 1006 Acked Write 10 Done Sent Done false
868 1007 Acked Write 10 Done Sent Done false
869 1008 Acked Write 7 Done Sent Done false
870 1009 Acked Write 9 Done Sent Done false
871 1010 Acked Write 1 Done Sent Done false
872 1011 Acked Read 4 Done Sent Done false
873 1012 Acked Read 4 Done Sent Done false
874 1013 Acked Read 7 Done Sent Done false
875 1014 Acked Read 9 Done Sent Done false
876 1015 Acked Read 7 Done Sent Done false
877 1016 Acked Read 3 Done Sent Done false
878 1017 Acked Flush 0 Done Sent Done false
879 1018 Acked Write 7 Done Sent Done false
880 1019 Acked Write 6 Done Sent Done false
881 1020 Acked Flush 0 Done Sent Done false
882 1021 Acked Read 4 Done Sent Done false
883 1022 Acked Flush 0 Done Sent Done false
884 1023 Acked Write 10 Done Sent Done false
885 1024 Acked Write 2 Done Sent Done false
886 1025 Acked Read 5 Done Sent Done false
887 1026 Acked Write 6 Done Sent Done false
888 1027 Acked Write 6 Done Sent Done false
889 1028 Acked Read 10 Done Sent Done false
890 1029 Acked Write 2 Done Sent Done false
891 1030 Acked Read 4 Done Sent Done false
892 1031 Acked Read 6 Done Sent Done false
893 1032 Acked Write 6 Done Sent Done false
894 1033 Acked Flush 0 Done Sent Done false
895 1034 Acked Read 1 Done Sent Done false
896 1035 Acked Read 10 Done Sent Done false
897 1036 Acked Read 8 Done Sent Done false
898 1037 Acked Read 7 Done Sent Done false
899 1038 Acked Write 2 Done Sent Done false
900 1039 Acked Read 10 Done Sent Done false
901 1040 Acked Read 5 Done Sent Done false
902 1041 Acked Write 2 Done Sent Done false
903 1042 Acked Write 4 Done Sent Done false
904 1043 Acked Write 2 Done Sent Done false
905 1044 Acked Flush 0 Done Sent Done false
906 1045 Acked Read 7 Done Sent Done false
907 1046 Acked Flush 0 Done Sent Done false
908 1047 Acked Read 1 Done Sent Done false
909 1048 Acked Flush 0 Done Sent Done false
910 1049 Acked Write 9 Done Sent Done false
911 1050 Acked Read 6 Done Sent Done false
912 1051 Acked Read 5 Done Sent Done false
913 1052 Acked Flush 0 Done Sent Done false
914 1053 Acked Read 7 Done Sent Done false
915 1054 Acked Write 9 Done Sent Done false
916 1055 Acked Write 6 Done Sent Done false
917 1056 Acked Flush 0 Done Sent Done false
918 1057 Acked Write 8 Done Sent Done false
919 1058 Acked Read 8 Done Sent Done false
920 1059 Acked Write 5 Done Sent Done false
921 1060 Acked Write 7 Done Sent Done false
922 1061 Acked Read 6 Done Sent Done false
923 1062 Acked Write 1 Done Sent Done false
924 1063 Acked Read 5 Done Sent Done false
925 1064 Acked Write 6 Done Sent Done false
926 1065 Acked Read 4 Done Sent Done false
927 1066 Acked Write 2 Done Sent Done false
928 1067 Acked Write 9 Done Sent Done false
929 1068 Acked Write 1 Done Sent Done false
930 1069 Acked Write 2 Done Sent Done false
931 1070 Acked Write 5 Done Sent Done false
932 1071 Acked Write 9 Done Sent Done false
933 1072 Acked Read 4 Done Sent Done false
934 1073 Acked Write 6 Done Sent Done false
935 1074 Acked Read 9 Done Sent Done false
936 1075 Acked Read 8 Done Sent Done false
937 1076 Acked Flush 0 Done Sent Done false
938 1077 Acked Read 5 Done Sent Done false
939 1078 Acked Read 2 Done Sent Done false
940 1079 Acked Read 3 Done Sent Done false
941 1080 Acked Read 5 Done Sent Done false
942 1081 Acked Write 4 Done Sent Done false
943 1082 Acked Write 7 Done Sent Done false
944 1083 Acked Write 3 Done Sent Done false
945 1084 Acked Write 7 Done Sent Done false
946 1085 Acked Read 6 Done Sent Done false
947 1086 Acked Write 1 Done Sent Done false
948 1087 Acked Read 6 Done Sent Done false
949 1088 Acked Write 5 Done Sent Done false
950 1089 Acked Write 1 Done Sent Done false
951 1090 Acked Read 9 Done Sent Done false
952 1091 Acked Read 8 Done Sent Done false
953 1092 Acked Write 7 Done Sent Done false
954 1093 Acked Read 3 Done Sent Done false
955 1094 Acked Write 7 Done Sent Done false
956 1095 Acked Read 2 Done Sent Done false
957 1096 Acked Write 7 Done Sent Done false
958 1097 Acked Write 6 Done Sent Done false
959 1098 Acked Read 9 Done Sent Done false
960 1099 Acked Write 2 Done Sent Done false
961 1100 Acked Write 5 Done Sent Done false
962 1101 Acked Read 5 Done Sent Done false
963 1102 Acked Read 9 Done Sent Done false
964 1103 Acked Write 8 Done Sent Done false
965 1104 Acked Write 5 Done Sent Done false
966 1105 Acked Write 2 Done Sent Done false
967 1106 Acked Read 2 Done Sent Done false
968 1107 Acked Flush 0 Done Sent Done false
969 1108 Acked Flush 0 Done Sent Done false
970 1109 Acked Write 1 Done Sent Done false
971 1110 Acked Read 10 Done Sent Done false
972 1111 Acked Flush 0 Done Sent Done false
973 1112 Acked Read 3 Done Sent Done false
974 1113 Acked Flush 0 Done Sent Done false
975 1114 Acked Read 2 Done Sent Done false
976 1115 Acked Flush 0 Done Sent Done false
977 1116 Acked Write 1 Done Sent Done false
978 1117 Acked Flush 0 Done Sent Done false
979 1118 Acked Write 5 Done Sent Done false
980 1119 Acked Read 5 Done Sent Done false
981 1120 Acked Write 5 Done Sent Done false
982 1121 Acked Flush 0 Done Sent Done false
983 1122 Acked Read 6 Done Sent Done false
984 1123 Acked Read 8 Done Sent Done false
985 1124 Acked Read 1 Done Sent Done false
986 1125 Acked Write 4 Done Sent Done false
987 1126 Acked Read 6 Done Sent Done false
988 1127 Acked Read 1 Done Sent Done false
989 1128 Acked Write 7 Done Sent Done false
990 1129 Acked Read 4 Done Sent Done false
991 1130 Acked Flush 0 Done Sent Done false
992 1131 Acked Read 10 Done Sent Done false
993 1132 Acked Write 8 Done Sent Done false
994 1133 Acked Write 5 Done Sent Done false
995 1134 Acked Read 2 Done Sent Done false
996 1135 Acked Read 5 Done Sent Done false
997 1136 Acked Read 1 Done Sent Done false
998 1137 Acked Write 3 Done Sent Done false
999 1138 Acked Read 8 Done Sent Done false
1000 1139 Acked Read 2 Done Sent Done false
1001 1140 Acked Write 7 Done Sent Done false
1002 1141 Acked Read 5 Done Sent Done false
1003 1142 Acked Read 5 Done Sent Done false
1004 1143 Acked Read 5 Done Sent Done false
1005 1144 Acked Write 1 Done Sent Done false
1006 1145 Acked Write 7 Done Sent Done false
1007 1146 Acked Read 8 Done Sent Done false
1008 1147 Acked Read 1 Done Sent Done false
1009 1148 Acked Read 6 Done Sent Done false
1010 1149 Acked Read 5 Done Sent Done false
1011 1150 Acked Read 6 Done Sent Done false
1012 1151 Acked Read 10 Done Sent Done false
1013 1152 Acked Read 5 Done Sent Done false
1014 1153 Acked Read 6 Done Sent Done false
1015 1154 Acked Flush 0 Done Sent Done false
1016 1155 Acked Flush 0 Done Sent Done false
1017 1156 Acked Flush 0 Done Sent Done false
1018 1157 Acked Read 8 Done Sent Done false
1019 1158 Acked Write 6 Done Sent Done false
1020 1159 Acked Write 7 Done Sent Done false
1021 1160 Acked Write 9 Done Sent Done false
1022 1161 Acked Read 8 Done Sent Done false
1023 1162 Acked Read 6 Done Sent Done false
1024 1163 Acked Read 4 Done Sent Done false
1025 1164 Acked Flush 0 Done Sent Done false
1026 1165 Acked Read 2 Done Sent Done false
1027 1166 Acked Flush 0 Done Sent Done false
1028 1167 Acked Flush 0 Done Sent Done false
1029 1168 Acked Read 6 Done Sent Done false
1030 1169 Acked Write 10 Done Sent Done false
1031 1170 Acked Write 1 Done Sent Done false
1032 1171 Acked Read 5 Done Sent Done false
1033 1172 Acked Write 1 Done Sent Done false
1034 1173 Acked Read 6 Done Sent Done false
1035 1174 Acked Write 10 Done Sent Done false
1036 1175 Acked Write 8 Done Sent Done false
1037 1176 Acked Read 9 Done Sent Done false
1038 1177 Acked Write 9 Done Sent Done false
1039 1178 Acked Read 4 Done Sent Done false
1040 1179 Acked Write 9 Done Sent Done false
1041 1180 Acked Read 9 Done Sent Done false
1042 1181 Acked Read 3 Done Sent Done false
1043 1182 Acked Flush 0 Done Sent Done false
1044 1183 Acked Write 1 Done Sent Done false
1045 1184 Acked Read 8 Done Sent Done false
1046 1185 Acked Flush 0 Done Sent Done false
1047 1186 Acked Write 7 Done Sent Done false
1048 1187 Acked Read 8 Done Sent Done false
1049 1188 Acked Write 7 Done Sent Done false
1050 1189 Acked Write 10 Done Sent Done false
1051 1190 Acked Write 4 Done Sent Done false
1052 1191 Acked Write 9 Done Sent Done false
1053 1192 Acked Write 3 Done Sent Done false
1054 1193 Acked Read 1 Done Sent Done false
1055 1194 Acked Read 9 Done Sent Done false
1056 1195 Acked Read 6 Done Sent Done false
1057 1196 Acked Read 2 Done Sent Done false
1058 1197 Acked Read 1 Done Sent Done false
1059 1198 Acked Read 1 Done Sent Done false
1060 1199 Acked Read 8 Done Sent Done false
1061 1200 Acked Flush 0 Done Sent Done false
1062 1201 Acked Write 10 Done Sent Done false
1063 1202 Acked Read 9 Done Sent Done false
1064 1203 Acked Write 8 Done Sent Done false
1065 1204 Acked Read 4 Done Sent Done false
1066 1205 Acked Read 9 Done Sent Done false
1067 1206 Acked Read 1 Done Sent Done false
1068 1207 Acked Read 10 Done Sent Done false
1069 1208 Acked Write 3 Done Sent Done false
1070 1209 Acked Write 10 Done Sent Done false
1071 1210 Acked Read 10 Done Sent Done false
1072 1211 Acked Write 7 Done Sent Done false
1073 1212 Acked Read 8 Done Sent Done false
1074 1213 Acked Write 4 Done Sent Done false
1075 1214 Acked Read 10 Done Sent Done false
1076 1215 Acked Write 1 Done Sent Done false
1077 1216 Acked Write 2 Done Sent Done false
1078 1217 Acked Write 5 Done Sent Done false
1079 1218 Acked Read 10 Done Sent Done false
1080 1219 Acked Write 8 Done Sent Done false
1081 1220 Acked Read 10 Done Sent Done false
1082 1221 Acked Write 4 Done Sent Done false
1083 1222 Acked Read 6 Done Sent Done false
1084 1223 Acked Read 8 Done Sent Done false
1085 1224 Acked Write 2 Done Sent Done false
1086 1225 Acked Write 8 Done Sent Done false
1087 1226 Acked Write 5 Done Sent Done false
1088 1227 Acked Write 4 Done Sent Done false
1089 1228 Acked Write 8 Done Sent Done false
1090 1229 Acked Write 10 Done Sent Done false
1091 1230 Acked Write 3 Done Sent Done false
1092 1231 Acked Read 1 Done Sent Done false
1093 1232 Acked Read 8 Done Sent Done false
1094 1233 Acked Read 3 Done Sent Done false
1095 1234 Acked Read 4 Done Sent Done false
1096 1235 Acked Write 3 Done Sent Done false
1097 1236 Acked Read 7 Done Sent Done false
1098 1237 Acked Write 1 Done Sent Done false
1099 1238 Acked Write 3 Done Sent Done false
1100 1239 Acked Read 3 Done Sent Done false
1101 1240 Acked Read 6 Done Sent Done false
1102 1241 Acked Read 4 Done Sent Done false
1103 1242 Acked Read 3 Done Sent Done false
1104 1243 Acked Write 10 Done Sent Done false
1105 1244 Acked Write 5 Done Sent Done false
1106 1245 Acked Flush 0 Done Sent Done false
1107 1246 Acked Write 1 Done Sent Done false
1108 1247 Acked Read 8 Done Sent Done false
1109 1248 Acked Read 6 Done Sent Done false
1110 1249 Acked Read 5 Done Sent Done false
1111 1250 Acked Read 3 Done Sent Done false
1112 1251 Acked Read 10 Done Sent Done false
1113 1252 Acked Read 1 Done Sent Done false
1114 1253 Acked Read 10 Done Sent Done false
1115 1254 Acked Read 3 Done Sent Done false
1116 1255 Acked Flush 0 Done Sent Done false
1117 1256 Acked Write 5 Done Sent Done false
1118 1257 Acked Read 8 Done Sent Done false
1119 1258 Acked Read 8 Done Sent Done false
1120 1259 Acked Read 10 Done Sent Done false
1121 1260 Acked Write 6 Done Sent Done false
1122 1261 Acked Write 2 Done Sent Done false
1123 1262 Acked Read 9 Done Sent Done false
1124 1263 Acked Read 10 Done Sent Done false
1125 1264 Acked Write 2 Done Sent Done false
1126 1265 Acked Write 9 Done Sent Done false
1127 1266 Acked Flush 0 Done Sent Done false
1128 1267 Acked Read 8 Done Sent Done false
1129 1268 Acked Read 6 Done Sent Done false
1130 1269 Acked Read 6 Done Sent Done false
1131 1270 Acked Write 9 Done Sent Done false
1132 1271 Acked Read 3 Done Sent Done false
1133 1272 Acked Flush 0 Done Sent Done false
1134 1273 Acked Read 1 Done Sent Done false
1135 1274 Acked Read 5 Done Sent Done false
1136 1275 Acked Write 4 Done Sent Done false
1137 1276 Acked Read 2 Done Sent Done false
1138 1277 Acked Read 2 Done Sent Done false
1139 1278 Acked Read 7 Done Sent Done false
1140 1279 Acked Read 9 Done Sent Done false
1141 1280 Acked Write 8 Done Sent Done false
1142 1281 Acked Write 1 Done Sent Done false
1143 1282 Acked Write 8 Done Sent Done false
1144 1283 Acked Read 10 Done Sent Done false
1145 1284 Acked Write 1 Done Sent Done false
1146 1285 Acked Write 1 Done Sent Done false
1147 1286 Acked Flush 0 Done Sent Done false
1148 1287 Acked Flush 0 Done Sent Done false
1149 1288 Acked Read 4 Done Sent Done false
1150 1289 Acked Read 10 Done Sent Done false
1151 1290 Acked Read 10 Done Sent Done false
1152 1291 Acked Write 2 Done Sent Done false
1153 1292 Acked Flush 0 Done Sent Done false
1154 1293 Acked Read 3 Done Sent Done false
1155 1294 Acked Write 1 Done Sent Done false
1156 1295 Acked Write 1 Done Sent Done false
1157 1296 Acked Flush 0 Done Sent Done false
1158 1297 Acked Read 5 Done Sent Done false
1159 1298 Acked Read 3 Done Sent Done false
1160 1299 Acked Write 6 Done Sent Done false
1161 1300 Acked Read 9 Done Sent Done false
1162 STATES DS:0 DS:1 DS:2 TOTAL
1163 Sent 0 301 0 301
1164 Done 301 0 301 602
1165 Skipped 0 0 0 0
1166 Error 0 0 0 0
1167 Last Flush: Some(JobId(1296)) None Some(JobId(1296))
1168 Downstairs last five completed:
1169 Upstairs last five completed: 1300 1299 1298 1297 1296
1170 CLIENT: Up:0 ds:301 act:2
11712025-03-17T22:34:41.588ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11722025-03-17T22:34:42.309ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11732025-03-17T22:34:42.836ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11742025-03-17T22:34:42.836ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53519) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11752025-03-17T22:34:42.837ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11762025-03-17T22:34:42.837ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11772025-03-17T22:34:42.837ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11782025-03-17T22:34:42.837ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11792025-03-17T22:34:42.837ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11802025-03-17T22:34:42.837ZINFOcrucible: [1] client re-new 301 jobs since flush None = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11812025-03-17T22:34:42.838ZINFOcrucible: [1] Replayed 301 jobs since flush: None = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1182 ----------------------------------------------------------------
1183 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1184 Downstairs last five completed: 1301 1300 1299 1298 1297
1185 Upstairs last five completed: 1301 1300 1299 1298 1297
1186 CLIENT: Up:0 ds:0 act:3
1187 Replay: All jobs finished
1188 Main task has work to do, go find it
1189 got dsc Stop(2)
1190 stop 2
1191 Mar 17 22:34:45.089 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/2, method: POST, req_id: b56cdaa4-6775-4c5f-a82d-884e691b4016, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
1192 [8830] Got stop action so:false kr:false
1193 Mar 17 22:34:45.089 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7de6b311-c97d-4a9a-a4b8-3ab8b846dfe7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
11942025-03-17T22:34:45.089ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11952025-03-17T22:34:45.089ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11962025-03-17T22:34:45.089ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11972025-03-17T22:34:45.090ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1198 [8830] Exited with: ExitStatus(unix_wait_status(9))
1199 [8830][2] reports Exit
1200 Mar 17 22:34:49.091 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/2, method: GET, req_id: fcffc3fd-62b3-48bc-979d-4b34eace5f9a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
1201 001/300 Write block 918 len 24576 data: 2 2 2 3 2 2
1202 002/300 Read block 4648 len 40960
1203 003/300 Read block 1528 len 36864
1204 004/300 Read block 4889 len 8192
1205 005/300 Write block 3896 len 28672 data: 3 3 2 2 2 2 2
1206 006/300 Read block 1609 len 4096
1207 007/300 Write block 390 len 20480 data: 2 2 2 2 2
1208 008/300 Read block 2489 len 28672
1209 009/300 Write block 677 len 16384 data: 2 2 2 2
1210 010/300 Write block 3910 len 36864 data: 3 3 3 2 2 2 2 2 2
1211 011/300 Write block 4966 len 16384 data: 2 2 2 2
1212 012/300 Write block 2132 len 32768 data: 2 2 2 2 2 2 2 2
1213 013/300 Read block 3096 len 40960
1214 014/300 Read block 3797 len 8192
1215 015/300 Read block 4515 len 32768
1216 016/300 Read block 4000 len 36864
1217 017/300 Read block 4803 len 20480
1218 018/300 Write block 3570 len 24576 data: 2 2 2 2 2 2
1219 019/300 Read block 4268 len 24576
1220 020/300 Read block 3409 len 24576
1221 021/300 Read block 1995 len 20480
1222 022/300 Read block 3213 len 24576
1223 023/300 Read block 254 len 16384
1224 024/300 Read block 28 len 28672
1225 025/300 Write block 3409 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1226 026/300 Read block 2095 len 40960
1227 027/300 Read block 3589 len 20480
1228 028/300 Write block 1907 len 32768 data: 2 2 2 2 2 2 2 2
1229 029/300 Read block 2572 len 40960
1230 030/300 Write block 1381 len 40960 data: 2 2 3 2 2 2 2 2 2 2
1231 031/300 Write block 3029 len 36864 data: 2 2 2 2 2 2 2 2 2
1232 032/300 Write block 1890 len 4096 data: 2
1233 033/300 Read block 3605 len 28672
1234 034/300 Read block 1275 len 28672
1235 035/300 Write block 250 len 20480 data: 2 2 2 2 3
1236 036/300 Write block 4940 len 12288 data: 2 2 2
1237 037/300 Read block 1817 len 40960
1238 038/300 Write block 3047 len 32768 data: 2 2 2 2 3 3 3 3
1239 039/300 Flush
1240 040/300 Read block 3329 len 36864
1241 041/300 Write block 4339 len 8192 data: 2 2
1242 042/300 Flush
1243 043/300 Write block 2921 len 16384 data: 2 2 2 2
1244 044/300 Write block 3421 len 32768 data: 2 2 2 2 2 2 2 2
1245 045/300 Write block 2608 len 32768 data: 2 2 2 2 2 2 2 2
1246 046/300 Flush
1247 047/300 Read block 4634 len 36864
1248 048/300 Write block 2099 len 8192 data: 2 2
1249 049/300 Read block 611 len 12288
1250 050/300 Read block 3347 len 36864
1251 051/300 Write block 675 len 12288 data: 2 2 3
1252 052/300 Write block 812 len 40960 data: 2 2 2 2 2 2 3 3 2 2
1253 053/300 Write block 3882 len 4096 data: 2
1254 054/300 Flush
1255 055/300 Read block 167 len 8192
1256 056/300 Write block 4688 len 16384 data: 2 2 2 2
1257 057/300 Read block 3230 len 36864
1258 058/300 Read block 2979 len 4096
1259 059/300 Read block 610 len 28672
1260 060/300 Read block 3682 len 36864
1261 061/300 Read block 2720 len 40960
1262 062/300 Write block 929 len 4096 data: 2
1263 063/300 Write block 3123 len 36864 data: 2 2 2 2 2 2 2 2 2
1264 064/300 Write block 2102 len 32768 data: 2 2 2 2 2 2 2 2
1265 065/300 Write block 4634 len 16384 data: 3 3 3 3
1266 066/300 Write block 1939 len 32768 data: 2 2 2 2 2 2 2 2
1267 067/300 Flush
1268 068/300 Read block 518 len 4096
1269 069/300 Read block 943 len 28672
1270 070/300 Write block 865 len 24576 data: 2 2 2 2 2 2
1271 071/300 Read block 2752 len 12288
1272 072/300 Write block 178 len 20480 data: 2 2 2 2 2
1273 073/300 Read block 3266 len 4096
1274 074/300 Read block 1544 len 12288
1275 075/300 Flush
1276 076/300 Write block 2951 len 32768 data: 2 2 2 2 2 2 2 2
1277 077/300 Read block 1240 len 8192
1278 078/300 Read block 3718 len 40960
1279 079/300 Write block 1236 len 4096 data: 2
1280 080/300 Write block 4943 len 8192 data: 2 2
1281 081/300 Write block 83 len 4096 data: 2
1282 082/300 Write block 448 len 16384 data: 3 3 3 2
1283 083/300 Flush
1284 084/300 Read block 1170 len 8192
1285 085/300 Read block 3672 len 36864
1286 086/300 Read block 4383 len 36864
1287 087/300 Read block 3509 len 12288
1288 088/300 Write block 3810 len 20480 data: 3 3 3 3 2
1289 089/300 Flush
1290 090/300 Read block 2920 len 20480
1291 091/300 Write block 4595 len 16384 data: 2 2 2 2
1292 092/300 Read block 1558 len 40960
1293 093/300 Write block 408 len 12288 data: 3 3 3
1294 094/300 Write block 813 len 20480 data: 3 3 3 3 3
1295 095/300 Write block 1923 len 20480 data: 2 2 2 2 2
1296 096/300 Write block 3832 len 8192 data: 2 2
1297 097/300 Flush
1298 098/300 Read block 1676 len 8192
1299 099/300 Flush
1300 100/300 Write block 1765 len 32768 data: 2 2 2 2 2 2 2 2
1301 101/300 Write block 2465 len 24576 data: 2 2 2 2 2 2
1302 102/300 Read block 4034 len 8192
1303 103/300 Read block 4249 len 36864
1304 104/300 Read block 1228 len 20480
1305 105/300 Read block 4871 len 24576
1306 106/300 Write block 4072 len 16384 data: 2 2 2 2
1307 107/300 Read block 4081 len 28672
1308 108/300 Flush
1309 109/300 Write block 3891 len 4096 data: 2
1310 110/300 Read block 473 len 16384
1311 111/300 Flush
1312 112/300 Flush
1313 113/300 Read block 3381 len 20480
1314 114/300 Write block 1087 len 16384 data: 3 3 2 3
1315 115/300 Write block 1296 len 8192 data: 3 3
1316 116/300 Read block 606 len 28672
1317 117/300 Write block 1468 len 36864 data: 2 2 2 2 2 2 2 2 2
1318 118/300 Write block 1497 len 20480 data: 2 2 2 2 2
1319 119/300 Read block 4779 len 40960
1320 120/300 Flush
1321 121/300 Write block 1375 len 16384 data: 2 2 2 2
1322 122/300 Flush
1323 123/300 Read block 3659 len 36864
1324 124/300 Read block 3488 len 28672
1325 125/300 Write block 1826 len 36864 data: 2 2 2 3 2 2 2 2 2
1326 126/300 Write block 297 len 4096 data: 2
1327 127/300 Write block 2764 len 28672 data: 2 2 2 2 2 2 2
1328 128/300 Read block 3626 len 20480
1329 129/300 Read block 2436 len 32768
1330 130/300 Read block 1107 len 28672
1331 131/300 Write block 4367 len 8192 data: 2 2
1332 132/300 Read block 777 len 40960
1333 133/300 Read block 4832 len 8192
1334 134/300 Flush
1335 135/300 Flush
1336 136/300 Write block 35 len 4096 data: 2
1337 137/300 Write block 3318 len 32768 data: 2 2 2 2 2 2 2 2
1338 138/300 Read block 4389 len 24576
1339 139/300 Read block 2876 len 4096
1340 140/300 Read block 1245 len 36864
1341 141/300 Read block 1616 len 36864
1342 142/300 Write block 3669 len 8192 data: 2 2
1343 143/300 Write block 3758 len 12288 data: 2 2 2
1344 144/300 Read block 3995 len 32768
1345 145/300 Read block 193 len 8192
1346 146/300 Read block 1121 len 20480
1347 147/300 Read block 3738 len 24576
1348 148/300 Write block 3597 len 24576 data: 2 2 2 2 2 2
1349 149/300 Write block 741 len 40960 data: 2 2 2 2 2 2 2 3 2 2
1350 150/300 Read block 3142 len 20480
1351 151/300 Write block 4354 len 28672 data: 3 2 2 2 2 2 2
1352 152/300 Read block 4179 len 32768
1353 153/300 Read block 3154 len 24576
1354 154/300 Read block 1845 len 4096
1355 155/300 Read block 3399 len 4096
1356 156/300 Write block 4792 len 28672 data: 2 2 2 2 2 2 2
1357 157/300 Read block 2220 len 36864
1358 158/300 Read block 3152 len 36864
1359 159/300 Flush
1360 160/300 Read block 1726 len 28672
1361 161/300 Read block 4752 len 24576
1362 162/300 Read block 4130 len 16384
1363 163/300 Read block 3571 len 16384
1364 164/300 Write block 3308 len 16384 data: 2 2 2 2
1365 165/300 Read block 4131 len 32768
1366 166/300 Write block 4904 len 8192 data: 2 2
1367 167/300 Flush
1368 168/300 Write block 3261 len 8192 data: 2 2
1369 169/300 Read block 692 len 28672
1370 170/300 Write block 2162 len 20480 data: 3 3 3 3 3
1371 171/300 Read block 3032 len 20480
1372 172/300 Read block 2287 len 8192
1373 173/300 Read block 4480 len 8192
1374 174/300 Read block 2836 len 32768
1375 175/300 Write block 2877 len 12288 data: 2 2 2
1376 176/300 Read block 567 len 32768
1377 177/300 Read block 2956 len 8192
1378 178/300 Flush
1379 179/300 Write block 3264 len 24576 data: 2 2 2 2 2 2
1380 180/300 Read block 2686 len 32768
1381 181/300 Write block 2267 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1382 182/300 Write block 3901 len 12288 data: 3 3 2
1383 183/300 Read block 3175 len 32768
1384 184/300 Read block 858 len 12288
1385 185/300 Write block 1835 len 12288 data: 2 2 2
1386 186/300 Read block 4938 len 12288
1387 187/300 Flush
1388 188/300 Read block 4409 len 12288
1389 189/300 Read block 3104 len 8192
1390 190/300 Write block 869 len 16384 data: 3 3 2 2
1391 191/300 Write block 3240 len 8192 data: 2 2
1392 192/300 Write block 1545 len 16384 data: 2 2 2 2
1393 193/300 Read block 1522 len 4096
1394 194/300 Read block 2922 len 40960
1395 195/300 Read block 3872 len 16384
1396 196/300 Write block 253 len 24576 data: 3 4 3 2 2 2
1397 197/300 Flush
1398 198/300 Write block 4775 len 36864 data: 3 3 3 3 3 2 2 2 2
1399 199/300 Read block 4369 len 40960
1400 200/300 Read block 1199 len 8192
1401 201/300 Read block 2546 len 32768
1402 202/300 Write block 3315 len 32768 data: 2 2 2 3 3 3 3 3
1403 203/300 Write block 4113 len 16384 data: 2 2 2 2
1404 204/300 Read block 13 len 8192
1405 205/300 Write block 102 len 20480 data: 2 2 2 2 2
1406 206/300 Write block 2814 len 16384 data: 2 2 2 2
1407 207/300 Read block 3947 len 28672
1408 208/300 Flush
1409 209/300 Flush
1410 210/300 Write block 1862 len 24576 data: 2 2 2 2 3 2
1411 211/300 Read block 732 len 32768
1412 212/300 Flush
1413 213/300 Read block 4348 len 20480
1414 214/300 Read block 1044 len 32768
1415 215/300 Write block 1100 len 20480 data: 2 2 2 2 2
1416 216/300 Read block 4012 len 32768
1417 217/300 Read block 3893 len 28672
1418 218/300 Flush
1419 219/300 Write block 1904 len 12288 data: 2 2 2
1420 220/300 Write block 3820 len 24576 data: 2 2 2 2 2 2
1421 221/300 Write block 4009 len 16384 data: 2 2 2 2
1422 222/300 Read block 1956 len 8192
1423 223/300 Flush
1424 224/300 Write block 3874 len 36864 data: 2 2 2 2 2 2 2 2 3
1425 225/300 Write block 3570 len 20480 data: 3 3 3 3 3
1426 226/300 Write block 3075 len 20480 data: 3 2 2 2 2
1427 227/300 Write block 4160 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1428 228/300 Write block 881 len 24576 data: 2 2 2 3 2 2
1429 229/300 Write block 517 len 16384 data: 2 2 3 3
1430 230/300 Read block 2201 len 32768
1431 231/300 Flush
1432 232/300 Write block 2400 len 20480 data: 2 2 2 2 2
1433 233/300 Write block 810 len 36864 data: 2 2 3 4 4 4 4 4 4
1434 234/300 Read block 3484 len 28672
1435 235/300 Write block 3960 len 8192 data: 2 2
1436 236/300 Write block 3936 len 32768 data: 2 2 2 2 2 2 2 2
1437 237/300 Read block 4874 len 28672
1438 238/300 Flush
1439 239/300 Read block 2835 len 12288
1440 240/300 Write block 981 len 16384 data: 2 2 2 2
1441 241/300 Read block 309 len 32768
1442 242/300 Read block 829 len 20480
1443 243/300 Read block 1668 len 16384
1444 244/300 Write block 1193 len 32768 data: 2 2 2 2 2 2 2 2
1445 245/300 Read block 4848 len 24576
1446 246/300 Write block 2909 len 20480 data: 2 2 3 3 3
1447 247/300 Write block 2436 len 24576 data: 2 2 2 2 2 2
1448 248/300 Read block 2979 len 28672
1449 249/300 Read block 19 len 16384
1450 250/300 Write block 2788 len 20480 data: 3 3 3 3 3
1451 251/300 Write block 3012 len 8192 data: 2 2
1452 252/300 Write block 1784 len 24576 data: 3 3 3 3 2 2
1453 253/300 Write block 1644 len 12288 data: 2 2 2
1454 254/300 Write block 3161 len 36864 data: 2 2 2 2 2 2 2 2 2
1455 255/300 Read block 1408 len 12288
1456 256/300 Read block 419 len 20480
1457 257/300 Write block 2592 len 36864 data: 2 2 2 2 2 2 2 2 2
1458 258/300 Read block 3205 len 40960
1459 259/300 Write block 53 len 20480 data: 2 2 2 2 2
1460 260/300 Read block 929 len 12288
1461 261/300 Write block 1442 len 8192 data: 2 2
1462 262/300 Flush
1463 263/300 Read block 1681 len 32768
1464 264/300 Write block 1411 len 32768 data: 2 2 2 2 2 2 2 2
1465 265/300 Read block 3040 len 20480
1466 266/300 Read block 84 len 28672
1467 267/300 Write block 3621 len 16384 data: 2 2 2 2
1468 268/300 Flush
1469 269/300 Read block 3508 len 40960
1470 270/300 Read block 1476 len 24576
1471 271/300 Read block 3848 len 40960
1472 272/300 Write block 4222 len 4096 data: 2
1473 273/300 Write block 3845 len 16384 data: 2 2 2 2
1474 274/300 Write block 4647 len 20480 data: 2 2 2 2 2
1475 275/300 Write block 2265 len 20480 data: 2 2 3 3 3
1476 276/300 Read block 2946 len 24576
1477 277/300 Write block 4260 len 20480 data: 2 2 2 2 2
1478 278/300 Read block 2648 len 28672
1479 279/300 Read block 1945 len 32768
1480 280/300 Write block 2929 len 8192 data: 2 2
1481 281/300 Write block 2027 len 4096 data: 3
1482 282/300 Read block 4084 len 16384
1483 283/300 Read block 564 len 28672
1484 284/300 Read block 4735 len 8192
1485 285/300 Read block 3110 len 16384
1486 286/300 Write block 2534 len 32768 data: 2 2 2 2 2 2 2 2
1487 287/300 Flush
1488 288/300 Read block 2313 len 24576
1489 289/300 Read block 3266 len 12288
1490 290/300 Read block 4680 len 28672
1491 291/300 Read block 2791 len 4096
1492 292/300 Read block 4097 len 8192
1493 293/300 Write block 1474 len 40960 data: 3 3 3 2 2 2 2 2 2 2
1494 294/300 Write block 1392 len 24576 data: 2 2 2 2 2 2
1495 295/300 Read block 711 len 8192
1496 296/300 Read block 2453 len 4096
1497 297/300 Read block 2436 len 8192
1498 298/300 Write block 4520 len 8192 data: 2 2
1499 299/300 Read block 3646 len 4096
1500 300/300 Read block 94 len 20480
1501 301/300 Read block 3353 len 40960
1502 Main task has work to do, go find it
1503 got dsc Start(2)
1504 start 2
1505 Mar 17 22:34:49.333 INFO request completed, latency_us: 303, response_code: 204, uri: /start/cid/2, method: POST, req_id: f72fbf72-1bcf-4df2-a9c0-13faae0ac3ff, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
1506 Replay: started 2, returned:Ok(())
1507 ----------------------------------------------------------------
1508 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1509 States: Active Active Offline
1510 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1511 1302 Acked Write 6 Done Done Sent false
1512 1303 Acked Read 10 Done Done Sent false
1513 1304 Acked Read 9 Done Done Sent false
1514 1305 Acked Read 2 Done Done Sent false
1515 1306 Acked Write 7 Done Done Sent false
1516 1307 Acked Read 1 Done Done Sent false
1517 1308 Acked Write 5 Done Done Sent false
1518 1309 Acked Read 7 Done Done Sent false
1519 1310 Acked Write 4 Done Done Sent false
1520 1311 Acked Write 9 Done Done Sent false
1521 1312 Acked Write 4 Done Done Sent false
1522 1313 Acked Write 8 Done Done Sent false
1523 1314 Acked Read 10 Done Done Sent false
1524 1315 Acked Read 2 Done Done Sent false
1525 1316 Acked Read 8 Done Done Sent false
1526 1317 Acked Read 9 Done Done Sent false
1527 1318 Acked Read 5 Done Done Sent false
1528 1319 Acked Write 6 Done Done Sent false
1529 1320 Acked Read 6 Done Done Sent false
1530 1321 Acked Read 6 Done Done Sent false
1531 1322 Acked Read 5 Done Done Sent false
1532 1323 Acked Read 6 Done Done Sent false
1533 1324 Acked Read 4 Done Done Sent false
1534 1325 Acked Read 7 Done Done Sent false
1535 1326 Acked Write 10 Done Done Sent false
1536 1327 Acked Read 10 Done Done Sent false
1537 1328 Acked Read 5 Done Done Sent false
1538 1329 Acked Write 8 Done Done Sent false
1539 1330 Acked Read 10 Done Done Sent false
1540 1331 Acked Write 10 Done Done Sent false
1541 1332 Acked Write 9 Done Done Sent false
1542 1333 Acked Write 1 Done Done Sent false
1543 1334 Acked Read 7 Done Done Sent false
1544 1335 Acked Read 7 Done Done Sent false
1545 1336 Acked Write 5 Done Done Sent false
1546 1337 Acked Write 3 Done Done Sent false
1547 1338 Acked Read 10 Done Done Sent false
1548 1339 Acked Write 8 Done Done Sent false
1549 1340 Acked Flush 0 Done Done Sent false
1550 1341 Acked Read 9 Done Done Sent false
1551 1342 Acked Write 2 Done Done Sent false
1552 1343 Acked Flush 0 Done Done Sent false
1553 1344 Acked Write 4 Done Done Sent false
1554 1345 Acked Write 8 Done Done Sent false
1555 1346 Acked Write 8 Done Done Sent false
1556 1347 Acked Flush 0 Done Done Sent false
1557 1348 Acked Read 9 Done Done Sent false
1558 1349 Acked Write 2 Done Done Sent false
1559 1350 Acked Read 3 Done Done Sent false
1560 1351 Acked Read 9 Done Done Sent false
1561 1352 Acked Write 3 Done Done Sent false
1562 1353 Acked Write 10 Done Done Sent false
1563 1354 Acked Write 1 Done Done Sent false
1564 1355 Acked Flush 0 Done Done Sent false
1565 1356 Acked Read 2 Done Done Sent false
1566 1357 Acked Write 4 Done Done Sent false
1567 1358 Acked Read 9 Done Done Sent false
1568 1359 Acked Read 1 Done Done Sent false
1569 1360 Acked Read 7 Done Done Sent false
1570 1361 Acked Read 9 Done Done Sent false
1571 1362 Acked Read 10 Done Done Sent false
1572 1363 Acked Write 1 Done Done Sent false
1573 1364 Acked Write 9 Done Done Sent false
1574 1365 Acked Write 8 Done Done Sent false
1575 1366 Acked Write 4 Done Done Sent false
1576 1367 Acked Write 8 Done Done Sent false
1577 1368 Acked Flush 0 Done Done Sent false
1578 1369 Acked Read 1 Done Done Sent false
1579 1370 Acked Read 7 Done Done Sent false
1580 1371 Acked Write 6 Done Done Sent false
1581 1372 Acked Read 3 Done Done Sent false
1582 1373 Acked Write 5 Done Done Sent false
1583 1374 Acked Read 1 Done Done Sent false
1584 1375 Acked Read 3 Done Done Sent false
1585 1376 Acked Flush 0 Done Done Sent false
1586 1377 Acked Write 8 Done Done Sent false
1587 1378 Acked Read 2 Done Done Sent false
1588 1379 Acked Read 10 Done Done Sent false
1589 1380 Acked Write 1 Done Done Sent false
1590 1381 Acked Write 2 Done Done Sent false
1591 1382 Acked Write 1 Done Done Sent false
1592 1383 Acked Write 4 Done Done Sent false
1593 1384 Acked Flush 0 Done Done Sent false
1594 1385 Acked Read 2 Done Done Sent false
1595 1386 Acked Read 9 Done Done Sent false
1596 1387 Acked Read 9 Done Done Sent false
1597 1388 Acked Read 3 Done Done Sent false
1598 1389 Acked Write 5 Done Done Sent false
1599 1390 Acked Flush 0 Done Done Sent false
1600 1391 Acked Read 5 Done Done Sent false
1601 1392 Acked Write 4 Done Done Sent false
1602 1393 Acked Read 10 Done Done Sent false
1603 1394 Acked Write 3 Done Done Sent false
1604 1395 Acked Write 5 Done Done Sent false
1605 1396 Acked Write 5 Done Done Sent false
1606 1397 Acked Write 2 Done Done Sent false
1607 1398 Acked Flush 0 Done Done Sent false
1608 1399 Acked Read 2 Done Done Sent false
1609 1400 Acked Flush 0 Done Done Sent false
1610 1401 Acked Write 8 Done Done Sent false
1611 1402 Acked Write 6 Done Done Sent false
1612 1403 Acked Read 2 Done Done Sent false
1613 1404 Acked Read 9 Done Done Sent false
1614 1405 Acked Read 5 Done Done Sent false
1615 1406 Acked Read 6 Done Done Sent false
1616 1407 Acked Write 4 Done Done Sent false
1617 1408 Acked Read 7 Done Done Sent false
1618 1409 Acked Flush 0 Done Done Sent false
1619 1410 Acked Write 1 Done Done Sent false
1620 1411 Acked Read 4 Done Done Sent false
1621 1412 Acked Flush 0 Done Done Sent false
1622 1413 Acked Flush 0 Done Done Sent false
1623 1414 Acked Read 5 Done Done Sent false
1624 1415 Acked Write 4 Done Done Sent false
1625 1416 Acked Write 2 Done Done Sent false
1626 1417 Acked Read 7 Done Done Sent false
1627 1418 Acked Write 9 Done Done Sent false
1628 1419 Acked Write 5 Done Done Sent false
1629 1420 Acked Read 10 Done Done Sent false
1630 1421 Acked Flush 0 Done Done Sent false
1631 1422 Acked Write 4 Done Done Sent false
1632 1423 Acked Flush 0 Done Done Sent false
1633 1424 Acked Read 9 Done Done Sent false
1634 1425 Acked Read 7 Done Done Sent false
1635 1426 Acked Write 9 Done Done Sent false
1636 1427 Acked Write 1 Done Done Sent false
1637 1428 Acked Write 7 Done Done Sent false
1638 1429 Acked Read 5 Done Done Sent false
1639 1430 Acked Read 8 Done Done Sent false
1640 1431 Acked Read 7 Done Done Sent false
1641 1432 Acked Write 2 Done Done Sent false
1642 1433 Acked Read 10 Done Done Sent false
1643 1434 Acked Read 2 Done Done Sent false
1644 1435 Acked Flush 0 Done Done Sent false
1645 1436 Acked Flush 0 Done Done Sent false
1646 1437 Acked Write 1 Done Done Sent false
1647 1438 Acked Write 8 Done Done Sent false
1648 1439 Acked Read 6 Done Done Sent false
1649 1440 Acked Read 1 Done Done Sent false
1650 1441 Acked Read 9 Done Done Sent false
1651 1442 Acked Read 9 Done Done Sent false
1652 1443 Acked Write 2 Done Done Sent false
1653 1444 Acked Write 3 Done Done Sent false
1654 1445 Acked Read 8 Done Done Sent false
1655 1446 Acked Read 2 Done Done Sent false
1656 1447 Acked Read 5 Done Done Sent false
1657 1448 Acked Read 6 Done Done Sent false
1658 1449 Acked Write 6 Done Done Sent false
1659 1450 Acked Write 10 Done Done Sent false
1660 1451 Acked Read 5 Done Done Sent false
1661 1452 Acked Write 7 Done Done Sent false
1662 1453 Acked Read 8 Done Done Sent false
1663 1454 Acked Read 6 Done Done Sent false
1664 1455 Acked Read 1 Done Done Sent false
1665 1456 Acked Read 1 Done Done Sent false
1666 1457 Acked Write 7 Done Done Sent false
1667 1458 Acked Read 9 Done Done Sent false
1668 1459 Acked Read 9 Done Done Sent false
1669 1460 Acked Flush 0 Done Done Sent false
1670 1461 Acked Read 7 Done Done Sent false
1671 1462 Acked Read 6 Done Done Sent false
1672 1463 Acked Read 4 Done Done Sent false
1673 1464 Acked Read 4 Done Done Sent false
1674 1465 Acked Write 4 Done Done Sent false
1675 1466 Acked Read 8 Done Done Sent false
1676 1467 Acked Write 2 Done Done Sent false
1677 1468 Acked Flush 0 Done Done Sent false
1678 1469 Acked Write 2 Done Done Sent false
1679 1470 Acked Read 7 Done Done Sent false
1680 1471 Acked Write 5 Done Done Sent false
1681 1472 Acked Read 5 Done Done Sent false
1682 1473 Acked Read 2 Done Done Sent false
1683 1474 Acked Read 2 Done Done Sent false
1684 1475 Acked Read 8 Done Done Sent false
1685 1476 Acked Write 3 Done Done Sent false
1686 1477 Acked Read 8 Done Done Sent false
1687 1478 Acked Read 2 Done Done Sent false
1688 1479 Acked Flush 0 Done Done Sent false
1689 1480 Acked Write 6 Done Done Sent false
1690 1481 Acked Read 8 Done Done Sent false
1691 1482 Acked Write 10 Done Done Sent false
1692 1483 Acked Write 3 Done Done Sent false
1693 1484 Acked Read 8 Done Done Sent false
1694 1485 Acked Read 3 Done Done Sent false
1695 1486 Acked Write 3 Done Done Sent false
1696 1487 Acked Read 3 Done Done Sent false
1697 1488 Acked Flush 0 Done Done Sent false
1698 1489 Acked Read 3 Done Done Sent false
1699 1490 Acked Read 2 Done Done Sent false
1700 1491 Acked Write 4 Done Done Sent false
1701 1492 Acked Write 2 Done Done Sent false
1702 1493 Acked Write 4 Done Done Sent false
1703 1494 Acked Read 1 Done Done Sent false
1704 1495 Acked Read 10 Done Done Sent false
1705 1496 Acked Read 4 Done Done Sent false
1706 1497 Acked Write 6 Done Done Sent false
1707 1498 Acked Flush 0 Done Done Sent false
1708 1499 Acked Write 9 Done Done Sent false
1709 1500 Acked Read 10 Done Done Sent false
1710 1501 Acked Read 2 Done Done Sent false
1711 1502 Acked Read 8 Done Done Sent false
1712 1503 Acked Write 8 Done Done Sent false
1713 1504 Acked Write 4 Done Done Sent false
1714 1505 Acked Read 2 Done Done Sent false
1715 1506 Acked Write 5 Done Done Sent false
1716 1507 Acked Write 4 Done Done Sent false
1717 1508 Acked Read 7 Done Done Sent false
1718 1509 Acked Flush 0 Done Done Sent false
1719 1510 Acked Flush 0 Done Done Sent false
1720 1511 Acked Write 6 Done Done Sent false
1721 1512 Acked Read 8 Done Done Sent false
1722 1513 Acked Flush 0 Done Done Sent false
1723 1514 Acked Read 5 Done Done Sent false
1724 1515 Acked Read 8 Done Done Sent false
1725 1516 Acked Write 5 Done Done Sent false
1726 1517 Acked Read 8 Done Done Sent false
1727 1518 Acked Read 7 Done Done Sent false
1728 1519 Acked Flush 0 Done Done Sent false
1729 1520 Acked Write 3 Done Done Sent false
1730 1521 Acked Write 6 Done Done Sent false
1731 1522 Acked Write 4 Done Done Sent false
1732 1523 Acked Read 2 Done Done Sent false
1733 1524 Acked Flush 0 Done Done Sent false
1734 1525 Acked Write 9 Done Done Sent false
1735 1526 Acked Write 5 Done Done Sent false
1736 1527 Acked Write 5 Done Done Sent false
1737 1528 Acked Write 10 Done Done Sent false
1738 1529 Acked Write 6 Done Done Sent false
1739 1530 Acked Write 4 Done Done Sent false
1740 1531 Acked Read 8 Done Done Sent false
1741 1532 Acked Flush 0 Done Done Sent false
1742 1533 Acked Write 5 Done Done Sent false
1743 1534 Acked Write 9 Done Done Sent false
1744 1535 Acked Read 7 Done Done Sent false
1745 1536 Acked Write 2 Done Done Sent false
1746 1537 Acked Write 8 Done Done Sent false
1747 1538 Acked Read 7 Done Done Sent false
1748 1539 Acked Flush 0 Done Done Sent false
1749 1540 Acked Read 3 Done Done Sent false
1750 1541 Acked Write 4 Done Done Sent false
1751 1542 Acked Read 8 Done Done Sent false
1752 1543 Acked Read 5 Done Done Sent false
1753 1544 Acked Read 4 Done Done Sent false
1754 1545 Acked Write 8 Done Done Sent false
1755 1546 Acked Read 6 Done Done Sent false
1756 1547 Acked Write 5 Done Done Sent false
1757 1548 Acked Write 6 Done Done Sent false
1758 1549 Acked Read 7 Done Done Sent false
1759 1550 Acked Read 4 Done Done Sent false
1760 1551 Acked Write 5 Done Done Sent false
1761 1552 Acked Write 2 Done Done Sent false
1762 1553 Acked Write 6 Done Done Sent false
1763 1554 Acked Write 3 Done Done Sent false
1764 1555 Acked Write 9 Done Done Sent false
1765 1556 Acked Read 3 Done Done Sent false
1766 1557 Acked Read 5 Done Done Sent false
1767 1558 Acked Write 9 Done Done Sent false
1768 1559 Acked Read 10 Done Done Sent false
1769 1560 Acked Write 5 Done Done Sent false
1770 1561 Acked Read 3 Done Done Sent false
1771 1562 Acked Write 2 Done Done Sent false
1772 1563 Acked Flush 0 Done Done Sent false
1773 1564 Acked Read 8 Done Done Sent false
1774 1565 Acked Write 8 Done Done Sent false
1775 1566 Acked Read 5 Done Done Sent false
1776 1567 Acked Read 7 Done Done Sent false
1777 1568 Acked Write 4 Done Done Sent false
1778 1569 Acked Flush 0 Done Done Sent false
1779 1570 Acked Read 10 Done Done Sent false
1780 1571 Acked Read 6 Done Done Sent false
1781 1572 Acked Read 10 Done Done Sent false
1782 1573 Acked Write 1 Done Done Sent false
1783 1574 Acked Write 4 Done Done Sent false
1784 1575 Acked Write 5 Done Done Sent false
1785 1576 Acked Write 5 Done Done Sent false
1786 1577 Acked Read 6 Done Done Sent false
1787 1578 Acked Write 5 Done Done Sent false
1788 1579 Acked Read 7 Done Done Sent false
1789 1580 Acked Read 8 Done Done Sent false
1790 1581 Acked Write 2 Done Done Sent false
1791 1582 Acked Write 1 Done Done Sent false
1792 1583 Acked Read 4 Done Done Sent false
1793 1584 Acked Read 7 Done Done Sent false
1794 1585 Acked Read 2 Done Done Sent false
1795 1586 Acked Read 4 Done Done Sent false
1796 1587 Acked Write 8 Done Done Sent false
1797 1588 Acked Flush 0 Done Done Sent false
1798 1589 Acked Read 6 Done Done Sent false
1799 1590 Acked Read 3 Done Done Sent false
1800 1591 Acked Read 7 Done Done Sent false
1801 1592 Acked Read 1 Done Done Sent false
1802 1593 Acked Read 2 Done Done Sent false
1803 1594 Acked Write 10 Done Done Sent false
1804 1595 Acked Write 6 Done Done Sent false
1805 1596 Acked Read 2 Done Done Sent false
1806 1597 Acked Read 1 Done Done Sent false
1807 1598 Acked Read 2 Done Done Sent false
1808 1599 Acked Write 2 Done Done Sent false
1809 1600 Acked Read 1 Done Done Sent false
1810 1601 Acked Read 5 Done Done Sent false
1811 1602 Acked Read 10 Done Done Sent false
1812 STATES DS:0 DS:1 DS:2 TOTAL
1813 Sent 0 0 301 301
1814 Done 301 301 0 602
1815 Skipped 0 0 0 0
1816 Error 0 0 0 0
1817 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
1818 Downstairs last five completed: 1301 1300 1299 1298 1297
1819 Upstairs last five completed: 1602 1601 1600 1599 1598
1820 CLIENT: Up:0 ds:301 act:2
18212025-03-17T22:34:50.322ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18222025-03-17T22:34:51.324ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18232025-03-17T22:34:52.326ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
1824 ----------------------------------------------------------------
1825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1826 States: Active Active Offline
1827 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1828 1302 Acked Write 6 Done Done Sent false
1829 1303 Acked Read 10 Done Done Sent false
1830 1304 Acked Read 9 Done Done Sent false
1831 1305 Acked Read 2 Done Done Sent false
1832 1306 Acked Write 7 Done Done Sent false
1833 1307 Acked Read 1 Done Done Sent false
1834 1308 Acked Write 5 Done Done Sent false
1835 1309 Acked Read 7 Done Done Sent false
1836 1310 Acked Write 4 Done Done Sent false
1837 1311 Acked Write 9 Done Done Sent false
1838 1312 Acked Write 4 Done Done Sent false
1839 1313 Acked Write 8 Done Done Sent false
1840 1314 Acked Read 10 Done Done Sent false
1841 1315 Acked Read 2 Done Done Sent false
1842 1316 Acked Read 8 Done Done Sent false
1843 1317 Acked Read 9 Done Done Sent false
1844 1318 Acked Read 5 Done Done Sent false
1845 1319 Acked Write 6 Done Done Sent false
1846 1320 Acked Read 6 Done Done Sent false
1847 1321 Acked Read 6 Done Done Sent false
1848 1322 Acked Read 5 Done Done Sent false
1849 1323 Acked Read 6 Done Done Sent false
1850 1324 Acked Read 4 Done Done Sent false
1851 1325 Acked Read 7 Done Done Sent false
1852 1326 Acked Write 10 Done Done Sent false
1853 1327 Acked Read 10 Done Done Sent false
1854 1328 Acked Read 5 Done Done Sent false
1855 1329 Acked Write 8 Done Done Sent false
1856 1330 Acked Read 10 Done Done Sent false
1857 1331 Acked Write 10 Done Done Sent false
1858 1332 Acked Write 9 Done Done Sent false
1859 1333 Acked Write 1 Done Done Sent false
1860 1334 Acked Read 7 Done Done Sent false
1861 1335 Acked Read 7 Done Done Sent false
1862 1336 Acked Write 5 Done Done Sent false
1863 1337 Acked Write 3 Done Done Sent false
1864 1338 Acked Read 10 Done Done Sent false
1865 1339 Acked Write 8 Done Done Sent false
1866 1340 Acked Flush 0 Done Done Sent false
1867 1341 Acked Read 9 Done Done Sent false
1868 1342 Acked Write 2 Done Done Sent false
1869 1343 Acked Flush 0 Done Done Sent false
1870 1344 Acked Write 4 Done Done Sent false
1871 1345 Acked Write 8 Done Done Sent false
1872 1346 Acked Write 8 Done Done Sent false
1873 1347 Acked Flush 0 Done Done Sent false
1874 1348 Acked Read 9 Done Done Sent false
1875 1349 Acked Write 2 Done Done Sent false
1876 1350 Acked Read 3 Done Done Sent false
1877 1351 Acked Read 9 Done Done Sent false
1878 1352 Acked Write 3 Done Done Sent false
1879 1353 Acked Write 10 Done Done Sent false
1880 1354 Acked Write 1 Done Done Sent false
1881 1355 Acked Flush 0 Done Done Sent false
1882 1356 Acked Read 2 Done Done Sent false
1883 1357 Acked Write 4 Done Done Sent false
1884 1358 Acked Read 9 Done Done Sent false
1885 1359 Acked Read 1 Done Done Sent false
1886 1360 Acked Read 7 Done Done Sent false
1887 1361 Acked Read 9 Done Done Sent false
1888 1362 Acked Read 10 Done Done Sent false
1889 1363 Acked Write 1 Done Done Sent false
1890 1364 Acked Write 9 Done Done Sent false
1891 1365 Acked Write 8 Done Done Sent false
1892 1366 Acked Write 4 Done Done Sent false
1893 1367 Acked Write 8 Done Done Sent false
1894 1368 Acked Flush 0 Done Done Sent false
1895 1369 Acked Read 1 Done Done Sent false
1896 1370 Acked Read 7 Done Done Sent false
1897 1371 Acked Write 6 Done Done Sent false
1898 1372 Acked Read 3 Done Done Sent false
1899 1373 Acked Write 5 Done Done Sent false
1900 1374 Acked Read 1 Done Done Sent false
1901 1375 Acked Read 3 Done Done Sent false
1902 1376 Acked Flush 0 Done Done Sent false
1903 1377 Acked Write 8 Done Done Sent false
1904 1378 Acked Read 2 Done Done Sent false
1905 1379 Acked Read 10 Done Done Sent false
1906 1380 Acked Write 1 Done Done Sent false
1907 1381 Acked Write 2 Done Done Sent false
1908 1382 Acked Write 1 Done Done Sent false
1909 1383 Acked Write 4 Done Done Sent false
1910 1384 Acked Flush 0 Done Done Sent false
1911 1385 Acked Read 2 Done Done Sent false
1912 1386 Acked Read 9 Done Done Sent false
1913 1387 Acked Read 9 Done Done Sent false
1914 1388 Acked Read 3 Done Done Sent false
1915 1389 Acked Write 5 Done Done Sent false
1916 1390 Acked Flush 0 Done Done Sent false
1917 1391 Acked Read 5 Done Done Sent false
1918 1392 Acked Write 4 Done Done Sent false
1919 1393 Acked Read 10 Done Done Sent false
1920 1394 Acked Write 3 Done Done Sent false
1921 1395 Acked Write 5 Done Done Sent false
1922 1396 Acked Write 5 Done Done Sent false
1923 1397 Acked Write 2 Done Done Sent false
1924 1398 Acked Flush 0 Done Done Sent false
1925 1399 Acked Read 2 Done Done Sent false
1926 1400 Acked Flush 0 Done Done Sent false
1927 1401 Acked Write 8 Done Done Sent false
1928 1402 Acked Write 6 Done Done Sent false
1929 1403 Acked Read 2 Done Done Sent false
1930 1404 Acked Read 9 Done Done Sent false
1931 1405 Acked Read 5 Done Done Sent false
1932 1406 Acked Read 6 Done Done Sent false
1933 1407 Acked Write 4 Done Done Sent false
1934 1408 Acked Read 7 Done Done Sent false
1935 1409 Acked Flush 0 Done Done Sent false
1936 1410 Acked Write 1 Done Done Sent false
1937 1411 Acked Read 4 Done Done Sent false
1938 1412 Acked Flush 0 Done Done Sent false
1939 1413 Acked Flush 0 Done Done Sent false
1940 1414 Acked Read 5 Done Done Sent false
1941 1415 Acked Write 4 Done Done Sent false
1942 1416 Acked Write 2 Done Done Sent false
1943 1417 Acked Read 7 Done Done Sent false
1944 1418 Acked Write 9 Done Done Sent false
1945 1419 Acked Write 5 Done Done Sent false
1946 1420 Acked Read 10 Done Done Sent false
1947 1421 Acked Flush 0 Done Done Sent false
1948 1422 Acked Write 4 Done Done Sent false
1949 1423 Acked Flush 0 Done Done Sent false
1950 1424 Acked Read 9 Done Done Sent false
1951 1425 Acked Read 7 Done Done Sent false
1952 1426 Acked Write 9 Done Done Sent false
1953 1427 Acked Write 1 Done Done Sent false
1954 1428 Acked Write 7 Done Done Sent false
1955 1429 Acked Read 5 Done Done Sent false
1956 1430 Acked Read 8 Done Done Sent false
1957 1431 Acked Read 7 Done Done Sent false
1958 1432 Acked Write 2 Done Done Sent false
1959 1433 Acked Read 10 Done Done Sent false
1960 1434 Acked Read 2 Done Done Sent false
1961 1435 Acked Flush 0 Done Done Sent false
1962 1436 Acked Flush 0 Done Done Sent false
1963 1437 Acked Write 1 Done Done Sent false
1964 1438 Acked Write 8 Done Done Sent false
1965 1439 Acked Read 6 Done Done Sent false
1966 1440 Acked Read 1 Done Done Sent false
1967 1441 Acked Read 9 Done Done Sent false
1968 1442 Acked Read 9 Done Done Sent false
1969 1443 Acked Write 2 Done Done Sent false
1970 1444 Acked Write 3 Done Done Sent false
1971 1445 Acked Read 8 Done Done Sent false
1972 1446 Acked Read 2 Done Done Sent false
1973 1447 Acked Read 5 Done Done Sent false
1974 1448 Acked Read 6 Done Done Sent false
1975 1449 Acked Write 6 Done Done Sent false
1976 1450 Acked Write 10 Done Done Sent false
1977 1451 Acked Read 5 Done Done Sent false
1978 1452 Acked Write 7 Done Done Sent false
1979 1453 Acked Read 8 Done Done Sent false
1980 1454 Acked Read 6 Done Done Sent false
1981 1455 Acked Read 1 Done Done Sent false
1982 1456 Acked Read 1 Done Done Sent false
1983 1457 Acked Write 7 Done Done Sent false
1984 1458 Acked Read 9 Done Done Sent false
1985 1459 Acked Read 9 Done Done Sent false
1986 1460 Acked Flush 0 Done Done Sent false
1987 1461 Acked Read 7 Done Done Sent false
1988 1462 Acked Read 6 Done Done Sent false
1989 1463 Acked Read 4 Done Done Sent false
1990 1464 Acked Read 4 Done Done Sent false
1991 1465 Acked Write 4 Done Done Sent false
1992 1466 Acked Read 8 Done Done Sent false
1993 1467 Acked Write 2 Done Done Sent false
1994 1468 Acked Flush 0 Done Done Sent false
1995 1469 Acked Write 2 Done Done Sent false
1996 1470 Acked Read 7 Done Done Sent false
1997 1471 Acked Write 5 Done Done Sent false
1998 1472 Acked Read 5 Done Done Sent false
1999 1473 Acked Read 2 Done Done Sent false
2000 1474 Acked Read 2 Done Done Sent false
2001 1475 Acked Read 8 Done Done Sent false
2002 1476 Acked Write 3 Done Done Sent false
2003 1477 Acked Read 8 Done Done Sent false
2004 1478 Acked Read 2 Done Done Sent false
2005 1479 Acked Flush 0 Done Done Sent false
2006 1480 Acked Write 6 Done Done Sent false
2007 1481 Acked Read 8 Done Done Sent false
2008 1482 Acked Write 10 Done Done Sent false
2009 1483 Acked Write 3 Done Done Sent false
2010 1484 Acked Read 8 Done Done Sent false
2011 1485 Acked Read 3 Done Done Sent false
2012 1486 Acked Write 3 Done Done Sent false
2013 1487 Acked Read 3 Done Done Sent false
2014 1488 Acked Flush 0 Done Done Sent false
2015 1489 Acked Read 3 Done Done Sent false
2016 1490 Acked Read 2 Done Done Sent false
2017 1491 Acked Write 4 Done Done Sent false
2018 1492 Acked Write 2 Done Done Sent false
2019 1493 Acked Write 4 Done Done Sent false
2020 1494 Acked Read 1 Done Done Sent false
2021 1495 Acked Read 10 Done Done Sent false
2022 1496 Acked Read 4 Done Done Sent false
2023 1497 Acked Write 6 Done Done Sent false
2024 1498 Acked Flush 0 Done Done Sent false
2025 1499 Acked Write 9 Done Done Sent false
2026 1500 Acked Read 10 Done Done Sent false
2027 1501 Acked Read 2 Done Done Sent false
2028 1502 Acked Read 8 Done Done Sent false
2029 1503 Acked Write 8 Done Done Sent false
2030 1504 Acked Write 4 Done Done Sent false
2031 1505 Acked Read 2 Done Done Sent false
2032 1506 Acked Write 5 Done Done Sent false
2033 1507 Acked Write 4 Done Done Sent false
2034 1508 Acked Read 7 Done Done Sent false
2035 1509 Acked Flush 0 Done Done Sent false
2036 1510 Acked Flush 0 Done Done Sent false
2037 1511 Acked Write 6 Done Done Sent false
2038 1512 Acked Read 8 Done Done Sent false
2039 1513 Acked Flush 0 Done Done Sent false
2040 1514 Acked Read 5 Done Done Sent false
2041 1515 Acked Read 8 Done Done Sent false
2042 1516 Acked Write 5 Done Done Sent false
2043 1517 Acked Read 8 Done Done Sent false
2044 1518 Acked Read 7 Done Done Sent false
2045 1519 Acked Flush 0 Done Done Sent false
2046 1520 Acked Write 3 Done Done Sent false
2047 1521 Acked Write 6 Done Done Sent false
2048 1522 Acked Write 4 Done Done Sent false
2049 1523 Acked Read 2 Done Done Sent false
2050 1524 Acked Flush 0 Done Done Sent false
2051 1525 Acked Write 9 Done Done Sent false
2052 1526 Acked Write 5 Done Done Sent false
2053 1527 Acked Write 5 Done Done Sent false
2054 1528 Acked Write 10 Done Done Sent false
2055 1529 Acked Write 6 Done Done Sent false
2056 1530 Acked Write 4 Done Done Sent false
2057 1531 Acked Read 8 Done Done Sent false
2058 1532 Acked Flush 0 Done Done Sent false
2059 1533 Acked Write 5 Done Done Sent false
2060 1534 Acked Write 9 Done Done Sent false
2061 1535 Acked Read 7 Done Done Sent false
2062 1536 Acked Write 2 Done Done Sent false
2063 1537 Acked Write 8 Done Done Sent false
2064 1538 Acked Read 7 Done Done Sent false
2065 1539 Acked Flush 0 Done Done Sent false
2066 1540 Acked Read 3 Done Done Sent false
2067 1541 Acked Write 4 Done Done Sent false
2068 1542 Acked Read 8 Done Done Sent false
2069 1543 Acked Read 5 Done Done Sent false
2070 1544 Acked Read 4 Done Done Sent false
2071 1545 Acked Write 8 Done Done Sent false
2072 1546 Acked Read 6 Done Done Sent false
2073 1547 Acked Write 5 Done Done Sent false
2074 1548 Acked Write 6 Done Done Sent false
2075 1549 Acked Read 7 Done Done Sent false
2076 1550 Acked Read 4 Done Done Sent false
2077 1551 Acked Write 5 Done Done Sent false
2078 1552 Acked Write 2 Done Done Sent false
2079 1553 Acked Write 6 Done Done Sent false
2080 1554 Acked Write 3 Done Done Sent false
2081 1555 Acked Write 9 Done Done Sent false
2082 1556 Acked Read 3 Done Done Sent false
2083 1557 Acked Read 5 Done Done Sent false
2084 1558 Acked Write 9 Done Done Sent false
2085 1559 Acked Read 10 Done Done Sent false
2086 1560 Acked Write 5 Done Done Sent false
2087 1561 Acked Read 3 Done Done Sent false
2088 1562 Acked Write 2 Done Done Sent false
2089 1563 Acked Flush 0 Done Done Sent false
2090 1564 Acked Read 8 Done Done Sent false
2091 1565 Acked Write 8 Done Done Sent false
2092 1566 Acked Read 5 Done Done Sent false
2093 1567 Acked Read 7 Done Done Sent false
2094 1568 Acked Write 4 Done Done Sent false
2095 1569 Acked Flush 0 Done Done Sent false
2096 1570 Acked Read 10 Done Done Sent false
2097 1571 Acked Read 6 Done Done Sent false
2098 1572 Acked Read 10 Done Done Sent false
2099 1573 Acked Write 1 Done Done Sent false
2100 1574 Acked Write 4 Done Done Sent false
2101 1575 Acked Write 5 Done Done Sent false
2102 1576 Acked Write 5 Done Done Sent false
2103 1577 Acked Read 6 Done Done Sent false
2104 1578 Acked Write 5 Done Done Sent false
2105 1579 Acked Read 7 Done Done Sent false
2106 1580 Acked Read 8 Done Done Sent false
2107 1581 Acked Write 2 Done Done Sent false
2108 1582 Acked Write 1 Done Done Sent false
2109 1583 Acked Read 4 Done Done Sent false
2110 1584 Acked Read 7 Done Done Sent false
2111 1585 Acked Read 2 Done Done Sent false
2112 1586 Acked Read 4 Done Done Sent false
2113 1587 Acked Write 8 Done Done Sent false
2114 1588 Acked Flush 0 Done Done Sent false
2115 1589 Acked Read 6 Done Done Sent false
2116 1590 Acked Read 3 Done Done Sent false
2117 1591 Acked Read 7 Done Done Sent false
2118 1592 Acked Read 1 Done Done Sent false
2119 1593 Acked Read 2 Done Done Sent false
2120 1594 Acked Write 10 Done Done Sent false
2121 1595 Acked Write 6 Done Done Sent false
2122 1596 Acked Read 2 Done Done Sent false
2123 1597 Acked Read 1 Done Done Sent false
2124 1598 Acked Read 2 Done Done Sent false
2125 1599 Acked Write 2 Done Done Sent false
2126 1600 Acked Read 1 Done Done Sent false
2127 1601 Acked Read 5 Done Done Sent false
2128 1602 Acked Read 10 Done Done Sent false
2129 STATES DS:0 DS:1 DS:2 TOTAL
2130 Sent 0 0 301 301
2131 Done 301 301 0 602
2132 Skipped 0 0 0 0
2133 Error 0 0 0 0
2134 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2135 Downstairs last five completed: 1301 1300 1299 1298 1297
2136 Upstairs last five completed: 1602 1601 1600 1599 1598
2137 CLIENT: Up:0 ds:301 act:2
21382025-03-17T22:34:54.330ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21392025-03-17T22:34:55.091ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21402025-03-17T22:34:55.091ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21412025-03-17T22:34:55.091ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21422025-03-17T22:34:55.091ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21432025-03-17T22:34:55.092ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21442025-03-17T22:34:55.593ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21452025-03-17T22:34:56.334ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21462025-03-17T22:34:57.336ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2147 ----------------------------------------------------------------
2148 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2149 States: Active Active Offline
2150 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2151 1302 Acked Write 6 Done Done Sent false
2152 1303 Acked Read 10 Done Done Sent false
2153 1304 Acked Read 9 Done Done Sent false
2154 1305 Acked Read 2 Done Done Sent false
2155 1306 Acked Write 7 Done Done Sent false
2156 1307 Acked Read 1 Done Done Sent false
2157 1308 Acked Write 5 Done Done Sent false
2158 1309 Acked Read 7 Done Done Sent false
2159 1310 Acked Write 4 Done Done Sent false
2160 1311 Acked Write 9 Done Done Sent false
2161 1312 Acked Write 4 Done Done Sent false
2162 1313 Acked Write 8 Done Done Sent false
2163 1314 Acked Read 10 Done Done Sent false
2164 1315 Acked Read 2 Done Done Sent false
2165 1316 Acked Read 8 Done Done Sent false
2166 1317 Acked Read 9 Done Done Sent false
2167 1318 Acked Read 5 Done Done Sent false
2168 1319 Acked Write 6 Done Done Sent false
2169 1320 Acked Read 6 Done Done Sent false
2170 1321 Acked Read 6 Done Done Sent false
2171 1322 Acked Read 5 Done Done Sent false
2172 1323 Acked Read 6 Done Done Sent false
2173 1324 Acked Read 4 Done Done Sent false
2174 1325 Acked Read 7 Done Done Sent false
2175 1326 Acked Write 10 Done Done Sent false
2176 1327 Acked Read 10 Done Done Sent false
2177 1328 Acked Read 5 Done Done Sent false
2178 1329 Acked Write 8 Done Done Sent false
2179 1330 Acked Read 10 Done Done Sent false
2180 1331 Acked Write 10 Done Done Sent false
2181 1332 Acked Write 9 Done Done Sent false
2182 1333 Acked Write 1 Done Done Sent false
2183 1334 Acked Read 7 Done Done Sent false
2184 1335 Acked Read 7 Done Done Sent false
2185 1336 Acked Write 5 Done Done Sent false
2186 1337 Acked Write 3 Done Done Sent false
2187 1338 Acked Read 10 Done Done Sent false
2188 1339 Acked Write 8 Done Done Sent false
2189 1340 Acked Flush 0 Done Done Sent false
2190 1341 Acked Read 9 Done Done Sent false
2191 1342 Acked Write 2 Done Done Sent false
2192 1343 Acked Flush 0 Done Done Sent false
2193 1344 Acked Write 4 Done Done Sent false
2194 1345 Acked Write 8 Done Done Sent false
2195 1346 Acked Write 8 Done Done Sent false
2196 1347 Acked Flush 0 Done Done Sent false
2197 1348 Acked Read 9 Done Done Sent false
2198 1349 Acked Write 2 Done Done Sent false
2199 1350 Acked Read 3 Done Done Sent false
2200 1351 Acked Read 9 Done Done Sent false
2201 1352 Acked Write 3 Done Done Sent false
2202 1353 Acked Write 10 Done Done Sent false
2203 1354 Acked Write 1 Done Done Sent false
2204 1355 Acked Flush 0 Done Done Sent false
2205 1356 Acked Read 2 Done Done Sent false
2206 1357 Acked Write 4 Done Done Sent false
2207 1358 Acked Read 9 Done Done Sent false
2208 1359 Acked Read 1 Done Done Sent false
2209 1360 Acked Read 7 Done Done Sent false
2210 1361 Acked Read 9 Done Done Sent false
2211 1362 Acked Read 10 Done Done Sent false
2212 1363 Acked Write 1 Done Done Sent false
2213 1364 Acked Write 9 Done Done Sent false
2214 1365 Acked Write 8 Done Done Sent false
2215 1366 Acked Write 4 Done Done Sent false
2216 1367 Acked Write 8 Done Done Sent false
2217 1368 Acked Flush 0 Done Done Sent false
2218 1369 Acked Read 1 Done Done Sent false
2219 1370 Acked Read 7 Done Done Sent false
2220 1371 Acked Write 6 Done Done Sent false
2221 1372 Acked Read 3 Done Done Sent false
2222 1373 Acked Write 5 Done Done Sent false
2223 1374 Acked Read 1 Done Done Sent false
2224 1375 Acked Read 3 Done Done Sent false
2225 1376 Acked Flush 0 Done Done Sent false
2226 1377 Acked Write 8 Done Done Sent false
2227 1378 Acked Read 2 Done Done Sent false
2228 1379 Acked Read 10 Done Done Sent false
2229 1380 Acked Write 1 Done Done Sent false
2230 1381 Acked Write 2 Done Done Sent false
2231 1382 Acked Write 1 Done Done Sent false
2232 1383 Acked Write 4 Done Done Sent false
2233 1384 Acked Flush 0 Done Done Sent false
2234 1385 Acked Read 2 Done Done Sent false
2235 1386 Acked Read 9 Done Done Sent false
2236 1387 Acked Read 9 Done Done Sent false
2237 1388 Acked Read 3 Done Done Sent false
2238 1389 Acked Write 5 Done Done Sent false
2239 1390 Acked Flush 0 Done Done Sent false
2240 1391 Acked Read 5 Done Done Sent false
2241 1392 Acked Write 4 Done Done Sent false
2242 1393 Acked Read 10 Done Done Sent false
2243 1394 Acked Write 3 Done Done Sent false
2244 1395 Acked Write 5 Done Done Sent false
2245 1396 Acked Write 5 Done Done Sent false
2246 1397 Acked Write 2 Done Done Sent false
2247 1398 Acked Flush 0 Done Done Sent false
2248 1399 Acked Read 2 Done Done Sent false
2249 1400 Acked Flush 0 Done Done Sent false
2250 1401 Acked Write 8 Done Done Sent false
2251 1402 Acked Write 6 Done Done Sent false
2252 1403 Acked Read 2 Done Done Sent false
2253 1404 Acked Read 9 Done Done Sent false
2254 1405 Acked Read 5 Done Done Sent false
2255 1406 Acked Read 6 Done Done Sent false
2256 1407 Acked Write 4 Done Done Sent false
2257 1408 Acked Read 7 Done Done Sent false
2258 1409 Acked Flush 0 Done Done Sent false
2259 1410 Acked Write 1 Done Done Sent false
2260 1411 Acked Read 4 Done Done Sent false
2261 1412 Acked Flush 0 Done Done Sent false
2262 1413 Acked Flush 0 Done Done Sent false
2263 1414 Acked Read 5 Done Done Sent false
2264 1415 Acked Write 4 Done Done Sent false
2265 1416 Acked Write 2 Done Done Sent false
2266 1417 Acked Read 7 Done Done Sent false
2267 1418 Acked Write 9 Done Done Sent false
2268 1419 Acked Write 5 Done Done Sent false
2269 1420 Acked Read 10 Done Done Sent false
2270 1421 Acked Flush 0 Done Done Sent false
2271 1422 Acked Write 4 Done Done Sent false
2272 1423 Acked Flush 0 Done Done Sent false
2273 1424 Acked Read 9 Done Done Sent false
2274 1425 Acked Read 7 Done Done Sent false
2275 1426 Acked Write 9 Done Done Sent false
2276 1427 Acked Write 1 Done Done Sent false
2277 1428 Acked Write 7 Done Done Sent false
2278 1429 Acked Read 5 Done Done Sent false
2279 1430 Acked Read 8 Done Done Sent false
2280 1431 Acked Read 7 Done Done Sent false
2281 1432 Acked Write 2 Done Done Sent false
2282 1433 Acked Read 10 Done Done Sent false
2283 1434 Acked Read 2 Done Done Sent false
2284 1435 Acked Flush 0 Done Done Sent false
2285 1436 Acked Flush 0 Done Done Sent false
2286 1437 Acked Write 1 Done Done Sent false
2287 1438 Acked Write 8 Done Done Sent false
2288 1439 Acked Read 6 Done Done Sent false
2289 1440 Acked Read 1 Done Done Sent false
2290 1441 Acked Read 9 Done Done Sent false
2291 1442 Acked Read 9 Done Done Sent false
2292 1443 Acked Write 2 Done Done Sent false
2293 1444 Acked Write 3 Done Done Sent false
2294 1445 Acked Read 8 Done Done Sent false
2295 1446 Acked Read 2 Done Done Sent false
2296 1447 Acked Read 5 Done Done Sent false
2297 1448 Acked Read 6 Done Done Sent false
2298 1449 Acked Write 6 Done Done Sent false
2299 1450 Acked Write 10 Done Done Sent false
2300 1451 Acked Read 5 Done Done Sent false
2301 1452 Acked Write 7 Done Done Sent false
2302 1453 Acked Read 8 Done Done Sent false
2303 1454 Acked Read 6 Done Done Sent false
2304 1455 Acked Read 1 Done Done Sent false
2305 1456 Acked Read 1 Done Done Sent false
2306 1457 Acked Write 7 Done Done Sent false
2307 1458 Acked Read 9 Done Done Sent false
2308 1459 Acked Read 9 Done Done Sent false
2309 1460 Acked Flush 0 Done Done Sent false
2310 1461 Acked Read 7 Done Done Sent false
2311 1462 Acked Read 6 Done Done Sent false
2312 1463 Acked Read 4 Done Done Sent false
2313 1464 Acked Read 4 Done Done Sent false
2314 1465 Acked Write 4 Done Done Sent false
2315 1466 Acked Read 8 Done Done Sent false
2316 1467 Acked Write 2 Done Done Sent false
2317 1468 Acked Flush 0 Done Done Sent false
2318 1469 Acked Write 2 Done Done Sent false
2319 1470 Acked Read 7 Done Done Sent false
2320 1471 Acked Write 5 Done Done Sent false
2321 1472 Acked Read 5 Done Done Sent false
2322 1473 Acked Read 2 Done Done Sent false
2323 1474 Acked Read 2 Done Done Sent false
2324 1475 Acked Read 8 Done Done Sent false
2325 1476 Acked Write 3 Done Done Sent false
2326 1477 Acked Read 8 Done Done Sent false
2327 1478 Acked Read 2 Done Done Sent false
2328 1479 Acked Flush 0 Done Done Sent false
2329 1480 Acked Write 6 Done Done Sent false
2330 1481 Acked Read 8 Done Done Sent false
2331 1482 Acked Write 10 Done Done Sent false
2332 1483 Acked Write 3 Done Done Sent false
2333 1484 Acked Read 8 Done Done Sent false
2334 1485 Acked Read 3 Done Done Sent false
2335 1486 Acked Write 3 Done Done Sent false
2336 1487 Acked Read 3 Done Done Sent false
2337 1488 Acked Flush 0 Done Done Sent false
2338 1489 Acked Read 3 Done Done Sent false
2339 1490 Acked Read 2 Done Done Sent false
2340 1491 Acked Write 4 Done Done Sent false
2341 1492 Acked Write 2 Done Done Sent false
2342 1493 Acked Write 4 Done Done Sent false
2343 1494 Acked Read 1 Done Done Sent false
2344 1495 Acked Read 10 Done Done Sent false
2345 1496 Acked Read 4 Done Done Sent false
2346 1497 Acked Write 6 Done Done Sent false
2347 1498 Acked Flush 0 Done Done Sent false
2348 1499 Acked Write 9 Done Done Sent false
2349 1500 Acked Read 10 Done Done Sent false
2350 1501 Acked Read 2 Done Done Sent false
2351 1502 Acked Read 8 Done Done Sent false
2352 1503 Acked Write 8 Done Done Sent false
2353 1504 Acked Write 4 Done Done Sent false
2354 1505 Acked Read 2 Done Done Sent false
2355 1506 Acked Write 5 Done Done Sent false
2356 1507 Acked Write 4 Done Done Sent false
2357 1508 Acked Read 7 Done Done Sent false
2358 1509 Acked Flush 0 Done Done Sent false
2359 1510 Acked Flush 0 Done Done Sent false
2360 1511 Acked Write 6 Done Done Sent false
2361 1512 Acked Read 8 Done Done Sent false
2362 1513 Acked Flush 0 Done Done Sent false
2363 1514 Acked Read 5 Done Done Sent false
2364 1515 Acked Read 8 Done Done Sent false
2365 1516 Acked Write 5 Done Done Sent false
2366 1517 Acked Read 8 Done Done Sent false
2367 1518 Acked Read 7 Done Done Sent false
2368 1519 Acked Flush 0 Done Done Sent false
2369 1520 Acked Write 3 Done Done Sent false
2370 1521 Acked Write 6 Done Done Sent false
2371 1522 Acked Write 4 Done Done Sent false
2372 1523 Acked Read 2 Done Done Sent false
2373 1524 Acked Flush 0 Done Done Sent false
2374 1525 Acked Write 9 Done Done Sent false
2375 1526 Acked Write 5 Done Done Sent false
2376 1527 Acked Write 5 Done Done Sent false
2377 1528 Acked Write 10 Done Done Sent false
2378 1529 Acked Write 6 Done Done Sent false
2379 1530 Acked Write 4 Done Done Sent false
2380 1531 Acked Read 8 Done Done Sent false
2381 1532 Acked Flush 0 Done Done Sent false
2382 1533 Acked Write 5 Done Done Sent false
2383 1534 Acked Write 9 Done Done Sent false
2384 1535 Acked Read 7 Done Done Sent false
2385 1536 Acked Write 2 Done Done Sent false
2386 1537 Acked Write 8 Done Done Sent false
2387 1538 Acked Read 7 Done Done Sent false
2388 1539 Acked Flush 0 Done Done Sent false
2389 1540 Acked Read 3 Done Done Sent false
2390 1541 Acked Write 4 Done Done Sent false
2391 1542 Acked Read 8 Done Done Sent false
2392 1543 Acked Read 5 Done Done Sent false
2393 1544 Acked Read 4 Done Done Sent false
2394 1545 Acked Write 8 Done Done Sent false
2395 1546 Acked Read 6 Done Done Sent false
2396 1547 Acked Write 5 Done Done Sent false
2397 1548 Acked Write 6 Done Done Sent false
2398 1549 Acked Read 7 Done Done Sent false
2399 1550 Acked Read 4 Done Done Sent false
2400 1551 Acked Write 5 Done Done Sent false
2401 1552 Acked Write 2 Done Done Sent false
2402 1553 Acked Write 6 Done Done Sent false
2403 1554 Acked Write 3 Done Done Sent false
2404 1555 Acked Write 9 Done Done Sent false
2405 1556 Acked Read 3 Done Done Sent false
2406 1557 Acked Read 5 Done Done Sent false
2407 1558 Acked Write 9 Done Done Sent false
2408 1559 Acked Read 10 Done Done Sent false
2409 1560 Acked Write 5 Done Done Sent false
2410 1561 Acked Read 3 Done Done Sent false
2411 1562 Acked Write 2 Done Done Sent false
2412 1563 Acked Flush 0 Done Done Sent false
2413 1564 Acked Read 8 Done Done Sent false
2414 1565 Acked Write 8 Done Done Sent false
2415 1566 Acked Read 5 Done Done Sent false
2416 1567 Acked Read 7 Done Done Sent false
2417 1568 Acked Write 4 Done Done Sent false
2418 1569 Acked Flush 0 Done Done Sent false
2419 1570 Acked Read 10 Done Done Sent false
2420 1571 Acked Read 6 Done Done Sent false
2421 1572 Acked Read 10 Done Done Sent false
2422 1573 Acked Write 1 Done Done Sent false
2423 1574 Acked Write 4 Done Done Sent false
2424 1575 Acked Write 5 Done Done Sent false
2425 1576 Acked Write 5 Done Done Sent false
2426 1577 Acked Read 6 Done Done Sent false
2427 1578 Acked Write 5 Done Done Sent false
2428 1579 Acked Read 7 Done Done Sent false
2429 1580 Acked Read 8 Done Done Sent false
2430 1581 Acked Write 2 Done Done Sent false
2431 1582 Acked Write 1 Done Done Sent false
2432 1583 Acked Read 4 Done Done Sent false
2433 1584 Acked Read 7 Done Done Sent false
2434 1585 Acked Read 2 Done Done Sent false
2435 1586 Acked Read 4 Done Done Sent false
2436 1587 Acked Write 8 Done Done Sent false
2437 1588 Acked Flush 0 Done Done Sent false
2438 1589 Acked Read 6 Done Done Sent false
2439 1590 Acked Read 3 Done Done Sent false
2440 1591 Acked Read 7 Done Done Sent false
2441 1592 Acked Read 1 Done Done Sent false
2442 1593 Acked Read 2 Done Done Sent false
2443 1594 Acked Write 10 Done Done Sent false
2444 1595 Acked Write 6 Done Done Sent false
2445 1596 Acked Read 2 Done Done Sent false
2446 1597 Acked Read 1 Done Done Sent false
2447 1598 Acked Read 2 Done Done Sent false
2448 1599 Acked Write 2 Done Done Sent false
2449 1600 Acked Read 1 Done Done Sent false
2450 1601 Acked Read 5 Done Done Sent false
2451 1602 Acked Read 10 Done Done Sent false
2452 STATES DS:0 DS:1 DS:2 TOTAL
2453 Sent 0 0 301 301
2454 Done 301 301 0 602
2455 Skipped 0 0 0 0
2456 Error 0 0 0 0
2457 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2458 Downstairs last five completed: 1301 1300 1299 1298 1297
2459 Upstairs last five completed: 1602 1601 1600 1599 1598
2460 CLIENT: Up:0 ds:301 act:2
24612025-03-17T22:34:58.340ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24622025-03-17T22:34:59.341ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24632025-03-17T22:35:00.342ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2464 [8830] got start action
2465 [8830] I am going to restart
2466 Starting downstairs at port 8830
2467 Make output file at "/tmp/dsc/downstairs-8830.txt"
2468 [8830][2] reports Starting
2469 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1210)
2470 [8830][2] reports Running
2471 ----------------------------------------------------------------
2472 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2473 States: Active Active Offline
2474 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2475 1302 Acked Write 6 Done Done Sent false
2476 1303 Acked Read 10 Done Done Sent false
2477 1304 Acked Read 9 Done Done Sent false
2478 1305 Acked Read 2 Done Done Sent false
2479 1306 Acked Write 7 Done Done Sent false
2480 1307 Acked Read 1 Done Done Sent false
2481 1308 Acked Write 5 Done Done Sent false
2482 1309 Acked Read 7 Done Done Sent false
2483 1310 Acked Write 4 Done Done Sent false
2484 1311 Acked Write 9 Done Done Sent false
2485 1312 Acked Write 4 Done Done Sent false
2486 1313 Acked Write 8 Done Done Sent false
2487 1314 Acked Read 10 Done Done Sent false
2488 1315 Acked Read 2 Done Done Sent false
2489 1316 Acked Read 8 Done Done Sent false
2490 1317 Acked Read 9 Done Done Sent false
2491 1318 Acked Read 5 Done Done Sent false
2492 1319 Acked Write 6 Done Done Sent false
2493 1320 Acked Read 6 Done Done Sent false
2494 1321 Acked Read 6 Done Done Sent false
2495 1322 Acked Read 5 Done Done Sent false
2496 1323 Acked Read 6 Done Done Sent false
2497 1324 Acked Read 4 Done Done Sent false
2498 1325 Acked Read 7 Done Done Sent false
2499 1326 Acked Write 10 Done Done Sent false
2500 1327 Acked Read 10 Done Done Sent false
2501 1328 Acked Read 5 Done Done Sent false
2502 1329 Acked Write 8 Done Done Sent false
2503 1330 Acked Read 10 Done Done Sent false
2504 1331 Acked Write 10 Done Done Sent false
2505 1332 Acked Write 9 Done Done Sent false
2506 1333 Acked Write 1 Done Done Sent false
2507 1334 Acked Read 7 Done Done Sent false
2508 1335 Acked Read 7 Done Done Sent false
2509 1336 Acked Write 5 Done Done Sent false
2510 1337 Acked Write 3 Done Done Sent false
2511 1338 Acked Read 10 Done Done Sent false
2512 1339 Acked Write 8 Done Done Sent false
2513 1340 Acked Flush 0 Done Done Sent false
2514 1341 Acked Read 9 Done Done Sent false
2515 1342 Acked Write 2 Done Done Sent false
2516 1343 Acked Flush 0 Done Done Sent false
2517 1344 Acked Write 4 Done Done Sent false
2518 1345 Acked Write 8 Done Done Sent false
2519 1346 Acked Write 8 Done Done Sent false
2520 1347 Acked Flush 0 Done Done Sent false
2521 1348 Acked Read 9 Done Done Sent false
2522 1349 Acked Write 2 Done Done Sent false
2523 1350 Acked Read 3 Done Done Sent false
2524 1351 Acked Read 9 Done Done Sent false
2525 1352 Acked Write 3 Done Done Sent false
2526 1353 Acked Write 10 Done Done Sent false
2527 1354 Acked Write 1 Done Done Sent false
2528 1355 Acked Flush 0 Done Done Sent false
2529 1356 Acked Read 2 Done Done Sent false
2530 1357 Acked Write 4 Done Done Sent false
2531 1358 Acked Read 9 Done Done Sent false
2532 1359 Acked Read 1 Done Done Sent false
2533 1360 Acked Read 7 Done Done Sent false
2534 1361 Acked Read 9 Done Done Sent false
2535 1362 Acked Read 10 Done Done Sent false
2536 1363 Acked Write 1 Done Done Sent false
2537 1364 Acked Write 9 Done Done Sent false
2538 1365 Acked Write 8 Done Done Sent false
2539 1366 Acked Write 4 Done Done Sent false
2540 1367 Acked Write 8 Done Done Sent false
2541 1368 Acked Flush 0 Done Done Sent false
2542 1369 Acked Read 1 Done Done Sent false
2543 1370 Acked Read 7 Done Done Sent false
2544 1371 Acked Write 6 Done Done Sent false
2545 1372 Acked Read 3 Done Done Sent false
2546 1373 Acked Write 5 Done Done Sent false
2547 1374 Acked Read 1 Done Done Sent false
2548 1375 Acked Read 3 Done Done Sent false
2549 1376 Acked Flush 0 Done Done Sent false
2550 1377 Acked Write 8 Done Done Sent false
2551 1378 Acked Read 2 Done Done Sent false
2552 1379 Acked Read 10 Done Done Sent false
2553 1380 Acked Write 1 Done Done Sent false
2554 1381 Acked Write 2 Done Done Sent false
2555 1382 Acked Write 1 Done Done Sent false
2556 1383 Acked Write 4 Done Done Sent false
2557 1384 Acked Flush 0 Done Done Sent false
2558 1385 Acked Read 2 Done Done Sent false
2559 1386 Acked Read 9 Done Done Sent false
2560 1387 Acked Read 9 Done Done Sent false
2561 1388 Acked Read 3 Done Done Sent false
2562 1389 Acked Write 5 Done Done Sent false
2563 1390 Acked Flush 0 Done Done Sent false
2564 1391 Acked Read 5 Done Done Sent false
2565 1392 Acked Write 4 Done Done Sent false
2566 1393 Acked Read 10 Done Done Sent false
2567 1394 Acked Write 3 Done Done Sent false
2568 1395 Acked Write 5 Done Done Sent false
2569 1396 Acked Write 5 Done Done Sent false
2570 1397 Acked Write 2 Done Done Sent false
2571 1398 Acked Flush 0 Done Done Sent false
2572 1399 Acked Read 2 Done Done Sent false
2573 1400 Acked Flush 0 Done Done Sent false
2574 1401 Acked Write 8 Done Done Sent false
2575 1402 Acked Write 6 Done Done Sent false
2576 1403 Acked Read 2 Done Done Sent false
2577 1404 Acked Read 9 Done Done Sent false
2578 1405 Acked Read 5 Done Done Sent false
2579 1406 Acked Read 6 Done Done Sent false
2580 1407 Acked Write 4 Done Done Sent false
2581 1408 Acked Read 7 Done Done Sent false
2582 1409 Acked Flush 0 Done Done Sent false
2583 1410 Acked Write 1 Done Done Sent false
2584 1411 Acked Read 4 Done Done Sent false
2585 1412 Acked Flush 0 Done Done Sent false
2586 1413 Acked Flush 0 Done Done Sent false
2587 1414 Acked Read 5 Done Done Sent false
2588 1415 Acked Write 4 Done Done Sent false
2589 1416 Acked Write 2 Done Done Sent false
2590 1417 Acked Read 7 Done Done Sent false
2591 1418 Acked Write 9 Done Done Sent false
2592 1419 Acked Write 5 Done Done Sent false
2593 1420 Acked Read 10 Done Done Sent false
2594 1421 Acked Flush 0 Done Done Sent false
2595 1422 Acked Write 4 Done Done Sent false
2596 1423 Acked Flush 0 Done Done Sent false
2597 1424 Acked Read 9 Done Done Sent false
2598 1425 Acked Read 7 Done Done Sent false
2599 1426 Acked Write 9 Done Done Sent false
2600 1427 Acked Write 1 Done Done Sent false
2601 1428 Acked Write 7 Done Done Sent false
2602 1429 Acked Read 5 Done Done Sent false
2603 1430 Acked Read 8 Done Done Sent false
2604 1431 Acked Read 7 Done Done Sent false
2605 1432 Acked Write 2 Done Done Sent false
2606 1433 Acked Read 10 Done Done Sent false
2607 1434 Acked Read 2 Done Done Sent false
2608 1435 Acked Flush 0 Done Done Sent false
2609 1436 Acked Flush 0 Done Done Sent false
2610 1437 Acked Write 1 Done Done Sent false
2611 1438 Acked Write 8 Done Done Sent false
2612 1439 Acked Read 6 Done Done Sent false
2613 1440 Acked Read 1 Done Done Sent false
2614 1441 Acked Read 9 Done Done Sent false
2615 1442 Acked Read 9 Done Done Sent false
2616 1443 Acked Write 2 Done Done Sent false
2617 1444 Acked Write 3 Done Done Sent false
2618 1445 Acked Read 8 Done Done Sent false
2619 1446 Acked Read 2 Done Done Sent false
2620 1447 Acked Read 5 Done Done Sent false
2621 1448 Acked Read 6 Done Done Sent false
2622 1449 Acked Write 6 Done Done Sent false
2623 1450 Acked Write 10 Done Done Sent false
2624 1451 Acked Read 5 Done Done Sent false
2625 1452 Acked Write 7 Done Done Sent false
2626 1453 Acked Read 8 Done Done Sent false
2627 1454 Acked Read 6 Done Done Sent false
2628 1455 Acked Read 1 Done Done Sent false
2629 1456 Acked Read 1 Done Done Sent false
2630 1457 Acked Write 7 Done Done Sent false
2631 1458 Acked Read 9 Done Done Sent false
2632 1459 Acked Read 9 Done Done Sent false
2633 1460 Acked Flush 0 Done Done Sent false
2634 1461 Acked Read 7 Done Done Sent false
2635 1462 Acked Read 6 Done Done Sent false
2636 1463 Acked Read 4 Done Done Sent false
2637 1464 Acked Read 4 Done Done Sent false
2638 1465 Acked Write 4 Done Done Sent false
2639 1466 Acked Read 8 Done Done Sent false
2640 1467 Acked Write 2 Done Done Sent false
2641 1468 Acked Flush 0 Done Done Sent false
2642 1469 Acked Write 2 Done Done Sent false
2643 1470 Acked Read 7 Done Done Sent false
2644 1471 Acked Write 5 Done Done Sent false
2645 1472 Acked Read 5 Done Done Sent false
2646 1473 Acked Read 2 Done Done Sent false
2647 1474 Acked Read 2 Done Done Sent false
2648 1475 Acked Read 8 Done Done Sent false
2649 1476 Acked Write 3 Done Done Sent false
2650 1477 Acked Read 8 Done Done Sent false
2651 1478 Acked Read 2 Done Done Sent false
2652 1479 Acked Flush 0 Done Done Sent false
2653 1480 Acked Write 6 Done Done Sent false
2654 1481 Acked Read 8 Done Done Sent false
2655 1482 Acked Write 10 Done Done Sent false
2656 1483 Acked Write 3 Done Done Sent false
2657 1484 Acked Read 8 Done Done Sent false
2658 1485 Acked Read 3 Done Done Sent false
2659 1486 Acked Write 3 Done Done Sent false
2660 1487 Acked Read 3 Done Done Sent false
2661 1488 Acked Flush 0 Done Done Sent false
2662 1489 Acked Read 3 Done Done Sent false
2663 1490 Acked Read 2 Done Done Sent false
2664 1491 Acked Write 4 Done Done Sent false
2665 1492 Acked Write 2 Done Done Sent false
2666 1493 Acked Write 4 Done Done Sent false
2667 1494 Acked Read 1 Done Done Sent false
2668 1495 Acked Read 10 Done Done Sent false
2669 1496 Acked Read 4 Done Done Sent false
2670 1497 Acked Write 6 Done Done Sent false
2671 1498 Acked Flush 0 Done Done Sent false
2672 1499 Acked Write 9 Done Done Sent false
2673 1500 Acked Read 10 Done Done Sent false
2674 1501 Acked Read 2 Done Done Sent false
2675 1502 Acked Read 8 Done Done Sent false
2676 1503 Acked Write 8 Done Done Sent false
2677 1504 Acked Write 4 Done Done Sent false
2678 1505 Acked Read 2 Done Done Sent false
2679 1506 Acked Write 5 Done Done Sent false
2680 1507 Acked Write 4 Done Done Sent false
2681 1508 Acked Read 7 Done Done Sent false
2682 1509 Acked Flush 0 Done Done Sent false
2683 1510 Acked Flush 0 Done Done Sent false
2684 1511 Acked Write 6 Done Done Sent false
2685 1512 Acked Read 8 Done Done Sent false
2686 1513 Acked Flush 0 Done Done Sent false
2687 1514 Acked Read 5 Done Done Sent false
2688 1515 Acked Read 8 Done Done Sent false
2689 1516 Acked Write 5 Done Done Sent false
2690 1517 Acked Read 8 Done Done Sent false
2691 1518 Acked Read 7 Done Done Sent false
2692 1519 Acked Flush 0 Done Done Sent false
2693 1520 Acked Write 3 Done Done Sent false
2694 1521 Acked Write 6 Done Done Sent false
2695 1522 Acked Write 4 Done Done Sent false
2696 1523 Acked Read 2 Done Done Sent false
2697 1524 Acked Flush 0 Done Done Sent false
2698 1525 Acked Write 9 Done Done Sent false
2699 1526 Acked Write 5 Done Done Sent false
2700 1527 Acked Write 5 Done Done Sent false
2701 1528 Acked Write 10 Done Done Sent false
2702 1529 Acked Write 6 Done Done Sent false
2703 1530 Acked Write 4 Done Done Sent false
2704 1531 Acked Read 8 Done Done Sent false
2705 1532 Acked Flush 0 Done Done Sent false
2706 1533 Acked Write 5 Done Done Sent false
2707 1534 Acked Write 9 Done Done Sent false
2708 1535 Acked Read 7 Done Done Sent false
2709 1536 Acked Write 2 Done Done Sent false
2710 1537 Acked Write 8 Done Done Sent false
2711 1538 Acked Read 7 Done Done Sent false
2712 1539 Acked Flush 0 Done Done Sent false
2713 1540 Acked Read 3 Done Done Sent false
2714 1541 Acked Write 4 Done Done Sent false
2715 1542 Acked Read 8 Done Done Sent false
2716 1543 Acked Read 5 Done Done Sent false
2717 1544 Acked Read 4 Done Done Sent false
2718 1545 Acked Write 8 Done Done Sent false
2719 1546 Acked Read 6 Done Done Sent false
2720 1547 Acked Write 5 Done Done Sent false
2721 1548 Acked Write 6 Done Done Sent false
2722 1549 Acked Read 7 Done Done Sent false
2723 1550 Acked Read 4 Done Done Sent false
2724 1551 Acked Write 5 Done Done Sent false
2725 1552 Acked Write 2 Done Done Sent false
2726 1553 Acked Write 6 Done Done Sent false
2727 1554 Acked Write 3 Done Done Sent false
2728 1555 Acked Write 9 Done Done Sent false
2729 1556 Acked Read 3 Done Done Sent false
2730 1557 Acked Read 5 Done Done Sent false
2731 1558 Acked Write 9 Done Done Sent false
2732 1559 Acked Read 10 Done Done Sent false
2733 1560 Acked Write 5 Done Done Sent false
2734 1561 Acked Read 3 Done Done Sent false
2735 1562 Acked Write 2 Done Done Sent false
2736 1563 Acked Flush 0 Done Done Sent false
2737 1564 Acked Read 8 Done Done Sent false
2738 1565 Acked Write 8 Done Done Sent false
2739 1566 Acked Read 5 Done Done Sent false
2740 1567 Acked Read 7 Done Done Sent false
2741 1568 Acked Write 4 Done Done Sent false
2742 1569 Acked Flush 0 Done Done Sent false
2743 1570 Acked Read 10 Done Done Sent false
2744 1571 Acked Read 6 Done Done Sent false
2745 1572 Acked Read 10 Done Done Sent false
2746 1573 Acked Write 1 Done Done Sent false
2747 1574 Acked Write 4 Done Done Sent false
2748 1575 Acked Write 5 Done Done Sent false
2749 1576 Acked Write 5 Done Done Sent false
2750 1577 Acked Read 6 Done Done Sent false
2751 1578 Acked Write 5 Done Done Sent false
2752 1579 Acked Read 7 Done Done Sent false
2753 1580 Acked Read 8 Done Done Sent false
2754 1581 Acked Write 2 Done Done Sent false
2755 1582 Acked Write 1 Done Done Sent false
2756 1583 Acked Read 4 Done Done Sent false
2757 1584 Acked Read 7 Done Done Sent false
2758 1585 Acked Read 2 Done Done Sent false
2759 1586 Acked Read 4 Done Done Sent false
2760 1587 Acked Write 8 Done Done Sent false
2761 1588 Acked Flush 0 Done Done Sent false
2762 1589 Acked Read 6 Done Done Sent false
2763 1590 Acked Read 3 Done Done Sent false
2764 1591 Acked Read 7 Done Done Sent false
2765 1592 Acked Read 1 Done Done Sent false
2766 1593 Acked Read 2 Done Done Sent false
2767 1594 Acked Write 10 Done Done Sent false
2768 1595 Acked Write 6 Done Done Sent false
2769 1596 Acked Read 2 Done Done Sent false
2770 1597 Acked Read 1 Done Done Sent false
2771 1598 Acked Read 2 Done Done Sent false
2772 1599 Acked Write 2 Done Done Sent false
2773 1600 Acked Read 1 Done Done Sent false
2774 1601 Acked Read 5 Done Done Sent false
2775 1602 Acked Read 10 Done Done Sent false
2776 STATES DS:0 DS:1 DS:2 TOTAL
2777 Sent 0 0 301 301
2778 Done 301 301 0 602
2779 Skipped 0 0 0 0
2780 Error 0 0 0 0
2781 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2782 Downstairs last five completed: 1301 1300 1299 1298 1297
2783 Upstairs last five completed: 1602 1601 1600 1599 1598
2784 CLIENT: Up:0 ds:301 act:2
27852025-03-17T22:35:01.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27862025-03-17T22:35:02.344ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27872025-03-17T22:35:03.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27882025-03-17T22:35:04.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27892025-03-17T22:35:05.093ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27902025-03-17T22:35:05.093ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:32787) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27912025-03-17T22:35:05.094ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27922025-03-17T22:35:05.094ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27932025-03-17T22:35:05.094ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27942025-03-17T22:35:05.094ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27952025-03-17T22:35:05.094ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27962025-03-17T22:35:05.094ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27972025-03-17T22:35:05.095ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2798 ----------------------------------------------------------------
2799 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
2800 States: Active Active Active
2801 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2802 1589 Acked Read 6 Done Done Done true
2803 1590 Acked Read 3 Done Done Done true
2804 1591 Acked Read 7 Done Done Done true
2805 1592 Acked Read 1 Done Done Done true
2806 1593 Acked Read 2 Done Done Done true
2807 1594 Acked Write 10 Done Done Done true
2808 1595 Acked Write 6 Done Done Done true
2809 1596 Acked Read 2 Done Done Done true
2810 1597 Acked Read 1 Done Done Done true
2811 1598 Acked Read 2 Done Done Done true
2812 1599 Acked Write 2 Done Done Done true
2813 1600 Acked Read 1 Done Done Done true
2814 1601 Acked Read 5 Done Done Done true
2815 1602 Acked Read 10 Done Done Done true
2816 STATES DS:0 DS:1 DS:2 TOTAL
2817 Sent 0 0 0 0
2818 Done 14 14 14 42
2819 Skipped 0 0 0 0
2820 Error 0 0 0 0
2821 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1588))
2822 Downstairs last five completed: 1588 1587 1586 1585 1584
2823 Upstairs last five completed: 1602 1601 1600 1599 1598
2824 CLIENT: Up:0 ds:14 act:3
2825 ----------------------------------------------------------------
2826 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2827 Downstairs last five completed: 1603 1602 1601 1600 1599
2828 Upstairs last five completed: 1603 1602 1601 1600 1599
2829 CLIENT: Up:0 ds:0 act:3
2830 Replay: All jobs finished
2831 Main task has work to do, go find it
2832 got dsc Stop(2)
2833 stop 2
2834 Mar 17 22:35:09.343 INFO request completed, latency_us: 367, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 58836995-a7fc-484b-8971-f04c152bfaf0, remote_addr: 127.0.0.1:35180, local_addr[8830] Got stop action so:false kr:false
2835 : 127.0.0.1:9998
2836 Mar 17 22:35:09.344 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/2, method: GET, req_id: eed685d2-9b94-47de-9188-89944066ccab, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
28372025-03-17T22:35:09.344ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28382025-03-17T22:35:09.344ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28392025-03-17T22:35:09.344ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28402025-03-17T22:35:09.344ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
2841 [8830] Exited with: ExitStatus(unix_wait_status(9))
2842 [8830][2] reports Exit
2843 Mar 17 22:35:13.346 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/2, method: GET, req_id: ae326736-8b91-4a58-b5d7-7c33a810ce85, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
2844 001/300 Read block 658 len 28672
2845 002/300 Write block 3153 len 8192 data: 2 2
2846 003/300 Flush
2847 004/300 Read block 3796 len 40960
2848 005/300 Read block 1028 len 36864
2849 006/300 Write block 3362 len 36864 data: 2 2 2 2 2 3 3 3 3
2850 007/300 Write block 2065 len 28672 data: 2 2 2 2 2 2 2
2851 008/300 Write block 4535 len 4096 data: 2
2852 009/300 Read block 3357 len 20480
2853 010/300 Write block 2561 len 16384 data: 2 2 2 2
2854 011/300 Write block 2379 len 8192 data: 2 2
2855 012/300 Write block 3261 len 16384 data: 3 3 2 3
2856 013/300 Read block 1841 len 36864
2857 014/300 Write block 3632 len 20480 data: 2 2 2 2 2
2858 015/300 Write block 3680 len 16384 data: 2 2 2 2
2859 016/300 Read block 3596 len 40960
2860 017/300 Read block 3515 len 36864
2861 018/300 Read block 1391 len 32768
2862 019/300 Read block 1292 len 16384
2863 020/300 Write block 4583 len 24576 data: 2 2 2 2 2 2
2864 021/300 Read block 3900 len 4096
2865 022/300 Flush
2866 023/300 Read block 2557 len 24576
2867 024/300 Write block 1160 len 12288 data: 2 2 2
2868 025/300 Read block 1113 len 4096
2869 026/300 Read block 3386 len 4096
2870 027/300 Read block 885 len 4096
2871 028/300 Flush
2872 029/300 Write block 65 len 32768 data: 2 2 2 2 2 2 2 2
2873 030/300 Write block 4971 len 12288 data: 2 2 2
2874 031/300 Read block 1505 len 40960
2875 032/300 Flush
2876 033/300 Read block 559 len 36864
2877 034/300 Write block 3227 len 4096 data: 2
2878 035/300 Write block 4396 len 20480 data: 2 2 2 2 2
2879 036/300 Write block 428 len 8192 data: 2 2
2880 037/300 Flush
2881 038/300 Read block 2609 len 8192
2882 039/300 Read block 1913 len 40960
2883 040/300 Read block 2287 len 8192
2884 041/300 Flush
2885 042/300 Write block 2181 len 16384 data: 2 2 2 2
2886 043/300 Read block 4666 len 40960
2887 044/300 Write block 4254 len 24576 data: 2 2 2 2 2 2
2888 045/300 Read block 2410 len 24576
2889 046/300 Write block 3595 len 24576 data: 2 2 3 3 3 3
2890 047/300 Read block 3553 len 4096
2891 048/300 Write block 3987 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2892 049/300 Read block 2983 len 28672
2893 050/300 Read block 356 len 20480
2894 051/300 Read block 4478 len 4096
2895 052/300 Read block 4489 len 4096
2896 053/300 Flush
2897 054/300 Write block 671 len 32768 data: 2 2 2 2 3 3 4 3
2898 055/300 Write block 639 len 4096 data: 2
2899 056/300 Read block 360 len 4096
2900 057/300 Write block 1162 len 12288 data: 3 2 2
2901 058/300 Write block 223 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2902 059/300 Write block 308 len 4096 data: 2
2903 060/300 Read block 1546 len 32768
2904 061/300 Write block 2071 len 8192 data: 3 2
2905 062/300 Read block 3301 len 4096
2906 063/300 Write block 2035 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2907 064/300 Read block 2161 len 28672
2908 065/300 Write block 2540 len 16384 data: 3 3 2 2
2909 066/300 Flush
2910 067/300 Read block 3497 len 16384
2911 068/300 Write block 1453 len 28672 data: 2 2 2 2 2 2 2
2912 069/300 Read block 488 len 8192
2913 070/300 Write block 4639 len 16384 data: 3 3 2 2
2914 071/300 Read block 4910 len 36864
2915 072/300 Write block 3603 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2916 073/300 Read block 4291 len 16384
2917 074/300 Read block 4172 len 20480
2918 075/300 Read block 3515 len 36864
2919 076/300 Write block 1831 len 12288 data: 3 3 3
2920 077/300 Read block 3744 len 12288
2921 078/300 Read block 4670 len 20480
2922 079/300 Write block 3418 len 36864 data: 3 2 2 3 3 3 3 3 3
2923 080/300 Read block 3561 len 36864
2924 081/300 Read block 3497 len 8192
2925 082/300 Read block 2461 len 4096
2926 083/300 Read block 2860 len 20480
2927 084/300 Write block 3288 len 32768 data: 2 2 2 2 2 2 2 2
2928 085/300 Read block 624 len 32768
2929 086/300 Read block 2659 len 16384
2930 087/300 Write block 1190 len 16384 data: 2 2 2 3
2931 088/300 Flush
2932 089/300 Flush
2933 090/300 Read block 115 len 8192
2934 091/300 Write block 4696 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2935 092/300 Read block 978 len 32768
2936 093/300 Read block 2268 len 16384
2937 094/300 Read block 4690 len 16384
2938 095/300 Write block 2400 len 32768 data: 3 3 3 3 3 2 2 2
2939 096/300 Write block 3070 len 16384 data: 3 3 3 3
2940 097/300 Write block 663 len 12288 data: 2 2 2
2941 098/300 Read block 3768 len 36864
2942 099/300 Read block 1708 len 4096
2943 100/300 Read block 1564 len 28672
2944 101/300 Write block 1306 len 16384 data: 2 2 2 2
2945 102/300 Flush
2946 103/300 Flush
2947 104/300 Read block 2391 len 40960
2948 105/300 Write block 4659 len 28672 data: 3 3 3 3 3 2 2
2949 106/300 Read block 4946 len 20480
2950 107/300 Write block 2687 len 12288 data: 2 2 2
2951 108/300 Write block 175 len 16384 data: 2 2 2 3
2952 109/300 Read block 4092 len 20480
2953 110/300 Write block 4389 len 24576 data: 2 2 2 2 2 2
2954 111/300 Read block 4996 len 4096
2955 112/300 Read block 320 len 20480
2956 113/300 Read block 571 len 16384
2957 114/300 Read block 4465 len 24576
2958 115/300 Flush
2959 116/300 Write block 4863 len 28672 data: 2 2 2 2 2 2 2
2960 117/300 Write block 4641 len 24576 data: 3 3 2 2 2 2
2961 118/300 Flush
2962 119/300 Read block 887 len 8192
2963 120/300 Write block 4783 len 8192 data: 3 2
2964 121/300 Read block 1317 len 8192
2965 122/300 Read block 4746 len 36864
2966 123/300 Read block 2034 len 16384
2967 124/300 Read block 13 len 4096
2968 125/300 Read block 2513 len 20480
2969 126/300 Write block 4214 len 4096 data: 2
2970 127/300 Read block 2353 len 4096
2971 128/300 Read block 2847 len 4096
2972 129/300 Write block 2637 len 24576 data: 2 2 2 2 2 2
2973 130/300 Read block 4522 len 28672
2974 131/300 Write block 1396 len 28672 data: 3 3 2 3 3 3 2
2975 132/300 Write block 1532 len 20480 data: 3 4 3 2 2
2976 133/300 Read block 635 len 40960
2977 134/300 Write block 4500 len 32768 data: 2 3 3 3 3 3 3 2
2978 135/300 Read block 2103 len 28672
2979 136/300 Write block 2474 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2980 137/300 Read block 3637 len 16384
2981 138/300 Read block 3938 len 4096
2982 139/300 Read block 141 len 24576
2983 140/300 Read block 2055 len 16384
2984 141/300 Flush
2985 142/300 Write block 3210 len 40960 data: 3 3 3 3 3 3 3 3 3 2
2986 143/300 Write block 2662 len 36864 data: 2 2 2 2 2 2 2 2 2
2987 144/300 Read block 1349 len 12288
2988 145/300 Flush
2989 146/300 Read block 1907 len 36864
2990 147/300 Read block 523 len 8192
2991 148/300 Read block 4054 len 20480
2992 149/300 Flush
2993 150/300 Read block 4178 len 12288
2994 151/300 Write block 1953 len 24576 data: 2 2 2 2 2 2
2995 152/300 Read block 834 len 40960
2996 153/300 Read block 3599 len 20480
2997 154/300 Write block 643 len 16384 data: 2 2 2 2
2998 155/300 Read block 1976 len 24576
2999 156/300 Flush
3000 157/300 Read block 152 len 16384
3001 158/300 Read block 2261 len 28672
3002 159/300 Read block 3390 len 40960
3003 160/300 Flush
3004 161/300 Read block 2787 len 40960
3005 162/300 Write block 2902 len 4096 data: 2
3006 163/300 Read block 3948 len 12288
3007 164/300 Write block 4113 len 32768 data: 3 3 3 3 2 2 2 2
3008 165/300 Read block 1675 len 36864
3009 166/300 Read block 766 len 36864
3010 167/300 Read block 4445 len 12288
3011 168/300 Read block 2216 len 20480
3012 169/300 Read block 2820 len 8192
3013 170/300 Read block 510 len 24576
3014 171/300 Flush
3015 172/300 Read block 978 len 32768
3016 173/300 Write block 1479 len 16384 data: 3 3 3 3
3017 174/300 Write block 2792 len 40960 data: 4 3 3 2 2 2 2 2 2 2
3018 175/300 Flush
3019 176/300 Read block 3340 len 40960
3020 177/300 Read block 4005 len 8192
3021 178/300 Write block 2233 len 12288 data: 2 2 2
3022 179/300 Read block 4586 len 4096
3023 180/300 Read block 90 len 16384
3024 181/300 Write block 135 len 16384 data: 2 2 2 2
3025 182/300 Read block 894 len 20480
3026 183/300 Read block 1925 len 40960
3027 184/300 Write block 2210 len 28672 data: 2 2 2 2 2 2 2
3028 185/300 Write block 4630 len 8192 data: 2 2
3029 186/300 Write block 3733 len 12288 data: 3 3 3
3030 187/300 Read block 605 len 28672
3031 188/300 Write block 4611 len 32768 data: 2 2 2 2 2 2 2 2
3032 189/300 Read block 4840 len 8192
3033 190/300 Read block 590 len 4096
3034 191/300 Write block 2895 len 8192 data: 2 2
3035 192/300 Write block 4698 len 20480 data: 3 3 3 3 3
3036 193/300 Read block 2784 len 8192
3037 194/300 Read block 1477 len 16384
3038 195/300 Read block 4188 len 32768
3039 196/300 Write block 609 len 4096 data: 2
3040 197/300 Flush
3041 198/300 Write block 602 len 36864 data: 2 2 2 2 2 2 2 3 2
3042 199/300 Read block 1873 len 12288
3043 200/300 Read block 3175 len 12288
3044 201/300 Write block 2992 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3045 202/300 Write block 1189 len 28672 data: 2 3 3 3 4 3 3
3046 203/300 Read block 150 len 20480
3047 204/300 Write block 1771 len 24576 data: 3 3 2 2 2 2
3048 205/300 Read block 1239 len 32768
3049 206/300 Write block 1912 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3050 207/300 Read block 540 len 8192
3051 208/300 Write block 979 len 24576 data: 2 2 3 3 3 3
3052 209/300 Write block 901 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3053 210/300 Write block 970 len 32768 data: 3 3 3 3 3 3 2 2
3054 211/300 Read block 4102 len 12288
3055 212/300 Flush
3056 213/300 Read block 3965 len 24576
3057 214/300 Write block 4501 len 12288 data: 4 4 4
3058 215/300 Write block 4291 len 36864 data: 2 2 2 2 2 2 2 2 2
3059 216/300 Write block 2513 len 28672 data: 2 2 2 2 2 2 2
3060 217/300 Write block 4981 len 24576 data: 2 2 2 2 2 2
3061 218/300 Write block 2373 len 12288 data: 3 3 3
3062 219/300 Write block 1655 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3063 220/300 Flush
3064 221/300 Write block 3357 len 12288 data: 2 2 2
3065 222/300 Read block 4271 len 40960
3066 223/300 Write block 1304 len 4096 data: 2
3067 224/300 Flush
3068 225/300 Read block 3845 len 24576
3069 226/300 Write block 4464 len 8192 data: 2 2
3070 227/300 Read block 2087 len 32768
3071 228/300 Write block 2571 len 40960 data: 2 2 2 2 2 3 3 3 3 3
3072 229/300 Flush
3073 230/300 Read block 257 len 20480
3074 231/300 Read block 1905 len 16384
3075 232/300 Write block 4820 len 4096 data: 2
3076 233/300 Read block 3075 len 24576
3077 234/300 Read block 160 len 8192
3078 235/300 Read block 712 len 28672
3079 236/300 Write block 1380 len 20480 data: 2 3 3 4 3
3080 237/300 Read block 1798 len 36864
3081 238/300 Write block 4179 len 28672 data: 2 2 2 2 2 2 3
3082 239/300 Read block 3733 len 32768
3083 240/300 Write block 613 len 36864 data: 2 2 2 2 2 2 2 2 2
3084 241/300 Write block 4231 len 12288 data: 3 3 3
3085 242/300 Read block 4850 len 36864
3086 243/300 Read block 747 len 20480
3087 244/300 Write block 129 len 24576 data: 2 2 2 2 2 2
3088 245/300 Read block 1295 len 36864
3089 246/300 Write block 4550 len 4096 data: 2
3090 247/300 Flush
3091 248/300 Read block 4508 len 4096
3092 249/300 Read block 38 len 28672
3093 250/300 Read block 4296 len 24576
3094 251/300 Write block 2009 len 28672 data: 2 2 2 2 2 2 2
3095 252/300 Flush
3096 253/300 Write block 1618 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3097 254/300 Flush
3098 255/300 Read block 357 len 32768
3099 256/300 Read block 1317 len 4096
3100 257/300 Read block 1772 len 8192
3101 258/300 Read block 4464 len 32768
3102 259/300 Read block 219 len 28672
3103 260/300 Read block 2633 len 4096
3104 261/300 Read block 4553 len 12288
3105 262/300 Write block 3640 len 12288 data: 2 2 2
3106 263/300 Read block 2537 len 16384
3107 264/300 Flush
3108 265/300 Write block 2718 len 20480 data: 2 2 2 2 2
3109 266/300 Read block 1366 len 20480
3110 267/300 Read block 806 len 40960
3111 268/300 Write block 3149 len 24576 data: 2 2 2 2 3 3
3112 269/300 Read block 3961 len 32768
3113 270/300 Write block 211 len 4096 data: 2
3114 271/300 Read block 558 len 20480
3115 272/300 Read block 4521 len 36864
3116 273/300 Read block 2592 len 8192
3117 274/300 Read block 3500 len 20480
3118 275/300 Read block 2012 len 8192
3119 276/300 Flush
3120 277/300 Read block 3705 len 24576
3121 278/300 Read block 2754 len 12288
3122 279/300 Read block 4056 len 24576
3123 280/300 Read block 3516 len 24576
3124 281/300 Write block 3556 len 4096 data: 2
3125 282/300 Read block 2143 len 40960
3126 283/300 Read block 4069 len 12288
3127 284/300 Read block 4076 len 8192
3128 285/300 Write block 3086 len 4096 data: 2
3129 286/300 Read block 2190 len 28672
3130 287/300 Read block 3372 len 16384
3131 288/300 Flush
3132 289/300 Write block 81 len 12288 data: 2 2 3
3133 290/300 Write block 103 len 16384 data: 3 3 3 3
3134 291/300 Write block 4303 len 16384 data: 2 2 2 2
3135 292/300 Write block 2258 len 20480 data: 2 2 2 2 2
3136 293/300 Read block 20 len 24576
3137 294/300 Read block 3224 len 4096
3138 295/300 Write block 4782 len 20480 data: 3 4 3 2 2
3139 296/300 Read block 4816 len 20480
3140 297/300 Write block 4549 len 32768 data: 2 3 2 2 2 2 2 2
3141 298/300 Flush
3142 299/300 Read block 1758 len 36864
3143 300/300 Read block 2267 len 12288
3144 301/300 Read block 3592 len 8192
3145 Main task has work to do, go find it
3146 got dsc Start(2)
3147 start 2
3148 Mar 17 22:35:13.587 INFO request completed, latency_us: 153, response_code: 204, uri: /start/cid/2, method: POST, req_id: 34147a5a-9b91-44a7-9134-17c7bd0e5278, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
3149 Replay: started 2, returned:Ok(())
3150 ----------------------------------------------------------------
3151 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3152 States: Active Active Offline
3153 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3154 1604 Acked Read 7 Done Done Sent false
3155 1605 Acked Write 2 Done Done Sent false
3156 1606 Acked Flush 0 Done Done Sent false
3157 1607 Acked Read 10 Done Done Sent false
3158 1608 Acked Read 9 Done Done Sent false
3159 1609 Acked Write 9 Done Done Sent false
3160 1610 Acked Write 7 Done Done Sent false
3161 1611 Acked Write 1 Done Done Sent false
3162 1612 Acked Read 5 Done Done Sent false
3163 1613 Acked Write 4 Done Done Sent false
3164 1614 Acked Write 2 Done Done Sent false
3165 1615 Acked Write 4 Done Done Sent false
3166 1616 Acked Read 9 Done Done Sent false
3167 1617 Acked Write 5 Done Done Sent false
3168 1618 Acked Write 4 Done Done Sent false
3169 1619 Acked Read 10 Done Done Sent false
3170 1620 Acked Read 9 Done Done Sent false
3171 1621 Acked Read 8 Done Done Sent false
3172 1622 Acked Read 4 Done Done Sent false
3173 1623 Acked Write 6 Done Done Sent false
3174 1624 Acked Read 1 Done Done Sent false
3175 1625 Acked Flush 0 Done Done Sent false
3176 1626 Acked Read 6 Done Done Sent false
3177 1627 Acked Write 3 Done Done Sent false
3178 1628 Acked Read 1 Done Done Sent false
3179 1629 Acked Read 1 Done Done Sent false
3180 1630 Acked Read 1 Done Done Sent false
3181 1631 Acked Flush 0 Done Done Sent false
3182 1632 Acked Write 8 Done Done Sent false
3183 1633 Acked Write 3 Done Done Sent false
3184 1634 Acked Read 10 Done Done Sent false
3185 1635 Acked Flush 0 Done Done Sent false
3186 1636 Acked Read 9 Done Done Sent false
3187 1637 Acked Write 1 Done Done Sent false
3188 1638 Acked Write 5 Done Done Sent false
3189 1639 Acked Write 2 Done Done Sent false
3190 1640 Acked Flush 0 Done Done Sent false
3191 1641 Acked Read 2 Done Done Sent false
3192 1642 Acked Read 10 Done Done Sent false
3193 1643 Acked Read 2 Done Done Sent false
3194 1644 Acked Flush 0 Done Done Sent false
3195 1645 Acked Write 4 Done Done Sent false
3196 1646 Acked Read 10 Done Done Sent false
3197 1647 Acked Write 6 Done Done Sent false
3198 1648 Acked Read 6 Done Done Sent false
3199 1649 Acked Write 6 Done Done Sent false
3200 1650 Acked Read 1 Done Done Sent false
3201 1651 Acked Write 10 Done Done Sent false
3202 1652 Acked Read 7 Done Done Sent false
3203 1653 Acked Read 5 Done Done Sent false
3204 1654 Acked Read 1 Done Done Sent false
3205 1655 Acked Read 1 Done Done Sent false
3206 1656 Acked Flush 0 Done Done Sent false
3207 1657 Acked Write 8 Done Done Sent false
3208 1658 Acked Write 1 Done Done Sent false
3209 1659 Acked Read 1 Done Done Sent false
3210 1660 Acked Write 3 Done Done Sent false
3211 1661 Acked Write 10 Done Done Sent false
3212 1662 Acked Write 1 Done Done Sent false
3213 1663 Acked Read 8 Done Done Sent false
3214 1664 Acked Write 2 Done Done Sent false
3215 1665 Acked Read 1 Done Done Sent false
3216 1666 Acked Write 10 Done Done Sent false
3217 1667 Acked Read 7 Done Done Sent false
3218 1668 Acked Write 4 Done Done Sent false
3219 1669 Acked Flush 0 Done Done Sent false
3220 1670 Acked Read 4 Done Done Sent false
3221 1671 Acked Write 7 Done Done Sent false
3222 1672 Acked Read 2 Done Done Sent false
3223 1673 Acked Write 4 Done Done Sent false
3224 1674 Acked Read 9 Done Done Sent false
3225 1675 Acked Write 10 Done Done Sent false
3226 1676 Acked Read 4 Done Done Sent false
3227 1677 Acked Read 5 Done Done Sent false
3228 1678 Acked Read 9 Done Done Sent false
3229 1679 Acked Write 3 Done Done Sent false
3230 1680 Acked Read 3 Done Done Sent false
3231 1681 Acked Read 5 Done Done Sent false
3232 1682 Acked Write 9 Done Done Sent false
3233 1683 Acked Read 9 Done Done Sent false
3234 1684 Acked Read 2 Done Done Sent false
3235 1685 Acked Read 1 Done Done Sent false
3236 1686 Acked Read 5 Done Done Sent false
3237 1687 Acked Write 8 Done Done Sent false
3238 1688 Acked Read 8 Done Done Sent false
3239 1689 Acked Read 4 Done Done Sent false
3240 1690 Acked Write 4 Done Done Sent false
3241 1691 Acked Flush 0 Done Done Sent false
3242 1692 Acked Flush 0 Done Done Sent false
3243 1693 Acked Read 2 Done Done Sent false
3244 1694 Acked Write 10 Done Done Sent false
3245 1695 Acked Read 8 Done Done Sent false
3246 1696 Acked Read 4 Done Done Sent false
3247 1697 Acked Read 4 Done Done Sent false
3248 1698 Acked Write 8 Done Done Sent false
3249 1699 Acked Write 4 Done Done Sent false
3250 1700 Acked Write 3 Done Done Sent false
3251 1701 Acked Read 9 Done Done Sent false
3252 1702 Acked Read 1 Done Done Sent false
3253 1703 Acked Read 7 Done Done Sent false
3254 1704 Acked Write 4 Done Done Sent false
3255 1705 Acked Flush 0 Done Done Sent false
3256 1706 Acked Flush 0 Done Done Sent false
3257 1707 Acked Read 10 Done Done Sent false
3258 1708 Acked Write 7 Done Done Sent false
3259 1709 Acked Read 5 Done Done Sent false
3260 1710 Acked Write 3 Done Done Sent false
3261 1711 Acked Write 4 Done Done Sent false
3262 1712 Acked Read 5 Done Done Sent false
3263 1713 Acked Write 6 Done Done Sent false
3264 1714 Acked Read 1 Done Done Sent false
3265 1715 Acked Read 5 Done Done Sent false
3266 1716 Acked Read 4 Done Done Sent false
3267 1717 Acked Read 6 Done Done Sent false
3268 1718 Acked Flush 0 Done Done Sent false
3269 1719 Acked Write 7 Done Done Sent false
3270 1720 Acked Write 6 Done Done Sent false
3271 1721 Acked Flush 0 Done Done Sent false
3272 1722 Acked Read 2 Done Done Sent false
3273 1723 Acked Write 2 Done Done Sent false
3274 1724 Acked Read 2 Done Done Sent false
3275 1725 Acked Read 9 Done Done Sent false
3276 1726 Acked Read 4 Done Done Sent false
3277 1727 Acked Read 1 Done Done Sent false
3278 1728 Acked Read 5 Done Done Sent false
3279 1729 Acked Write 1 Done Done Sent false
3280 1730 Acked Read 1 Done Done Sent false
3281 1731 Acked Read 1 Done Done Sent false
3282 1732 Acked Write 6 Done Done Sent false
3283 1733 Acked Read 7 Done Done Sent false
3284 1734 Acked Write 7 Done Done Sent false
3285 1735 Acked Write 5 Done Done Sent false
3286 1736 Acked Read 10 Done Done Sent false
3287 1737 Acked Write 8 Done Done Sent false
3288 1738 Acked Read 7 Done Done Sent false
3289 1739 Acked Write 10 Done Done Sent false
3290 1740 Acked Read 4 Done Done Sent false
3291 1741 Acked Read 1 Done Done Sent false
3292 1742 Acked Read 6 Done Done Sent false
3293 1743 Acked Read 4 Done Done Sent false
3294 1744 Acked Flush 0 Done Done Sent false
3295 1745 Acked Write 10 Done Done Sent false
3296 1746 Acked Write 9 Done Done Sent false
3297 1747 Acked Read 3 Done Done Sent false
3298 1748 Acked Flush 0 Done Done Sent false
3299 1749 Acked Read 9 Done Done Sent false
3300 1750 Acked Read 2 Done Done Sent false
3301 1751 Acked Read 5 Done Done Sent false
3302 1752 Acked Flush 0 Done Done Sent false
3303 1753 Acked Read 3 Done Done Sent false
3304 1754 Acked Write 6 Done Done Sent false
3305 1755 Acked Read 10 Done Done Sent false
3306 1756 Acked Read 5 Done Done Sent false
3307 1757 Acked Write 4 Done Done Sent false
3308 1758 Acked Read 6 Done Done Sent false
3309 1759 Acked Flush 0 Done Done Sent false
3310 1760 Acked Read 4 Done Done Sent false
3311 1761 Acked Read 7 Done Done Sent false
3312 1762 Acked Read 10 Done Done Sent false
3313 1763 Acked Flush 0 Done Done Sent false
3314 1764 Acked Read 10 Done Done Sent false
3315 1765 Acked Write 1 Done Done Sent false
3316 1766 Acked Read 3 Done Done Sent false
3317 1767 Acked Write 8 Done Done Sent false
3318 1768 Acked Read 9 Done Done Sent false
3319 1769 Acked Read 9 Done Done Sent false
3320 1770 Acked Read 3 Done Done Sent false
3321 1771 Acked Read 5 Done Done Sent false
3322 1772 Acked Read 2 Done Done Sent false
3323 1773 Acked Read 6 Done Done Sent false
3324 1774 Acked Flush 0 Done Done Sent false
3325 1775 Acked Read 8 Done Done Sent false
3326 1776 Acked Write 4 Done Done Sent false
3327 1777 Acked Write 10 Done Done Sent false
3328 1778 Acked Flush 0 Done Done Sent false
3329 1779 Acked Read 10 Done Done Sent false
3330 1780 Acked Read 2 Done Done Sent false
3331 1781 Acked Write 3 Done Done Sent false
3332 1782 Acked Read 1 Done Done Sent false
3333 1783 Acked Read 4 Done Done Sent false
3334 1784 Acked Write 4 Done Done Sent false
3335 1785 Acked Read 5 Done Done Sent false
3336 1786 Acked Read 10 Done Done Sent false
3337 1787 Acked Write 7 Done Done Sent false
3338 1788 Acked Write 2 Done Done Sent false
3339 1789 Acked Write 3 Done Done Sent false
3340 1790 Acked Read 7 Done Done Sent false
3341 1791 Acked Write 8 Done Done Sent false
3342 1792 Acked Read 2 Done Done Sent false
3343 1793 Acked Read 1 Done Done Sent false
3344 1794 Acked Write 2 Done Done Sent false
3345 1795 Acked Write 5 Done Done Sent false
3346 1796 Acked Read 2 Done Done Sent false
3347 1797 Acked Read 4 Done Done Sent false
3348 1798 Acked Read 8 Done Done Sent false
3349 1799 Acked Write 1 Done Done Sent false
3350 1800 Acked Flush 0 Done Done Sent false
3351 1801 Acked Write 9 Done Done Sent false
3352 1802 Acked Read 3 Done Done Sent false
3353 1803 Acked Read 3 Done Done Sent false
3354 1804 Acked Write 10 Done Done Sent false
3355 1805 Acked Write 7 Done Done Sent false
3356 1806 Acked Read 5 Done Done Sent false
3357 1807 Acked Write 6 Done Done Sent false
3358 1808 Acked Read 8 Done Done Sent false
3359 1809 Acked Write 10 Done Done Sent false
3360 1810 Acked Read 2 Done Done Sent false
3361 1811 Acked Write 6 Done Done Sent false
3362 1812 Acked Write 10 Done Done Sent false
3363 1813 Acked Write 8 Done Done Sent false
3364 1814 Acked Read 3 Done Done Sent false
3365 1815 Acked Flush 0 Done Done Sent false
3366 1816 Acked Read 6 Done Done Sent false
3367 1817 Acked Write 3 Done Done Sent false
3368 1818 Acked Write 9 Done Done Sent false
3369 1819 Acked Write 7 Done Done Sent false
3370 1820 Acked Write 6 Done Done Sent false
3371 1821 Acked Write 3 Done Done Sent false
3372 1822 Acked Write 10 Done Done Sent false
3373 1823 Acked Flush 0 Done Done Sent false
3374 1824 Acked Write 3 Done Done Sent false
3375 1825 Acked Read 10 Done Done Sent false
3376 1826 Acked Write 1 Done Done Sent false
3377 1827 Acked Flush 0 Done Done Sent false
3378 1828 Acked Read 6 Done Done Sent false
3379 1829 Acked Write 2 Done Done Sent false
3380 1830 Acked Read 8 Done Done Sent false
3381 1831 Acked Write 10 Done Done Sent false
3382 1832 Acked Flush 0 Done Done Sent false
3383 1833 Acked Read 5 Done Done Sent false
3384 1834 Acked Read 4 Done Done Sent false
3385 1835 Acked Write 1 Done Done Sent false
3386 1836 Acked Read 6 Done Done Sent false
3387 1837 Acked Read 2 Done Done Sent false
3388 1838 Acked Read 7 Done Done Sent false
3389 1839 Acked Write 5 Done Done Sent false
3390 1840 Acked Read 9 Done Done Sent false
3391 1841 Acked Write 7 Done Done Sent false
3392 1842 Acked Read 8 Done Done Sent false
3393 1843 Acked Write 9 Done Done Sent false
3394 1844 Acked Write 3 Done Done Sent false
3395 1845 Acked Read 9 Done Done Sent false
3396 1846 Acked Read 5 Done Done Sent false
3397 1847 Acked Write 6 Done Done Sent false
3398 1848 Acked Read 9 Done Done Sent false
3399 1849 Acked Write 1 Done Done Sent false
3400 1850 Acked Flush 0 Done Done Sent false
3401 1851 Acked Read 1 Done Done Sent false
3402 1852 Acked Read 7 Done Done Sent false
3403 1853 Acked Read 6 Done Done Sent false
3404 1854 Acked Write 7 Done Done Sent false
3405 1855 Acked Flush 0 Done Done Sent false
3406 1856 Acked Write 10 Done Done Sent false
3407 1857 Acked Flush 0 Done Done Sent false
3408 1858 Acked Read 8 Done Done Sent false
3409 1859 Acked Read 1 Done Done Sent false
3410 1860 Acked Read 2 Done Done Sent false
3411 1861 Acked Read 8 Done Done Sent false
3412 1862 Acked Read 7 Done Done Sent false
3413 1863 Acked Read 1 Done Done Sent false
3414 1864 Acked Read 3 Done Done Sent false
3415 1865 Acked Write 3 Done Done Sent false
3416 1866 Acked Read 4 Done Done Sent false
3417 1867 Acked Flush 0 Done Done Sent false
3418 1868 Acked Write 5 Done Done Sent false
3419 1869 Acked Read 5 Done Done Sent false
3420 1870 Acked Read 10 Done Done Sent false
3421 1871 Acked Write 6 Done Done Sent false
3422 1872 Acked Read 8 Done Done Sent false
3423 1873 Acked Write 1 Done Done Sent false
3424 1874 Acked Read 5 Done Done Sent false
3425 1875 Acked Read 9 Done Done Sent false
3426 1876 Acked Read 2 Done Done Sent false
3427 1877 Acked Read 5 Done Done Sent false
3428 1878 Acked Read 2 Done Done Sent false
3429 1879 Acked Flush 0 Done Done Sent false
3430 1880 Acked Read 6 Done Done Sent false
3431 1881 Acked Read 3 Done Done Sent false
3432 1882 Acked Read 6 Done Done Sent false
3433 1883 Acked Read 6 Done Done Sent false
3434 1884 Acked Write 1 Done Done Sent false
3435 1885 Acked Read 10 Done Done Sent false
3436 1886 Acked Read 3 Done Done Sent false
3437 1887 Acked Read 2 Done Done Sent false
3438 1888 Acked Write 1 Done Done Sent false
3439 1889 Acked Read 7 Done Done Sent false
3440 1890 Acked Read 4 Done Done Sent false
3441 1891 Acked Flush 0 Done Done Sent false
3442 1892 Acked Write 3 Done Done Sent false
3443 1893 Acked Write 4 Done Done Sent false
3444 1894 Acked Write 4 Done Done Sent false
3445 1895 Acked Write 5 Done Done Sent false
3446 1896 Acked Read 6 Done Done Sent false
3447 1897 Acked Read 1 Done Done Sent false
3448 1898 Acked Write 5 Done Done Sent false
3449 1899 Acked Read 5 Done Done Sent false
3450 1900 Acked Write 8 Done Done Sent false
3451 1901 Acked Flush 0 Done Done Sent false
3452 1902 Acked Read 9 Done Done Sent false
3453 1903 Acked Read 3 Done Done Sent false
3454 1904 Acked Read 2 Done Done Sent false
3455 STATES DS:0 DS:1 DS:2 TOTAL
3456 Sent 0 0 301 301
3457 Done 301 301 0 602
3458 Skipped 0 0 0 0
3459 Error 0 0 0 0
3460 Last Flush: Some(JobId(1901)) Some(JobId(1901)) Some(JobId(1603))
3461 Downstairs last five completed: 1603 1602 1601 1600 1599
3462 Upstairs last five completed: 1904 1903 1902 1901 1900
3463 CLIENT: Up:0 ds:301 act:2
34642025-03-17T22:35:14.363ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34652025-03-17T22:35:14.863ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
3466 [8830] got start action
3467 [8830] I am going to restart
3468 Starting downstairs at port 8830
3469 Make output file at "/tmp/dsc/downstairs-8830.txt"
3470 [8830][2] reports Starting
3471 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
3472 [8830][2] reports Running
34732025-03-17T22:35:15.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34742025-03-17T22:35:16.366ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34752025-03-17T22:35:17.367ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
3476 ----------------------------------------------------------------
3477 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3478 States: Active Active Offline
3479 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3480 1604 Acked Read 7 Done Done Sent false
3481 1605 Acked Write 2 Done Done Sent false
3482 1606 Acked Flush 0 Done Done Sent false
3483 1607 Acked Read 10 Done Done Sent false
3484 1608 Acked Read 9 Done Done Sent false
3485 1609 Acked Write 9 Done Done Sent false
3486 1610 Acked Write 7 Done Done Sent false
3487 1611 Acked Write 1 Done Done Sent false
3488 1612 Acked Read 5 Done Done Sent false
3489 1613 Acked Write 4 Done Done Sent false
3490 1614 Acked Write 2 Done Done Sent false
3491 1615 Acked Write 4 Done Done Sent false
3492 1616 Acked Read 9 Done Done Sent false
3493 1617 Acked Write 5 Done Done Sent false
3494 1618 Acked Write 4 Done Done Sent false
3495 1619 Acked Read 10 Done Done Sent false
3496 1620 Acked Read 9 Done Done Sent false
3497 1621 Acked Read 8 Done Done Sent false
3498 1622 Acked Read 4 Done Done Sent false
3499 1623 Acked Write 6 Done Done Sent false
3500 1624 Acked Read 1 Done Done Sent false
3501 1625 Acked Flush 0 Done Done Sent false
3502 1626 Acked Read 6 Done Done Sent false
3503 1627 Acked Write 3 Done Done Sent false
3504 1628 Acked Read 1 Done Done Sent false
3505 1629 Acked Read 1 Done Done Sent false
3506 1630 Acked Read 1 Done Done Sent false
3507 1631 Acked Flush 0 Done Done Sent false
3508 1632 Acked Write 8 Done Done Sent false
3509 1633 Acked Write 3 Done Done Sent false
3510 1634 Acked Read 10 Done Done Sent false
3511 1635 Acked Flush 0 Done Done Sent false
3512 1636 Acked Read 9 Done Done Sent false
3513 1637 Acked Write 1 Done Done Sent false
3514 1638 Acked Write 5 Done Done Sent false
3515 1639 Acked Write 2 Done Done Sent false
3516 1640 Acked Flush 0 Done Done Sent false
3517 1641 Acked Read 2 Done Done Sent false
3518 1642 Acked Read 10 Done Done Sent false
3519 1643 Acked Read 2 Done Done Sent false
3520 1644 Acked Flush 0 Done Done Sent false
3521 1645 Acked Write 4 Done Done Sent false
3522 1646 Acked Read 10 Done Done Sent false
3523 1647 Acked Write 6 Done Done Sent false
3524 1648 Acked Read 6 Done Done Sent false
3525 1649 Acked Write 6 Done Done Sent false
3526 1650 Acked Read 1 Done Done Sent false
3527 1651 Acked Write 10 Done Done Sent false
3528 1652 Acked Read 7 Done Done Sent false
3529 1653 Acked Read 5 Done Done Sent false
3530 1654 Acked Read 1 Done Done Sent false
3531 1655 Acked Read 1 Done Done Sent false
3532 1656 Acked Flush 0 Done Done Sent false
3533 1657 Acked Write 8 Done Done Sent false
3534 1658 Acked Write 1 Done Done Sent false
3535 1659 Acked Read 1 Done Done Sent false
3536 1660 Acked Write 3 Done Done Sent false
3537 1661 Acked Write 10 Done Done Sent false
3538 1662 Acked Write 1 Done Done Sent false
3539 1663 Acked Read 8 Done Done Sent false
3540 1664 Acked Write 2 Done Done Sent false
3541 1665 Acked Read 1 Done Done Sent false
3542 1666 Acked Write 10 Done Done Sent false
3543 1667 Acked Read 7 Done Done Sent false
3544 1668 Acked Write 4 Done Done Sent false
3545 1669 Acked Flush 0 Done Done Sent false
3546 1670 Acked Read 4 Done Done Sent false
3547 1671 Acked Write 7 Done Done Sent false
3548 1672 Acked Read 2 Done Done Sent false
3549 1673 Acked Write 4 Done Done Sent false
3550 1674 Acked Read 9 Done Done Sent false
3551 1675 Acked Write 10 Done Done Sent false
3552 1676 Acked Read 4 Done Done Sent false
3553 1677 Acked Read 5 Done Done Sent false
3554 1678 Acked Read 9 Done Done Sent false
3555 1679 Acked Write 3 Done Done Sent false
3556 1680 Acked Read 3 Done Done Sent false
3557 1681 Acked Read 5 Done Done Sent false
3558 1682 Acked Write 9 Done Done Sent false
3559 1683 Acked Read 9 Done Done Sent false
3560 1684 Acked Read 2 Done Done Sent false
3561 1685 Acked Read 1 Done Done Sent false
3562 1686 Acked Read 5 Done Done Sent false
3563 1687 Acked Write 8 Done Done Sent false
3564 1688 Acked Read 8 Done Done Sent false
3565 1689 Acked Read 4 Done Done Sent false
3566 1690 Acked Write 4 Done Done Sent false
3567 1691 Acked Flush 0 Done Done Sent false
3568 1692 Acked Flush 0 Done Done Sent false
3569 1693 Acked Read 2 Done Done Sent false
3570 1694 Acked Write 10 Done Done Sent false
3571 1695 Acked Read 8 Done Done Sent false
3572 1696 Acked Read 4 Done Done Sent false
3573 1697 Acked Read 4 Done Done Sent false
3574 1698 Acked Write 8 Done Done Sent false
3575 1699 Acked Write 4 Done Done Sent false
3576 1700 Acked Write 3 Done Done Sent false
3577 1701 Acked Read 9 Done Done Sent false
3578 1702 Acked Read 1 Done Done Sent false
3579 1703 Acked Read 7 Done Done Sent false
3580 1704 Acked Write 4 Done Done Sent false
3581 1705 Acked Flush 0 Done Done Sent false
3582 1706 Acked Flush 0 Done Done Sent false
3583 1707 Acked Read 10 Done Done Sent false
3584 1708 Acked Write 7 Done Done Sent false
3585 1709 Acked Read 5 Done Done Sent false
3586 1710 Acked Write 3 Done Done Sent false
3587 1711 Acked Write 4 Done Done Sent false
3588 1712 Acked Read 5 Done Done Sent false
3589 1713 Acked Write 6 Done Done Sent false
3590 1714 Acked Read 1 Done Done Sent false
3591 1715 Acked Read 5 Done Done Sent false
3592 1716 Acked Read 4 Done Done Sent false
3593 1717 Acked Read 6 Done Done Sent false
3594 1718 Acked Flush 0 Done Done Sent false
3595 1719 Acked Write 7 Done Done Sent false
3596 1720 Acked Write 6 Done Done Sent false
3597 1721 Acked Flush 0 Done Done Sent false
3598 1722 Acked Read 2 Done Done Sent false
3599 1723 Acked Write 2 Done Done Sent false
3600 1724 Acked Read 2 Done Done Sent false
3601 1725 Acked Read 9 Done Done Sent false
3602 1726 Acked Read 4 Done Done Sent false
3603 1727 Acked Read 1 Done Done Sent false
3604 1728 Acked Read 5 Done Done Sent false
3605 1729 Acked Write 1 Done Done Sent false
3606 1730 Acked Read 1 Done Done Sent false
3607 1731 Acked Read 1 Done Done Sent false
3608 1732 Acked Write 6 Done Done Sent false
3609 1733 Acked Read 7 Done Done Sent false
3610 1734 Acked Write 7 Done Done Sent false
3611 1735 Acked Write 5 Done Done Sent false
3612 1736 Acked Read 10 Done Done Sent false
3613 1737 Acked Write 8 Done Done Sent false
3614 1738 Acked Read 7 Done Done Sent false
3615 1739 Acked Write 10 Done Done Sent false
3616 1740 Acked Read 4 Done Done Sent false
3617 1741 Acked Read 1 Done Done Sent false
3618 1742 Acked Read 6 Done Done Sent false
3619 1743 Acked Read 4 Done Done Sent false
3620 1744 Acked Flush 0 Done Done Sent false
3621 1745 Acked Write 10 Done Done Sent false
3622 1746 Acked Write 9 Done Done Sent false
3623 1747 Acked Read 3 Done Done Sent false
3624 1748 Acked Flush 0 Done Done Sent false
3625 1749 Acked Read 9 Done Done Sent false
3626 1750 Acked Read 2 Done Done Sent false
3627 1751 Acked Read 5 Done Done Sent false
3628 1752 Acked Flush 0 Done Done Sent false
3629 1753 Acked Read 3 Done Done Sent false
3630 1754 Acked Write 6 Done Done Sent false
3631 1755 Acked Read 10 Done Done Sent false
3632 1756 Acked Read 5 Done Done Sent false
3633 1757 Acked Write 4 Done Done Sent false
3634 1758 Acked Read 6 Done Done Sent false
3635 1759 Acked Flush 0 Done Done Sent false
3636 1760 Acked Read 4 Done Done Sent false
3637 1761 Acked Read 7 Done Done Sent false
3638 1762 Acked Read 10 Done Done Sent false
3639 1763 Acked Flush 0 Done Done Sent false
3640 1764 Acked Read 10 Done Done Sent false
3641 1765 Acked Write 1 Done Done Sent false
3642 1766 Acked Read 3 Done Done Sent false
3643 1767 Acked Write 8 Done Done Sent false
3644 1768 Acked Read 9 Done Done Sent false
3645 1769 Acked Read 9 Done Done Sent false
3646 1770 Acked Read 3 Done Done Sent false
3647 1771 Acked Read 5 Done Done Sent false
3648 1772 Acked Read 2 Done Done Sent false
3649 1773 Acked Read 6 Done Done Sent false
3650 1774 Acked Flush 0 Done Done Sent false
3651 1775 Acked Read 8 Done Done Sent false
3652 1776 Acked Write 4 Done Done Sent false
3653 1777 Acked Write 10 Done Done Sent false
3654 1778 Acked Flush 0 Done Done Sent false
3655 1779 Acked Read 10 Done Done Sent false
3656 1780 Acked Read 2 Done Done Sent false
3657 1781 Acked Write 3 Done Done Sent false
3658 1782 Acked Read 1 Done Done Sent false
3659 1783 Acked Read 4 Done Done Sent false
3660 1784 Acked Write 4 Done Done Sent false
3661 1785 Acked Read 5 Done Done Sent false
3662 1786 Acked Read 10 Done Done Sent false
3663 1787 Acked Write 7 Done Done Sent false
3664 1788 Acked Write 2 Done Done Sent false
3665 1789 Acked Write 3 Done Done Sent false
3666 1790 Acked Read 7 Done Done Sent false
3667 1791 Acked Write 8 Done Done Sent false
3668 1792 Acked Read 2 Done Done Sent false
3669 1793 Acked Read 1 Done Done Sent false
3670 1794 Acked Write 2 Done Done Sent false
3671 1795 Acked Write 5 Done Done Sent false
3672 1796 Acked Read 2 Done Done Sent false
3673 1797 Acked Read 4 Done Done Sent false
3674 1798 Acked Read 8 Done Done Sent false
3675 1799 Acked Write 1 Done Done Sent false
3676 1800 Acked Flush 0 Done Done Sent false
3677 1801 Acked Write 9 Done Done Sent false
3678 1802 Acked Read 3 Done Done Sent false
3679 1803 Acked Read 3 Done Done Sent false
3680 1804 Acked Write 10 Done Done Sent false
3681 1805 Acked Write 7 Done Done Sent false
3682 1806 Acked Read 5 Done Done Sent false
3683 1807 Acked Write 6 Done Done Sent false
3684 1808 Acked Read 8 Done Done Sent false
3685 1809 Acked Write 10 Done Done Sent false
3686 1810 Acked Read 2 Done Done Sent false
3687 1811 Acked Write 6 Done Done Sent false
3688 1812 Acked Write 10 Done Done Sent false
3689 1813 Acked Write 8 Done Done Sent false
3690 1814 Acked Read 3 Done Done Sent false
3691 1815 Acked Flush 0 Done Done Sent false
3692 1816 Acked Read 6 Done Done Sent false
3693 1817 Acked Write 3 Done Done Sent false
3694 1818 Acked Write 9 Done Done Sent false
3695 1819 Acked Write 7 Done Done Sent false
3696 1820 Acked Write 6 Done Done Sent false
3697 1821 Acked Write 3 Done Done Sent false
3698 1822 Acked Write 10 Done Done Sent false
3699 1823 Acked Flush 0 Done Done Sent false
3700 1824 Acked Write 3 Done Done Sent false
3701 1825 Acked Read 10 Done Done Sent false
3702 1826 Acked Write 1 Done Done Sent false
3703 1827 Acked Flush 0 Done Done Sent false
3704 1828 Acked Read 6 Done Done Sent false
3705 1829 Acked Write 2 Done Done Sent false
3706 1830 Acked Read 8 Done Done Sent false
3707 1831 Acked Write 10 Done Done Sent false
3708 1832 Acked Flush 0 Done Done Sent false
3709 1833 Acked Read 5 Done Done Sent false
3710 1834 Acked Read 4 Done Done Sent false
3711 1835 Acked Write 1 Done Done Sent false
3712 1836 Acked Read 6 Done Done Sent false
3713 1837 Acked Read 2 Done Done Sent false
3714 1838 Acked Read 7 Done Done Sent false
3715 1839 Acked Write 5 Done Done Sent false
3716 1840 Acked Read 9 Done Done Sent false
3717 1841 Acked Write 7 Done Done Sent false
3718 1842 Acked Read 8 Done Done Sent false
3719 1843 Acked Write 9 Done Done Sent false
3720 1844 Acked Write 3 Done Done Sent false
3721 1845 Acked Read 9 Done Done Sent false
3722 1846 Acked Read 5 Done Done Sent false
3723 1847 Acked Write 6 Done Done Sent false
3724 1848 Acked Read 9 Done Done Sent false
3725 1849 Acked Write 1 Done Done Sent false
3726 1850 Acked Flush 0 Done Done Sent false
3727 1851 Acked Read 1 Done Done Sent false
3728 1852 Acked Read 7 Done Done Sent false
3729 1853 Acked Read 6 Done Done Sent false
3730 1854 Acked Write 7 Done Done Sent false
3731 1855 Acked Flush 0 Done Done Sent false
3732 1856 Acked Write 10 Done Done Sent false
3733 1857 Acked Flush 0 Done Done Sent false
3734 1858 Acked Read 8 Done Done Sent false
3735 1859 Acked Read 1 Done Done Sent false
3736 1860 Acked Read 2 Done Done Sent false
3737 1861 Acked Read 8 Done Done Sent false
3738 1862 Acked Read 7 Done Done Sent false
3739 1863 Acked Read 1 Done Done Sent false
3740 1864 Acked Read 3 Done Done Sent false
3741 1865 Acked Write 3 Done Done Sent false
3742 1866 Acked Read 4 Done Done Sent false
3743 1867 Acked Flush 0 Done Done Sent false
3744 1868 Acked Write 5 Done Done Sent false
3745 1869 Acked Read 5 Done Done Sent false
3746 1870 Acked Read 10 Done Done Sent false
3747 1871 Acked Write 6 Done Done Sent false
3748 1872 Acked Read 8 Done Done Sent false
3749 1873 Acked Write 1 Done Done Sent false
3750 1874 Acked Read 5 Done Done Sent false
3751 1875 Acked Read 9 Done Done Sent false
3752 1876 Acked Read 2 Done Done Sent false
3753 1877 Acked Read 5 Done Done Sent false
3754 1878 Acked Read 2 Done Done Sent false
3755 1879 Acked Flush 0 Done Done Sent false
3756 1880 Acked Read 6 Done Done Sent false
3757 1881 Acked Read 3 Done Done Sent false
3758 1882 Acked Read 6 Done Done Sent false
3759 1883 Acked Read 6 Done Done Sent false
3760 1884 Acked Write 1 Done Done Sent false
3761 1885 Acked Read 10 Done Done Sent false
3762 1886 Acked Read 3 Done Done Sent false
3763 1887 Acked Read 2 Done Done Sent false
3764 1888 Acked Write 1 Done Done Sent false
3765 1889 Acked Read 7 Done Done Sent false
3766 1890 Acked Read 4 Done Done Sent false
3767 1891 Acked Flush 0 Done Done Sent false
3768 1892 Acked Write 3 Done Done Sent false
3769 1893 Acked Write 4 Done Done Sent false
3770 1894 Acked Write 4 Done Done Sent false
3771 1895 Acked Write 5 Done Done Sent false
3772 1896 Acked Read 6 Done Done Sent false
3773 1897 Acked Read 1 Done Done Sent false
3774 1898 Acked Write 5 Done Done Sent false
3775 1899 Acked Read 5 Done Done Sent false
3776 1900 Acked Write 8 Done Done Sent false
3777 1901 Acked Flush 0 Done Done Sent false
3778 1902 Acked Read 9 Done Done Sent false
3779 1903 Acked Read 3 Done Done Sent false
3780 1904 Acked Read 2 Done Done Sent false
3781 STATES DS:0 DS:1 DS:2 TOTAL
3782 Sent 0 0 301 301
3783 Done 301 301 0 602
3784 Skipped 0 0 0 0
3785 Error 0 0 0 0
3786 Last Flush: Some(JobId(1901)) Some(JobId(1901)) Some(JobId(1603))
3787 Downstairs last five completed: 1603 1602 1601 1600 1599
3788 Upstairs last five completed: 1904 1903 1902 1901 1900
3789 CLIENT: Up:0 ds:301 act:2
37902025-03-17T22:35:18.369ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37912025-03-17T22:35:19.345ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37922025-03-17T22:35:19.345ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50722) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37932025-03-17T22:35:19.347ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37942025-03-17T22:35:19.347ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37952025-03-17T22:35:19.347ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37962025-03-17T22:35:19.347ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37972025-03-17T22:35:19.347ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37982025-03-17T22:35:19.347ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37992025-03-17T22:35:19.347ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
3800 ----------------------------------------------------------------
3801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3802 Downstairs last five completed: 1905 1904 1903 1902 1901
3803 Upstairs last five completed: 1905 1904 1903 1902 1901
3804 CLIENT: Up:0 ds:0 act:3
3805 Replay: All jobs finished
3806 Main task has work to do, go find it
3807 got dsc Stop(1)
3808 stop 1
3809 Mar 17 22:35:21.592 INFO request completed, latency_us: 332, response_code: 204, uri: /stop/cid/1[8820] Got stop action so:false kr:false
3810 , method: POST, req_id: cd97afe6-279a-4266-b444-63aa8357028a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
38112025-03-17T22:35:21.592ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
38122025-03-17T22:35:21.593ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
38132025-03-17T22:35:21.593ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
3814 Mar 17 22:35:21.593 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/1, method: GET, req_id: a066eed8-b31c-4bcb-91e1-18a8a3f40676, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
38152025-03-17T22:35:21.593ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
3816 [8820] Exited with: ExitStatus(unix_wait_status(9))
3817 [8820][1] reports Exit
3818 Mar 17 22:35:25.595 INFO request completed, latency_us: 316, response_code: 200, uri: /state/cid/1, method: GET, req_id: 821c5612-d7be-4652-bb43-a0957722b869, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
3819 001/300 Read block 4212 len 12288
3820 002/300 Read block 2720 len 4096
3821 003/300 Read block 4669 len 40960
3822 004/300 Read block 1334 len 32768
3823 005/300 Read block 4100 len 4096
3824 006/300 Read block 2235 len 16384
3825 007/300 Read block 2002 len 20480
3826 008/300 Read block 1532 len 8192
3827 009/300 Write block 959 len 4096 data: 2
3828 010/300 Read block 4926 len 4096
3829 011/300 Write block 4688 len 40960 data: 3 3 3 3 2 2 2 2 3 3
3830 012/300 Write block 1495 len 20480 data: 2 2 3 3 3
3831 013/300 Read block 2135 len 40960
3832 014/300 Read block 2198 len 16384
3833 015/300 Write block 307 len 32768 data: 2 3 2 2 2 2 2 2
3834 016/300 Write block 3010 len 8192 data: 2 2
3835 017/300 Read block 4625 len 40960
3836 018/300 Read block 740 len 16384
3837 019/300 Write block 3161 len 16384 data: 3 3 3 3
3838 020/300 Write block 657 len 32768 data: 2 2 2 2 2 2 3 3
3839 021/300 Write block 3785 len 16384 data: 3 2 2 2
3840 022/300 Read block 1313 len 12288
3841 023/300 Write block 414 len 28672 data: 3 4 3 3 3 3 3
3842 024/300 Read block 4713 len 12288
3843 025/300 Write block 3743 len 4096 data: 2
3844 026/300 Read block 2074 len 4096
3845 027/300 Read block 2440 len 36864
3846 028/300 Read block 814 len 4096
3847 029/300 Write block 4484 len 12288 data: 2 2 2
3848 030/300 Read block 57 len 40960
3849 031/300 Read block 3354 len 40960
3850 032/300 Read block 1747 len 4096
3851 033/300 Read block 1862 len 8192
3852 034/300 Read block 3853 len 8192
3853 035/300 Read block 914 len 28672
3854 036/300 Read block 3613 len 4096
3855 037/300 Flush
3856 038/300 Read block 4013 len 40960
3857 039/300 Write block 3662 len 8192 data: 2 2
3858 040/300 Flush
3859 041/300 Write block 4871 len 16384 data: 2 2 2 2
3860 042/300 Read block 2429 len 20480
3861 043/300 Write block 3194 len 16384 data: 2 2 2 2
3862 044/300 Read block 2252 len 40960
3863 045/300 Flush
3864 046/300 Read block 1003 len 32768
3865 047/300 Flush
3866 048/300 Read block 3328 len 40960
3867 049/300 Read block 2339 len 40960
3868 050/300 Read block 1070 len 20480
3869 051/300 Read block 2024 len 32768
3870 052/300 Flush
3871 053/300 Flush
3872 054/300 Write block 3015 len 12288 data: 2 2 2
3873 055/300 Write block 2274 len 8192 data: 3 3
3874 056/300 Read block 271 len 28672
3875 057/300 Read block 3907 len 40960
3876 058/300 Write block 4792 len 20480 data: 3 3 3 3 3
3877 059/300 Flush
3878 060/300 Read block 3555 len 4096
3879 061/300 Write block 925 len 28672 data: 2 2 2 2 3 2 2
3880 062/300 Read block 107 len 8192
3881 063/300 Write block 4803 len 36864 data: 2 2 2 2 2 2 2 2 2
3882 064/300 Write block 4080 len 12288 data: 2 2 2
3883 065/300 Read block 73 len 12288
3884 066/300 Read block 2076 len 28672
3885 067/300 Read block 2903 len 8192
3886 068/300 Read block 1449 len 40960
3887 069/300 Read block 4922 len 16384
3888 070/300 Write block 3581 len 8192 data: 2 2
3889 071/300 Read block 2223 len 4096
3890 072/300 Write block 4293 len 12288 data: 3 3 3
3891 073/300 Write block 2666 len 20480 data: 3 3 3 3 3
3892 074/300 Read block 3621 len 12288
3893 075/300 Write block 2729 len 28672 data: 3 3 3 3 3 3 3
3894 076/300 Flush
3895 077/300 Read block 1252 len 32768
3896 078/300 Write block 4903 len 36864 data: 2 3 3 2 2 2 2 3 3
3897 079/300 Read block 4715 len 16384
3898 080/300 Write block 4166 len 24576 data: 3 3 3 3 2 2
3899 081/300 Read block 3615 len 32768
3900 082/300 Read block 1830 len 20480
3901 083/300 Read block 1075 len 28672
3902 084/300 Read block 674 len 8192
3903 085/300 Flush
3904 086/300 Read block 2791 len 16384
3905 087/300 Write block 4252 len 20480 data: 3 3 3 3 3
3906 088/300 Write block 3839 len 8192 data: 2 2
3907 089/300 Write block 3305 len 8192 data: 3 2
3908 090/300 Read block 1721 len 28672
3909 091/300 Flush
3910 092/300 Flush
3911 093/300 Write block 2669 len 24576 data: 4 4 2 2 2 2
3912 094/300 Write block 4022 len 24576 data: 2 2 2 2 2 2
3913 095/300 Read block 1760 len 40960
3914 096/300 Read block 3159 len 8192
3915 097/300 Read block 860 len 24576
3916 098/300 Read block 74 len 20480
3917 099/300 Write block 4102 len 8192 data: 2 2
3918 100/300 Read block 2065 len 4096
3919 101/300 Write block 2273 len 4096 data: 3
3920 102/300 Write block 934 len 36864 data: 2 2 2 2 2 2 2 2 2
3921 103/300 Flush
3922 104/300 Read block 3614 len 20480
3923 105/300 Read block 458 len 20480
3924 106/300 Write block 2990 len 12288 data: 2 2 3
3925 107/300 Write block 1070 len 32768 data: 3 3 3 3 3 2 2 2
3926 108/300 Read block 3264 len 32768
3927 109/300 Write block 3725 len 4096 data: 2
3928 110/300 Write block 4274 len 20480 data: 2 2 2 2 2
3929 111/300 Read block 2270 len 8192
3930 112/300 Write block 2772 len 12288 data: 2 2 2
3931 113/300 Write block 3829 len 36864 data: 2 2 2 3 3 2 2 2 2
3932 114/300 Write block 2575 len 36864 data: 3 4 4 4 4 4 3 3 3
3933 115/300 Read block 3674 len 36864
3934 116/300 Read block 2129 len 24576
3935 117/300 Read block 2518 len 12288
3936 118/300 Read block 4676 len 4096
3937 119/300 Read block 4865 len 28672
3938 120/300 Write block 313 len 20480 data: 3 3 2 2 2
3939 121/300 Write block 1229 len 28672 data: 2 2 2 2 2 2 2
3940 122/300 Write block 2027 len 12288 data: 4 3 3
3941 123/300 Read block 1382 len 16384
3942 124/300 Read block 2710 len 4096
3943 125/300 Read block 2586 len 8192
3944 126/300 Write block 4067 len 24576 data: 2 2 2 2 2 3
3945 127/300 Read block 3342 len 24576
3946 128/300 Write block 171 len 32768 data: 2 2 2 2 3 3 3 4
3947 129/300 Write block 1941 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3948 130/300 Write block 4852 len 24576 data: 3 3 3 2 2 2
3949 131/300 Read block 3800 len 20480
3950 132/300 Write block 3175 len 32768 data: 2 2 2 3 3 3 4 4
3951 133/300 Read block 4961 len 40960
3952 134/300 Read block 2599 len 20480
3953 135/300 Read block 1873 len 16384
3954 136/300 Read block 4348 len 20480
3955 137/300 Write block 782 len 12288 data: 2 2 2
3956 138/300 Read block 63 len 12288
3957 139/300 Write block 743 len 24576 data: 3 3 3 3 3 4
3958 140/300 Write block 3520 len 12288 data: 3 3 4
3959 141/300 Flush
3960 142/300 Write block 2035 len 32768 data: 3 3 3 3 3 3 3 3
3961 143/300 Read block 1296 len 32768
3962 144/300 Flush
3963 145/300 Flush
3964 146/300 Read block 2335 len 8192
3965 147/300 Read block 590 len 16384
3966 148/300 Read block 1492 len 20480
3967 149/300 Write block 4118 len 8192 data: 3 3
3968 150/300 Flush
3969 151/300 Write block 4525 len 36864 data: 2 2 2 2 2 2 2 2 2
3970 152/300 Flush
3971 153/300 Read block 1029 len 36864
3972 154/300 Write block 2669 len 28672 data: 5 5 3 3 3 3 2
3973 155/300 Flush
3974 156/300 Write block 3679 len 12288 data: 2 3 3
3975 157/300 Write block 4198 len 12288 data: 2 2 2
3976 158/300 Read block 3260 len 32768
3977 159/300 Write block 4514 len 36864 data: 2 2 2 2 2 2 3 3 2
3978 160/300 Read block 3506 len 12288
3979 161/300 Read block 2292 len 32768
3980 162/300 Write block 2546 len 24576 data: 2 2 2 2 2 2
3981 163/300 Read block 4242 len 40960
3982 164/300 Write block 4423 len 32768 data: 2 2 2 2 2 2 2 2
3983 165/300 Write block 3753 len 32768 data: 2 2 2 2 2 3 3 3
3984 166/300 Read block 1772 len 36864
3985 167/300 Read block 3383 len 40960
3986 168/300 Read block 416 len 28672
3987 169/300 Read block 1535 len 8192
3988 170/300 Write block 4034 len 36864 data: 2 2 2 2 2 2 2 2 2
3989 171/300 Flush
3990 172/300 Read block 3852 len 28672
3991 173/300 Read block 4302 len 36864
3992 174/300 Flush
3993 175/300 Read block 253 len 12288
3994 176/300 Read block 2266 len 36864
3995 177/300 Read block 304 len 8192
3996 178/300 Write block 161 len 24576 data: 2 2 2 2 2 2
3997 179/300 Write block 1443 len 32768 data: 3 2 2 2 2 2 2 2
3998 180/300 Read block 4787 len 8192
3999 181/300 Write block 159 len 40960 data: 2 2 3 3 3 3 3 3 2 2
4000 182/300 Read block 1992 len 28672
4001 183/300 Read block 4198 len 12288
4002 184/300 Read block 1801 len 16384
4003 185/300 Write block 4028 len 4096 data: 2
4004 186/300 Write block 1357 len 36864 data: 2 2 2 2 2 2 2 2 2
4005 187/300 Flush
4006 188/300 Flush
4007 189/300 Read block 1436 len 20480
4008 190/300 Write block 2042 len 36864 data: 4 3 3 2 2 2 3 3 3
4009 191/300 Write block 1595 len 24576 data: 3 3 3 2 2 2
4010 192/300 Read block 2763 len 40960
4011 193/300 Write block 1882 len 4096 data: 2
4012 194/300 Read block 4079 len 28672
4013 195/300 Read block 4489 len 24576
4014 196/300 Flush
4015 197/300 Write block 1884 len 24576 data: 2 2 2 2 2 2
4016 198/300 Read block 1063 len 16384
4017 199/300 Read block 2366 len 40960
4018 200/300 Write block 2144 len 12288 data: 2 2 2
4019 201/300 Write block 2149 len 16384 data: 2 2 2 2
4020 202/300 Read block 2409 len 16384
4021 203/300 Read block 3197 len 20480
4022 204/300 Write block 949 len 28672 data: 2 2 2 2 2 2 2
4023 205/300 Flush
4024 206/300 Read block 3013 len 20480
4025 207/300 Write block 1003 len 32768 data: 2 2 2 2 3 2 2 2
4026 208/300 Flush
4027 209/300 Write block 4344 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4028 210/300 Write block 3668 len 16384 data: 2 3 3 2
4029 211/300 Flush
4030 212/300 Write block 3546 len 16384 data: 3 3 3 2
4031 213/300 Write block 3380 len 4096 data: 2
4032 214/300 Write block 3907 len 16384 data: 3 3 3 4
4033 215/300 Write block 2335 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4034 216/300 Read block 4963 len 28672
4035 217/300 Write block 1817 len 28672 data: 2 2 2 2 2 2 2
4036 218/300 Read block 889 len 36864
4037 219/300 Write block 3709 len 20480 data: 2 2 2 2 2
4038 220/300 Read block 2165 len 40960
4039 221/300 Write block 1479 len 16384 data: 4 4 4 4
4040 222/300 Flush
4041 223/300 Read block 4557 len 20480
4042 224/300 Read block 4529 len 20480
4043 225/300 Read block 2060 len 36864
4044 226/300 Read block 1393 len 36864
4045 227/300 Write block 4488 len 12288 data: 2 2 2
4046 228/300 Read block 2744 len 16384
4047 229/300 Write block 4814 len 24576 data: 2 2 2 2 2 2
4048 230/300 Write block 4333 len 8192 data: 2 2
4049 231/300 Flush
4050 232/300 Read block 4514 len 24576
4051 233/300 Write block 3988 len 4096 data: 3
4052 234/300 Flush
4053 235/300 Read block 4568 len 12288
4054 236/300 Read block 1544 len 12288
4055 237/300 Read block 3523 len 40960
4056 238/300 Write block 2265 len 32768 data: 3 3 4 4 4 3 3 3
4057 239/300 Flush
4058 240/300 Flush
4059 241/300 Read block 1702 len 32768
4060 242/300 Read block 2929 len 12288
4061 243/300 Read block 4296 len 12288
4062 244/300 Write block 1150 len 24576 data: 2 2 2 2 2 2
4063 245/300 Flush
4064 246/300 Read block 2034 len 36864
4065 247/300 Read block 4055 len 20480
4066 248/300 Read block 2937 len 28672
4067 249/300 Read block 4011 len 8192
4068 250/300 Read block 2166 len 28672
4069 251/300 Write block 2177 len 40960 data: 2 2 2 2 3 3 3 3 2 2
4070 252/300 Write block 449 len 28672 data: 4 4 3 2 2 2 2
4071 253/300 Write block 1207 len 24576 data: 3 3 2 2 2 2
4072 254/300 Read block 4301 len 28672
4073 255/300 Read block 1371 len 28672
4074 256/300 Write block 3926 len 28672 data: 2 2 2 2 2 2 2
4075 257/300 Read block 710 len 4096
4076 258/300 Flush
4077 259/300 Read block 2950 len 12288
4078 260/300 Read block 4664 len 28672
4079 261/300 Read block 4962 len 32768
4080 262/300 Read block 57 len 4096
4081 263/300 Read block 3499 len 36864
4082 264/300 Flush
4083 265/300 Write block 1326 len 28672 data: 2 2 2 2 2 2 2
4084 266/300 Read block 202 len 32768
4085 267/300 Read block 186 len 40960
4086 268/300 Read block 4949 len 16384
4087 269/300 Write block 2095 len 28672 data: 2 2 2 2 3 3 2
4088 270/300 Read block 2348 len 4096
4089 271/300 Write block 4671 len 4096 data: 2
4090 272/300 Write block 1560 len 12288 data: 2 2 2
4091 273/300 Write block 4157 len 8192 data: 3 3
4092 274/300 Read block 2739 len 28672
4093 275/300 Write block 3420 len 24576 data: 3 4 4 4 4 4
4094 276/300 Flush
4095 277/300 Read block 1951 len 40960
4096 278/300 Flush
4097 279/300 Read block 2490 len 24576
4098 280/300 Write block 2501 len 8192 data: 2 2
4099 281/300 Read block 3559 len 32768
4100 282/300 Write block 1849 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4101 283/300 Flush
4102 284/300 Read block 151 len 32768
4103 285/300 Write block 1403 len 12288 data: 2 2 2
4104 286/300 Write block 3409 len 16384 data: 3 3 3 3
4105 287/300 Write block 3421 len 40960 data: 5 5 5 5 5 4 3 3 2 2
4106 288/300 Write block 4790 len 40960 data: 2 2 4 4 4 4 4 3 3 2
4107 289/300 Write block 1292 len 28672 data: 3 3 3 3 4 4 2
4108 290/300 Read block 281 len 16384
4109 291/300 Flush
4110 292/300 Read block 371 len 4096
4111 293/300 Read block 326 len 12288
4112 294/300 Read block 1311 len 28672
4113 295/300 Write block 2682 len 28672 data: 2 2 2 2 2 3 3
4114 296/300 Read block 3034 len 36864
4115 297/300 Write block 4261 len 28672 data: 3 3 3 3 2 2 2
4116 298/300 Write block 2452 len 24576 data: 2 2 2 2 2 2
4117 299/300 Read block 4992 len 28672
4118 300/300 Write block 3400 len 24576 data: 2 2 2 2 2 2
4119 301/300 Write block 4921 len 32768 data: 2 2 3 3 3 3 3 4
4120 Main task has work to do, go find it
4121 got dsc Start(1)
4122 start 1
4123 Mar 17 22:35:25.841 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/1, method: POST, req_id: fa846b8c-5a5f-4b92-89f9-7f3a166862f7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
4124 Replay: started 1, returned:Ok(())
4125 ----------------------------------------------------------------
4126 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4127 States: Active Offline Active
4128 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4129 1906 Acked Read 3 Done Sent Done false
4130 1907 Acked Read 1 Done Sent Done false
4131 1908 Acked Read 10 Done Sent Done false
4132 1909 Acked Read 8 Done Sent Done false
4133 1910 Acked Read 1 Done Sent Done false
4134 1911 Acked Read 4 Done Sent Done false
4135 1912 Acked Read 5 Done Sent Done false
4136 1913 Acked Read 2 Done Sent Done false
4137 1914 Acked Write 1 Done Sent Done false
4138 1915 Acked Read 1 Done Sent Done false
4139 1916 Acked Write 10 Done Sent Done false
4140 1917 Acked Write 5 Done Sent Done false
4141 1918 Acked Read 10 Done Sent Done false
4142 1919 Acked Read 4 Done Sent Done false
4143 1920 Acked Write 8 Done Sent Done false
4144 1921 Acked Write 2 Done Sent Done false
4145 1922 Acked Read 10 Done Sent Done false
4146 1923 Acked Read 4 Done Sent Done false
4147 1924 Acked Write 4 Done Sent Done false
4148 1925 Acked Write 8 Done Sent Done false
4149 1926 Acked Write 4 Done Sent Done false
4150 1927 Acked Read 3 Done Sent Done false
4151 1928 Acked Write 7 Done Sent Done false
4152 1929 Acked Read 3 Done Sent Done false
4153 1930 Acked Write 1 Done Sent Done false
4154 1931 Acked Read 1 Done Sent Done false
4155 1932 Acked Read 9 Done Sent Done false
4156 1933 Acked Read 1 Done Sent Done false
4157 1934 Acked Write 3 Done Sent Done false
4158 1935 Acked Read 10 Done Sent Done false
4159 1936 Acked Read 10 Done Sent Done false
4160 1937 Acked Read 1 Done Sent Done false
4161 1938 Acked Read 2 Done Sent Done false
4162 1939 Acked Read 2 Done Sent Done false
4163 1940 Acked Read 7 Done Sent Done false
4164 1941 Acked Read 1 Done Sent Done false
4165 1942 Acked Flush 0 Done Sent Done false
4166 1943 Acked Read 10 Done Sent Done false
4167 1944 Acked Write 2 Done Sent Done false
4168 1945 Acked Flush 0 Done Sent Done false
4169 1946 Acked Write 4 Done Sent Done false
4170 1947 Acked Read 5 Done Sent Done false
4171 1948 Acked Write 4 Done Sent Done false
4172 1949 Acked Read 10 Done Sent Done false
4173 1950 Acked Flush 0 Done Sent Done false
4174 1951 Acked Read 8 Done Sent Done false
4175 1952 Acked Flush 0 Done Sent Done false
4176 1953 Acked Read 10 Done Sent Done false
4177 1954 Acked Read 10 Done Sent Done false
4178 1955 Acked Read 5 Done Sent Done false
4179 1956 Acked Read 8 Done Sent Done false
4180 1957 Acked Flush 0 Done Sent Done false
4181 1958 Acked Flush 0 Done Sent Done false
4182 1959 Acked Write 3 Done Sent Done false
4183 1960 Acked Write 2 Done Sent Done false
4184 1961 Acked Read 7 Done Sent Done false
4185 1962 Acked Read 10 Done Sent Done false
4186 1963 Acked Write 5 Done Sent Done false
4187 1964 Acked Flush 0 Done Sent Done false
4188 1965 Acked Read 1 Done Sent Done false
4189 1966 Acked Write 7 Done Sent Done false
4190 1967 Acked Read 2 Done Sent Done false
4191 1968 Acked Write 9 Done Sent Done false
4192 1969 Acked Write 3 Done Sent Done false
4193 1970 Acked Read 3 Done Sent Done false
4194 1971 Acked Read 7 Done Sent Done false
4195 1972 Acked Read 2 Done Sent Done false
4196 1973 Acked Read 10 Done Sent Done false
4197 1974 Acked Read 4 Done Sent Done false
4198 1975 Acked Write 2 Done Sent Done false
4199 1976 Acked Read 1 Done Sent Done false
4200 1977 Acked Write 3 Done Sent Done false
4201 1978 Acked Write 5 Done Sent Done false
4202 1979 Acked Read 3 Done Sent Done false
4203 1980 Acked Write 7 Done Sent Done false
4204 1981 Acked Flush 0 Done Sent Done false
4205 1982 Acked Read 8 Done Sent Done false
4206 1983 Acked Write 9 Done Sent Done false
4207 1984 Acked Read 4 Done Sent Done false
4208 1985 Acked Write 6 Done Sent Done false
4209 1986 Acked Read 8 Done Sent Done false
4210 1987 Acked Read 5 Done Sent Done false
4211 1988 Acked Read 7 Done Sent Done false
4212 1989 Acked Read 2 Done Sent Done false
4213 1990 Acked Flush 0 Done Sent Done false
4214 1991 Acked Read 4 Done Sent Done false
4215 1992 Acked Write 5 Done Sent Done false
4216 1993 Acked Write 2 Done Sent Done false
4217 1994 Acked Write 2 Done Sent Done false
4218 1995 Acked Read 7 Done Sent Done false
4219 1996 Acked Flush 0 Done Sent Done false
4220 1997 Acked Flush 0 Done Sent Done false
4221 1998 Acked Write 6 Done Sent Done false
4222 1999 Acked Write 6 Done Sent Done false
4223 2000 Acked Read 10 Done Sent Done false
4224 2001 Acked Read 2 Done Sent Done false
4225 2002 Acked Read 6 Done Sent Done false
4226 2003 Acked Read 5 Done Sent Done false
4227 2004 Acked Write 2 Done Sent Done false
4228 2005 Acked Read 1 Done Sent Done false
4229 2006 Acked Write 1 Done Sent Done false
4230 2007 Acked Write 9 Done Sent Done false
4231 2008 Acked Flush 0 Done Sent Done false
4232 2009 Acked Read 5 Done Sent Done false
4233 2010 Acked Read 5 Done Sent Done false
4234 2011 Acked Write 3 Done Sent Done false
4235 2012 Acked Write 8 Done Sent Done false
4236 2013 Acked Read 8 Done Sent Done false
4237 2014 Acked Write 1 Done Sent Done false
4238 2015 Acked Write 5 Done Sent Done false
4239 2016 Acked Read 2 Done Sent Done false
4240 2017 Acked Write 3 Done Sent Done false
4241 2018 Acked Write 9 Done Sent Done false
4242 2019 Acked Write 9 Done Sent Done false
4243 2020 Acked Read 9 Done Sent Done false
4244 2021 Acked Read 6 Done Sent Done false
4245 2022 Acked Read 3 Done Sent Done false
4246 2023 Acked Read 1 Done Sent Done false
4247 2024 Acked Read 7 Done Sent Done false
4248 2025 Acked Write 5 Done Sent Done false
4249 2026 Acked Write 7 Done Sent Done false
4250 2027 Acked Write 3 Done Sent Done false
4251 2028 Acked Read 4 Done Sent Done false
4252 2029 Acked Read 1 Done Sent Done false
4253 2030 Acked Read 2 Done Sent Done false
4254 2031 Acked Write 6 Done Sent Done false
4255 2032 Acked Read 6 Done Sent Done false
4256 2033 Acked Write 8 Done Sent Done false
4257 2034 Acked Write 10 Done Sent Done false
4258 2035 Acked Write 6 Done Sent Done false
4259 2036 Acked Read 5 Done Sent Done false
4260 2037 Acked Write 8 Done Sent Done false
4261 2038 Acked Read 10 Done Sent Done false
4262 2039 Acked Read 5 Done Sent Done false
4263 2040 Acked Read 4 Done Sent Done false
4264 2041 Acked Read 5 Done Sent Done false
4265 2042 Acked Write 3 Done Sent Done false
4266 2043 Acked Read 3 Done Sent Done false
4267 2044 Acked Write 6 Done Sent Done false
4268 2045 Acked Write 3 Done Sent Done false
4269 2046 Acked Flush 0 Done Sent Done false
4270 2047 Acked Write 8 Done Sent Done false
4271 2048 Acked Read 8 Done Sent Done false
4272 2049 Acked Flush 0 Done Sent Done false
4273 2050 Acked Flush 0 Done Sent Done false
4274 2051 Acked Read 2 Done Sent Done false
4275 2052 Acked Read 4 Done Sent Done false
4276 2053 Acked Read 5 Done Sent Done false
4277 2054 Acked Write 2 Done Sent Done false
4278 2055 Acked Flush 0 Done Sent Done false
4279 2056 Acked Write 9 Done Sent Done false
4280 2057 Acked Flush 0 Done Sent Done false
4281 2058 Acked Read 9 Done Sent Done false
4282 2059 Acked Write 7 Done Sent Done false
4283 2060 Acked Flush 0 Done Sent Done false
4284 2061 Acked Write 3 Done Sent Done false
4285 2062 Acked Write 3 Done Sent Done false
4286 2063 Acked Read 8 Done Sent Done false
4287 2064 Acked Write 9 Done Sent Done false
4288 2065 Acked Read 3 Done Sent Done false
4289 2066 Acked Read 8 Done Sent Done false
4290 2067 Acked Write 6 Done Sent Done false
4291 2068 Acked Read 10 Done Sent Done false
4292 2069 Acked Write 8 Done Sent Done false
4293 2070 Acked Write 8 Done Sent Done false
4294 2071 Acked Read 9 Done Sent Done false
4295 2072 Acked Read 10 Done Sent Done false
4296 2073 Acked Read 7 Done Sent Done false
4297 2074 Acked Read 2 Done Sent Done false
4298 2075 Acked Write 9 Done Sent Done false
4299 2076 Acked Flush 0 Done Sent Done false
4300 2077 Acked Read 7 Done Sent Done false
4301 2078 Acked Read 9 Done Sent Done false
4302 2079 Acked Flush 0 Done Sent Done false
4303 2080 Acked Read 3 Done Sent Done false
4304 2081 Acked Read 9 Done Sent Done false
4305 2082 Acked Read 2 Done Sent Done false
4306 2083 Acked Write 6 Done Sent Done false
4307 2084 Acked Write 8 Done Sent Done false
4308 2085 Acked Read 2 Done Sent Done false
4309 2086 Acked Write 10 Done Sent Done false
4310 2087 Acked Read 7 Done Sent Done false
4311 2088 Acked Read 3 Done Sent Done false
4312 2089 Acked Read 4 Done Sent Done false
4313 2090 Acked Write 1 Done Sent Done false
4314 2091 Acked Write 9 Done Sent Done false
4315 2092 Acked Flush 0 Done Sent Done false
4316 2093 Acked Flush 0 Done Sent Done false
4317 2094 Acked Read 5 Done Sent Done false
4318 2095 Acked Write 9 Done Sent Done false
4319 2096 Acked Write 6 Done Sent Done false
4320 2097 Acked Read 10 Done Sent Done false
4321 2098 Acked Write 1 Done Sent Done false
4322 2099 Acked Read 7 Done Sent Done false
4323 2100 Acked Read 6 Done Sent Done false
4324 2101 Acked Flush 0 Done Sent Done false
4325 2102 Acked Write 6 Done Sent Done false
4326 2103 Acked Read 4 Done Sent Done false
4327 2104 Acked Read 10 Done Sent Done false
4328 2105 Acked Write 3 Done Sent Done false
4329 2106 Acked Write 4 Done Sent Done false
4330 2107 Acked Read 4 Done Sent Done false
4331 2108 Acked Read 5 Done Sent Done false
4332 2109 Acked Write 7 Done Sent Done false
4333 2110 Acked Flush 0 Done Sent Done false
4334 2111 Acked Read 5 Done Sent Done false
4335 2112 Acked Write 8 Done Sent Done false
4336 2113 Acked Flush 0 Done Sent Done false
4337 2114 Acked Write 10 Done Sent Done false
4338 2115 Acked Write 4 Done Sent Done false
4339 2116 Acked Flush 0 Done Sent Done false
4340 2117 Acked Write 4 Done Sent Done false
4341 2118 Acked Write 1 Done Sent Done false
4342 2119 Acked Write 4 Done Sent Done false
4343 2120 Acked Write 10 Done Sent Done false
4344 2121 Acked Read 7 Done Sent Done false
4345 2122 Acked Write 7 Done Sent Done false
4346 2123 Acked Read 9 Done Sent Done false
4347 2124 Acked Write 5 Done Sent Done false
4348 2125 Acked Read 10 Done Sent Done false
4349 2126 Acked Write 4 Done Sent Done false
4350 2127 Acked Flush 0 Done Sent Done false
4351 2128 Acked Read 5 Done Sent Done false
4352 2129 Acked Read 5 Done Sent Done false
4353 2130 Acked Read 9 Done Sent Done false
4354 2131 Acked Read 9 Done Sent Done false
4355 2132 Acked Write 3 Done Sent Done false
4356 2133 Acked Read 4 Done Sent Done false
4357 2134 Acked Write 6 Done Sent Done false
4358 2135 Acked Write 2 Done Sent Done false
4359 2136 Acked Flush 0 Done Sent Done false
4360 2137 Acked Read 6 Done Sent Done false
4361 2138 Acked Write 1 Done Sent Done false
4362 2139 Acked Flush 0 Done Sent Done false
4363 2140 Acked Read 3 Done Sent Done false
4364 2141 Acked Read 3 Done Sent Done false
4365 2142 Acked Read 10 Done Sent Done false
4366 2143 Acked Write 8 Done Sent Done false
4367 2144 Acked Flush 0 Done Sent Done false
4368 2145 Acked Flush 0 Done Sent Done false
4369 2146 Acked Read 8 Done Sent Done false
4370 2147 Acked Read 3 Done Sent Done false
4371 2148 Acked Read 3 Done Sent Done false
4372 2149 Acked Write 6 Done Sent Done false
4373 2150 Acked Flush 0 Done Sent Done false
4374 2151 Acked Read 9 Done Sent Done false
4375 2152 Acked Read 5 Done Sent Done false
4376 2153 Acked Read 7 Done Sent Done false
4377 2154 Acked Read 2 Done Sent Done false
4378 2155 Acked Read 7 Done Sent Done false
4379 2156 Acked Write 10 Done Sent Done false
4380 2157 Acked Write 7 Done Sent Done false
4381 2158 Acked Write 6 Done Sent Done false
4382 2159 Acked Read 7 Done Sent Done false
4383 2160 Acked Read 7 Done Sent Done false
4384 2161 Acked Write 7 Done Sent Done false
4385 2162 Acked Read 1 Done Sent Done false
4386 2163 Acked Flush 0 Done Sent Done false
4387 2164 Acked Read 3 Done Sent Done false
4388 2165 Acked Read 7 Done Sent Done false
4389 2166 Acked Read 8 Done Sent Done false
4390 2167 Acked Read 1 Done Sent Done false
4391 2168 Acked Read 9 Done Sent Done false
4392 2169 Acked Flush 0 Done Sent Done false
4393 2170 Acked Write 7 Done Sent Done false
4394 2171 Acked Read 8 Done Sent Done false
4395 2172 Acked Read 10 Done Sent Done false
4396 2173 Acked Read 4 Done Sent Done false
4397 2174 Acked Write 7 Done Sent Done false
4398 2175 Acked Read 1 Done Sent Done false
4399 2176 Acked Write 1 Done Sent Done false
4400 2177 Acked Write 3 Done Sent Done false
4401 2178 Acked Write 2 Done Sent Done false
4402 2179 Acked Read 7 Done Sent Done false
4403 2180 Acked Write 6 Done Sent Done false
4404 2181 Acked Flush 0 Done Sent Done false
4405 2182 Acked Read 10 Done Sent Done false
4406 2183 Acked Flush 0 Done Sent Done false
4407 2184 Acked Read 6 Done Sent Done false
4408 2185 Acked Write 2 Done Sent Done false
4409 2186 Acked Read 8 Done Sent Done false
4410 2187 Acked Write 10 Done Sent Done false
4411 2188 Acked Flush 0 Done Sent Done false
4412 2189 Acked Read 8 Done Sent Done false
4413 2190 Acked Write 3 Done Sent Done false
4414 2191 Acked Write 4 Done Sent Done false
4415 2192 Acked Write 10 Done Sent Done false
4416 2193 Acked Write 10 Done Sent Done false
4417 2194 Acked Write 7 Done Sent Done false
4418 2195 Acked Read 4 Done Sent Done false
4419 2196 Acked Flush 0 Done Sent Done false
4420 2197 Acked Read 1 Done Sent Done false
4421 2198 Acked Read 3 Done Sent Done false
4422 2199 Acked Read 7 Done Sent Done false
4423 2200 Acked Write 7 Done Sent Done false
4424 2201 Acked Read 9 Done Sent Done false
4425 2202 Acked Write 7 Done Sent Done false
4426 2203 Acked Write 6 Done Sent Done false
4427 2204 Acked Read 7 Done Sent Done false
4428 2205 Acked Write 6 Done Sent Done false
4429 2206 Acked Write 8 Done Sent Done false
4430 STATES DS:0 DS:1 DS:2 TOTAL
4431 Sent 0 301 0 301
4432 Done 301 0 301 602
4433 Skipped 0 0 0 0
4434 Error 0 0 0 0
4435 Last Flush: Some(JobId(2196)) Some(JobId(1905)) Some(JobId(2196))
4436 Downstairs last five completed: 1905 1904 1903 1902 1901
4437 Upstairs last five completed: 2206 2205 2204 2203 2202
4438 CLIENT: Up:0 ds:301 act:2
44392025-03-17T22:35:26.384ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
44402025-03-17T22:35:27.386ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
4441 [8820] got start action
4442 [8820] I am going to restart
4443 Starting downstairs at port 8820
4444 Make output file at "/tmp/dsc/downstairs-8820.txt"
4445 [8820][1] reports Starting
4446 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
4447 [8820][1] reports Running
44482025-03-17T22:35:28.388ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
4449 ----------------------------------------------------------------
4450 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4451 States: Active Offline Active
4452 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4453 1906 Acked Read 3 Done Sent Done false
4454 1907 Acked Read 1 Done Sent Done false
4455 1908 Acked Read 10 Done Sent Done false
4456 1909 Acked Read 8 Done Sent Done false
4457 1910 Acked Read 1 Done Sent Done false
4458 1911 Acked Read 4 Done Sent Done false
4459 1912 Acked Read 5 Done Sent Done false
4460 1913 Acked Read 2 Done Sent Done false
4461 1914 Acked Write 1 Done Sent Done false
4462 1915 Acked Read 1 Done Sent Done false
4463 1916 Acked Write 10 Done Sent Done false
4464 1917 Acked Write 5 Done Sent Done false
4465 1918 Acked Read 10 Done Sent Done false
4466 1919 Acked Read 4 Done Sent Done false
4467 1920 Acked Write 8 Done Sent Done false
4468 1921 Acked Write 2 Done Sent Done false
4469 1922 Acked Read 10 Done Sent Done false
4470 1923 Acked Read 4 Done Sent Done false
4471 1924 Acked Write 4 Done Sent Done false
4472 1925 Acked Write 8 Done Sent Done false
4473 1926 Acked Write 4 Done Sent Done false
4474 1927 Acked Read 3 Done Sent Done false
4475 1928 Acked Write 7 Done Sent Done false
4476 1929 Acked Read 3 Done Sent Done false
4477 1930 Acked Write 1 Done Sent Done false
4478 1931 Acked Read 1 Done Sent Done false
4479 1932 Acked Read 9 Done Sent Done false
4480 1933 Acked Read 1 Done Sent Done false
4481 1934 Acked Write 3 Done Sent Done false
4482 1935 Acked Read 10 Done Sent Done false
4483 1936 Acked Read 10 Done Sent Done false
4484 1937 Acked Read 1 Done Sent Done false
4485 1938 Acked Read 2 Done Sent Done false
4486 1939 Acked Read 2 Done Sent Done false
4487 1940 Acked Read 7 Done Sent Done false
4488 1941 Acked Read 1 Done Sent Done false
4489 1942 Acked Flush 0 Done Sent Done false
4490 1943 Acked Read 10 Done Sent Done false
4491 1944 Acked Write 2 Done Sent Done false
4492 1945 Acked Flush 0 Done Sent Done false
4493 1946 Acked Write 4 Done Sent Done false
4494 1947 Acked Read 5 Done Sent Done false
4495 1948 Acked Write 4 Done Sent Done false
4496 1949 Acked Read 10 Done Sent Done false
4497 1950 Acked Flush 0 Done Sent Done false
4498 1951 Acked Read 8 Done Sent Done false
4499 1952 Acked Flush 0 Done Sent Done false
4500 1953 Acked Read 10 Done Sent Done false
4501 1954 Acked Read 10 Done Sent Done false
4502 1955 Acked Read 5 Done Sent Done false
4503 1956 Acked Read 8 Done Sent Done false
4504 1957 Acked Flush 0 Done Sent Done false
4505 1958 Acked Flush 0 Done Sent Done false
4506 1959 Acked Write 3 Done Sent Done false
4507 1960 Acked Write 2 Done Sent Done false
4508 1961 Acked Read 7 Done Sent Done false
4509 1962 Acked Read 10 Done Sent Done false
4510 1963 Acked Write 5 Done Sent Done false
4511 1964 Acked Flush 0 Done Sent Done false
4512 1965 Acked Read 1 Done Sent Done false
4513 1966 Acked Write 7 Done Sent Done false
4514 1967 Acked Read 2 Done Sent Done false
4515 1968 Acked Write 9 Done Sent Done false
4516 1969 Acked Write 3 Done Sent Done false
4517 1970 Acked Read 3 Done Sent Done false
4518 1971 Acked Read 7 Done Sent Done false
4519 1972 Acked Read 2 Done Sent Done false
4520 1973 Acked Read 10 Done Sent Done false
4521 1974 Acked Read 4 Done Sent Done false
4522 1975 Acked Write 2 Done Sent Done false
4523 1976 Acked Read 1 Done Sent Done false
4524 1977 Acked Write 3 Done Sent Done false
4525 1978 Acked Write 5 Done Sent Done false
4526 1979 Acked Read 3 Done Sent Done false
4527 1980 Acked Write 7 Done Sent Done false
4528 1981 Acked Flush 0 Done Sent Done false
4529 1982 Acked Read 8 Done Sent Done false
4530 1983 Acked Write 9 Done Sent Done false
4531 1984 Acked Read 4 Done Sent Done false
4532 1985 Acked Write 6 Done Sent Done false
4533 1986 Acked Read 8 Done Sent Done false
4534 1987 Acked Read 5 Done Sent Done false
4535 1988 Acked Read 7 Done Sent Done false
4536 1989 Acked Read 2 Done Sent Done false
4537 1990 Acked Flush 0 Done Sent Done false
4538 1991 Acked Read 4 Done Sent Done false
4539 1992 Acked Write 5 Done Sent Done false
4540 1993 Acked Write 2 Done Sent Done false
4541 1994 Acked Write 2 Done Sent Done false
4542 1995 Acked Read 7 Done Sent Done false
4543 1996 Acked Flush 0 Done Sent Done false
4544 1997 Acked Flush 0 Done Sent Done false
4545 1998 Acked Write 6 Done Sent Done false
4546 1999 Acked Write 6 Done Sent Done false
4547 2000 Acked Read 10 Done Sent Done false
4548 2001 Acked Read 2 Done Sent Done false
4549 2002 Acked Read 6 Done Sent Done false
4550 2003 Acked Read 5 Done Sent Done false
4551 2004 Acked Write 2 Done Sent Done false
4552 2005 Acked Read 1 Done Sent Done false
4553 2006 Acked Write 1 Done Sent Done false
4554 2007 Acked Write 9 Done Sent Done false
4555 2008 Acked Flush 0 Done Sent Done false
4556 2009 Acked Read 5 Done Sent Done false
4557 2010 Acked Read 5 Done Sent Done false
4558 2011 Acked Write 3 Done Sent Done false
4559 2012 Acked Write 8 Done Sent Done false
4560 2013 Acked Read 8 Done Sent Done false
4561 2014 Acked Write 1 Done Sent Done false
4562 2015 Acked Write 5 Done Sent Done false
4563 2016 Acked Read 2 Done Sent Done false
4564 2017 Acked Write 3 Done Sent Done false
4565 2018 Acked Write 9 Done Sent Done false
4566 2019 Acked Write 9 Done Sent Done false
4567 2020 Acked Read 9 Done Sent Done false
4568 2021 Acked Read 6 Done Sent Done false
4569 2022 Acked Read 3 Done Sent Done false
4570 2023 Acked Read 1 Done Sent Done false
4571 2024 Acked Read 7 Done Sent Done false
4572 2025 Acked Write 5 Done Sent Done false
4573 2026 Acked Write 7 Done Sent Done false
4574 2027 Acked Write 3 Done Sent Done false
4575 2028 Acked Read 4 Done Sent Done false
4576 2029 Acked Read 1 Done Sent Done false
4577 2030 Acked Read 2 Done Sent Done false
4578 2031 Acked Write 6 Done Sent Done false
4579 2032 Acked Read 6 Done Sent Done false
4580 2033 Acked Write 8 Done Sent Done false
4581 2034 Acked Write 10 Done Sent Done false
4582 2035 Acked Write 6 Done Sent Done false
4583 2036 Acked Read 5 Done Sent Done false
4584 2037 Acked Write 8 Done Sent Done false
4585 2038 Acked Read 10 Done Sent Done false
4586 2039 Acked Read 5 Done Sent Done false
4587 2040 Acked Read 4 Done Sent Done false
4588 2041 Acked Read 5 Done Sent Done false
4589 2042 Acked Write 3 Done Sent Done false
4590 2043 Acked Read 3 Done Sent Done false
4591 2044 Acked Write 6 Done Sent Done false
4592 2045 Acked Write 3 Done Sent Done false
4593 2046 Acked Flush 0 Done Sent Done false
4594 2047 Acked Write 8 Done Sent Done false
4595 2048 Acked Read 8 Done Sent Done false
4596 2049 Acked Flush 0 Done Sent Done false
4597 2050 Acked Flush 0 Done Sent Done false
4598 2051 Acked Read 2 Done Sent Done false
4599 2052 Acked Read 4 Done Sent Done false
4600 2053 Acked Read 5 Done Sent Done false
4601 2054 Acked Write 2 Done Sent Done false
4602 2055 Acked Flush 0 Done Sent Done false
4603 2056 Acked Write 9 Done Sent Done false
4604 2057 Acked Flush 0 Done Sent Done false
4605 2058 Acked Read 9 Done Sent Done false
4606 2059 Acked Write 7 Done Sent Done false
4607 2060 Acked Flush 0 Done Sent Done false
4608 2061 Acked Write 3 Done Sent Done false
4609 2062 Acked Write 3 Done Sent Done false
4610 2063 Acked Read 8 Done Sent Done false
4611 2064 Acked Write 9 Done Sent Done false
4612 2065 Acked Read 3 Done Sent Done false
4613 2066 Acked Read 8 Done Sent Done false
4614 2067 Acked Write 6 Done Sent Done false
4615 2068 Acked Read 10 Done Sent Done false
4616 2069 Acked Write 8 Done Sent Done false
4617 2070 Acked Write 8 Done Sent Done false
4618 2071 Acked Read 9 Done Sent Done false
4619 2072 Acked Read 10 Done Sent Done false
4620 2073 Acked Read 7 Done Sent Done false
4621 2074 Acked Read 2 Done Sent Done false
4622 2075 Acked Write 9 Done Sent Done false
4623 2076 Acked Flush 0 Done Sent Done false
4624 2077 Acked Read 7 Done Sent Done false
4625 2078 Acked Read 9 Done Sent Done false
4626 2079 Acked Flush 0 Done Sent Done false
4627 2080 Acked Read 3 Done Sent Done false
4628 2081 Acked Read 9 Done Sent Done false
4629 2082 Acked Read 2 Done Sent Done false
4630 2083 Acked Write 6 Done Sent Done false
4631 2084 Acked Write 8 Done Sent Done false
4632 2085 Acked Read 2 Done Sent Done false
4633 2086 Acked Write 10 Done Sent Done false
4634 2087 Acked Read 7 Done Sent Done false
4635 2088 Acked Read 3 Done Sent Done false
4636 2089 Acked Read 4 Done Sent Done false
4637 2090 Acked Write 1 Done Sent Done false
4638 2091 Acked Write 9 Done Sent Done false
4639 2092 Acked Flush 0 Done Sent Done false
4640 2093 Acked Flush 0 Done Sent Done false
4641 2094 Acked Read 5 Done Sent Done false
4642 2095 Acked Write 9 Done Sent Done false
4643 2096 Acked Write 6 Done Sent Done false
4644 2097 Acked Read 10 Done Sent Done false
4645 2098 Acked Write 1 Done Sent Done false
4646 2099 Acked Read 7 Done Sent Done false
4647 2100 Acked Read 6 Done Sent Done false
4648 2101 Acked Flush 0 Done Sent Done false
4649 2102 Acked Write 6 Done Sent Done false
4650 2103 Acked Read 4 Done Sent Done false
4651 2104 Acked Read 10 Done Sent Done false
4652 2105 Acked Write 3 Done Sent Done false
4653 2106 Acked Write 4 Done Sent Done false
4654 2107 Acked Read 4 Done Sent Done false
4655 2108 Acked Read 5 Done Sent Done false
4656 2109 Acked Write 7 Done Sent Done false
4657 2110 Acked Flush 0 Done Sent Done false
4658 2111 Acked Read 5 Done Sent Done false
4659 2112 Acked Write 8 Done Sent Done false
4660 2113 Acked Flush 0 Done Sent Done false
4661 2114 Acked Write 10 Done Sent Done false
4662 2115 Acked Write 4 Done Sent Done false
4663 2116 Acked Flush 0 Done Sent Done false
4664 2117 Acked Write 4 Done Sent Done false
4665 2118 Acked Write 1 Done Sent Done false
4666 2119 Acked Write 4 Done Sent Done false
4667 2120 Acked Write 10 Done Sent Done false
4668 2121 Acked Read 7 Done Sent Done false
4669 2122 Acked Write 7 Done Sent Done false
4670 2123 Acked Read 9 Done Sent Done false
4671 2124 Acked Write 5 Done Sent Done false
4672 2125 Acked Read 10 Done Sent Done false
4673 2126 Acked Write 4 Done Sent Done false
4674 2127 Acked Flush 0 Done Sent Done false
4675 2128 Acked Read 5 Done Sent Done false
4676 2129 Acked Read 5 Done Sent Done false
4677 2130 Acked Read 9 Done Sent Done false
4678 2131 Acked Read 9 Done Sent Done false
4679 2132 Acked Write 3 Done Sent Done false
4680 2133 Acked Read 4 Done Sent Done false
4681 2134 Acked Write 6 Done Sent Done false
4682 2135 Acked Write 2 Done Sent Done false
4683 2136 Acked Flush 0 Done Sent Done false
4684 2137 Acked Read 6 Done Sent Done false
4685 2138 Acked Write 1 Done Sent Done false
4686 2139 Acked Flush 0 Done Sent Done false
4687 2140 Acked Read 3 Done Sent Done false
4688 2141 Acked Read 3 Done Sent Done false
4689 2142 Acked Read 10 Done Sent Done false
4690 2143 Acked Write 8 Done Sent Done false
4691 2144 Acked Flush 0 Done Sent Done false
4692 2145 Acked Flush 0 Done Sent Done false
4693 2146 Acked Read 8 Done Sent Done false
4694 2147 Acked Read 3 Done Sent Done false
4695 2148 Acked Read 3 Done Sent Done false
4696 2149 Acked Write 6 Done Sent Done false
4697 2150 Acked Flush 0 Done Sent Done false
4698 2151 Acked Read 9 Done Sent Done false
4699 2152 Acked Read 5 Done Sent Done false
4700 2153 Acked Read 7 Done Sent Done false
4701 2154 Acked Read 2 Done Sent Done false
4702 2155 Acked Read 7 Done Sent Done false
4703 2156 Acked Write 10 Done Sent Done false
4704 2157 Acked Write 7 Done Sent Done false
4705 2158 Acked Write 6 Done Sent Done false
4706 2159 Acked Read 7 Done Sent Done false
4707 2160 Acked Read 7 Done Sent Done false
4708 2161 Acked Write 7 Done Sent Done false
4709 2162 Acked Read 1 Done Sent Done false
4710 2163 Acked Flush 0 Done Sent Done false
4711 2164 Acked Read 3 Done Sent Done false
4712 2165 Acked Read 7 Done Sent Done false
4713 2166 Acked Read 8 Done Sent Done false
4714 2167 Acked Read 1 Done Sent Done false
4715 2168 Acked Read 9 Done Sent Done false
4716 2169 Acked Flush 0 Done Sent Done false
4717 2170 Acked Write 7 Done Sent Done false
4718 2171 Acked Read 8 Done Sent Done false
4719 2172 Acked Read 10 Done Sent Done false
4720 2173 Acked Read 4 Done Sent Done false
4721 2174 Acked Write 7 Done Sent Done false
4722 2175 Acked Read 1 Done Sent Done false
4723 2176 Acked Write 1 Done Sent Done false
4724 2177 Acked Write 3 Done Sent Done false
4725 2178 Acked Write 2 Done Sent Done false
4726 2179 Acked Read 7 Done Sent Done false
4727 2180 Acked Write 6 Done Sent Done false
4728 2181 Acked Flush 0 Done Sent Done false
4729 2182 Acked Read 10 Done Sent Done false
4730 2183 Acked Flush 0 Done Sent Done false
4731 2184 Acked Read 6 Done Sent Done false
4732 2185 Acked Write 2 Done Sent Done false
4733 2186 Acked Read 8 Done Sent Done false
4734 2187 Acked Write 10 Done Sent Done false
4735 2188 Acked Flush 0 Done Sent Done false
4736 2189 Acked Read 8 Done Sent Done false
4737 2190 Acked Write 3 Done Sent Done false
4738 2191 Acked Write 4 Done Sent Done false
4739 2192 Acked Write 10 Done Sent Done false
4740 2193 Acked Write 10 Done Sent Done false
4741 2194 Acked Write 7 Done Sent Done false
4742 2195 Acked Read 4 Done Sent Done false
4743 2196 Acked Flush 0 Done Sent Done false
4744 2197 Acked Read 1 Done Sent Done false
4745 2198 Acked Read 3 Done Sent Done false
4746 2199 Acked Read 7 Done Sent Done false
4747 2200 Acked Write 7 Done Sent Done false
4748 2201 Acked Read 9 Done Sent Done false
4749 2202 Acked Write 7 Done Sent Done false
4750 2203 Acked Write 6 Done Sent Done false
4751 2204 Acked Read 7 Done Sent Done false
4752 2205 Acked Write 6 Done Sent Done false
4753 2206 Acked Write 8 Done Sent Done false
4754 STATES DS:0 DS:1 DS:2 TOTAL
4755 Sent 0 301 0 301
4756 Done 301 0 301 602
4757 Skipped 0 0 0 0
4758 Error 0 0 0 0
4759 Last Flush: Some(JobId(2196)) Some(JobId(1905)) Some(JobId(2196))
4760 Downstairs last five completed: 1905 1904 1903 1902 1901
4761 Upstairs last five completed: 2206 2205 2204 2203 2202
4762 CLIENT: Up:0 ds:301 act:2
47632025-03-17T22:35:30.392ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47642025-03-17T22:35:31.394ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47652025-03-17T22:35:31.594ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47662025-03-17T22:35:31.594ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62829) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47672025-03-17T22:35:31.595ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47682025-03-17T22:35:31.596ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47692025-03-17T22:35:31.596ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47702025-03-17T22:35:31.596ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47712025-03-17T22:35:31.596ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47722025-03-17T22:35:31.596ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47732025-03-17T22:35:31.596ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
4774 ----------------------------------------------------------------
4775 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4776 Downstairs last five completed: 2207 2206 2205 2204 2203
4777 Upstairs last five completed: 2207 2206 2205 2204 2203
4778 CLIENT: Up:0 ds:0 act:3
4779 Replay: All jobs finished
4780 Main task has work to do, go find it
4781 got dsc Stop(1)
4782 stop 1
4783 Mar 17 22:35:33.846 INFO request completed, latency_us: 370, response_code: 204, uri: /stop/cid/1, method: POST[8820] Got stop action so:false kr:false
4784 , req_id: 0b035552-7601-432f-864e-a2e8122d9347, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
4785 Mar 17 22:35:33.847 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/1, method: GET, req_id: e9bffd27-1fef-4648-a5aa-da8a7be42444, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
47862025-03-17T22:35:33.847ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47872025-03-17T22:35:33.847ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47882025-03-17T22:35:33.847ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
47892025-03-17T22:35:33.847ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
4790 [8820] Exited with: ExitStatus(unix_wait_status(9))
4791 [8820][1] reports Exit
4792 Mar 17 22:35:37.850 INFO request completed, latency_us: 322, response_code: 200, uri: /state/cid/1, method: GET, req_id: c68cd090-5cae-4c51-9bc4-f4811caa287a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
4793 001/300 Read block 2690 len 28672
4794 002/300 Write block 4945 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4795 003/300 Read block 2461 len 4096
4796 004/300 Write block 1616 len 12288 data: 2 2 3
4797 005/300 Read block 3261 len 32768
4798 006/300 Read block 3291 len 4096
4799 007/300 Write block 2350 len 8192 data: 2 2
4800 008/300 Write block 4308 len 28672 data: 2 2 2 2 2 2 2
4801 009/300 Write block 484 len 20480 data: 3 2 2 2 2
4802 010/300 Write block 1169 len 4096 data: 2
4803 011/300 Read block 4203 len 24576
4804 012/300 Write block 2556 len 12288 data: 2 2 2
4805 013/300 Read block 2667 len 16384
4806 014/300 Write block 4953 len 16384 data: 3 3 2 2
4807 015/300 Write block 2917 len 4096 data: 3
4808 016/300 Read block 3213 len 24576
4809 017/300 Read block 2583 len 36864
4810 018/300 Write block 2081 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4811 019/300 Read block 4270 len 16384
4812 020/300 Flush
4813 021/300 Write block 4671 len 12288 data: 3 2 2
4814 022/300 Read block 2042 len 28672
4815 023/300 Read block 2549 len 20480
4816 024/300 Read block 2329 len 36864
4817 025/300 Write block 92 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4818 026/300 Read block 3796 len 16384
4819 027/300 Read block 1375 len 20480
4820 028/300 Read block 1233 len 16384
4821 029/300 Read block 3206 len 28672
4822 030/300 Write block 1541 len 36864 data: 2 2 2 2 3 3 3 3 2
4823 031/300 Read block 2663 len 28672
4824 032/300 Read block 3776 len 40960
4825 033/300 Read block 949 len 12288
4826 034/300 Write block 1644 len 20480 data: 3 3 3 2 2
4827 035/300 Read block 1195 len 20480
4828 036/300 Read block 2132 len 16384
4829 037/300 Read block 1405 len 28672
4830 038/300 Write block 562 len 36864 data: 2 2 2 2 2 2 2 2 2
4831 039/300 Write block 1042 len 20480 data: 2 2 2 2 2
4832 040/300 Read block 3511 len 8192
4833 041/300 Read block 4145 len 32768
4834 042/300 Flush
4835 043/300 Read block 1272 len 36864
4836 044/300 Flush
4837 045/300 Write block 3627 len 40960 data: 2 2 2 2 2 3 3 3 3 3
4838 046/300 Read block 3730 len 24576
4839 047/300 Write block 3394 len 40960 data: 2 2 2 2 2 2 3 3 3 3
4840 048/300 Read block 642 len 20480
4841 049/300 Read block 4447 len 20480
4842 050/300 Write block 1474 len 40960 data: 4 4 4 3 3 5 5 5 5 3
4843 051/300 Write block 789 len 20480 data: 2 2 2 2 2
4844 052/300 Read block 88 len 20480
4845 053/300 Write block 2986 len 4096 data: 2
4846 054/300 Write block 941 len 28672 data: 3 3 3 3 3 3 3
4847 055/300 Read block 3506 len 8192
4848 056/300 Write block 4206 len 12288 data: 2 2 2
4849 057/300 Flush
4850 058/300 Read block 1328 len 20480
4851 059/300 Write block 2869 len 36864 data: 2 2 2 2 2 2 2 2 3
4852 060/300 Read block 271 len 16384
4853 061/300 Read block 3493 len 40960
4854 062/300 Write block 2251 len 4096 data: 2
4855 063/300 Read block 2777 len 24576
4856 064/300 Read block 3839 len 20480
4857 065/300 Read block 1351 len 32768
4858 066/300 Write block 4725 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4859 067/300 Write block 2119 len 8192 data: 2 2
4860 068/300 Write block 3933 len 28672 data: 2 2 2 3 3 3 3
4861 069/300 Write block 875 len 12288 data: 2 2 2
4862 070/300 Write block 130 len 40960 data: 3 3 3 3 3 3 3 3 3 2
4863 071/300 Read block 4295 len 4096
4864 072/300 Write block 4104 len 24576 data: 2 2 2 2 2 2
4865 073/300 Read block 619 len 32768
4866 074/300 Write block 4842 len 24576 data: 2 2 2 2 3 3
4867 075/300 Write block 894 len 4096 data: 2
4868 076/300 Read block 1531 len 8192
4869 077/300 Write block 3986 len 8192 data: 2 3
4870 078/300 Read block 408 len 12288
4871 079/300 Read block 1561 len 24576
4872 080/300 Write block 965 len 28672 data: 2 2 2 2 3 4 4
4873 081/300 Read block 1989 len 8192
4874 082/300 Read block 3734 len 20480
4875 083/300 Write block 3174 len 12288 data: 2 3 3
4876 084/300 Write block 3678 len 20480 data: 2 3 4 4 3
4877 085/300 Flush
4878 086/300 Read block 3746 len 20480
4879 087/300 Read block 1085 len 4096
4880 088/300 Read block 1690 len 24576
4881 089/300 Write block 1739 len 40960 data: 2 2 2 2 2 3 3 3 2 2
4882 090/300 Write block 3900 len 8192 data: 3 4
4883 091/300 Read block 2217 len 28672
4884 092/300 Write block 4142 len 4096 data: 2
4885 093/300 Read block 4509 len 20480
4886 094/300 Read block 4345 len 36864
4887 095/300 Read block 556 len 36864
4888 096/300 Write block 408 len 8192 data: 4 4
4889 097/300 Flush
4890 098/300 Write block 2149 len 40960 data: 3 3 3 3 2 2 2 2 2 2
4891 099/300 Read block 1532 len 12288
4892 100/300 Flush
4893 101/300 Write block 4459 len 32768 data: 2 2 2 2 2 3 3 2
4894 102/300 Write block 293 len 36864 data: 2 2 2 2 3 2 2 2 2
4895 103/300 Read block 2898 len 4096
4896 104/300 Read block 2320 len 36864
4897 105/300 Read block 1906 len 40960
4898 106/300 Read block 4950 len 32768
4899 107/300 Flush
4900 108/300 Write block 661 len 24576 data: 3 3 4 4 3 2
4901 109/300 Read block 4071 len 24576
4902 110/300 Read block 1822 len 20480
4903 111/300 Read block 37 len 36864
4904 112/300 Write block 4589 len 12288 data: 2 2 2
4905 113/300 Read block 2081 len 36864
4906 114/300 Read block 1681 len 28672
4907 115/300 Write block 3999 len 24576 data: 2 2 2 2 2 2
4908 116/300 Write block 3586 len 20480 data: 2 2 2 2 2
4909 117/300 Read block 3267 len 36864
4910 118/300 Read block 1127 len 20480
4911 119/300 Write block 3311 len 8192 data: 3 2
4912 120/300 Read block 2762 len 40960
4913 121/300 Read block 3665 len 20480
4914 122/300 Read block 1709 len 20480
4915 123/300 Read block 2895 len 36864
4916 124/300 Write block 3996 len 12288 data: 3 2 2
4917 125/300 Write block 4261 len 32768 data: 4 4 4 4 3 3 3 2
4918 126/300 Read block 3904 len 12288
4919 127/300 Flush
4920 128/300 Read block 1335 len 16384
4921 129/300 Write block 3976 len 12288 data: 2 2 2
4922 130/300 Read block 3050 len 8192
4923 131/300 Write block 1637 len 16384 data: 2 2 2 2
4924 132/300 Write block 1721 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4925 133/300 Write block 3445 len 36864 data: 2 2 2 2 2 2 2 2 2
4926 134/300 Flush
4927 135/300 Read block 507 len 28672
4928 136/300 Write block 4425 len 28672 data: 3 3 3 3 3 3 2
4929 137/300 Read block 4366 len 4096
4930 138/300 Write block 4262 len 32768 data: 5 5 5 4 4 4 3 2
4931 139/300 Write block 3440 len 20480 data: 3 2 2 2 2
4932 140/300 Read block 88 len 40960
4933 141/300 Read block 950 len 40960
4934 142/300 Read block 3500 len 24576
4935 143/300 Read block 9 len 36864
4936 144/300 Write block 4096 len 36864 data: 3 3 2 2 2 2 3 3 3
4937 145/300 Write block 4181 len 40960 data: 3 3 3 3 4 3 3 2 2 2
4938 146/300 Write block 1119 len 32768 data: 2 2 2 2 2 2 2 2
4939 147/300 Write block 4056 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4940 148/300 Write block 4215 len 40960 data: 2 2 2 2 2 2 2 3 2 2
4941 149/300 Write block 2496 len 4096 data: 3
4942 150/300 Write block 2205 len 4096 data: 2
4943 151/300 Read block 1962 len 40960
4944 152/300 Read block 2757 len 36864
4945 153/300 Write block 292 len 24576 data: 2 3 3 3 3 4
4946 154/300 Write block 1435 len 12288 data: 2 2 2
4947 155/300 Read block 2686 len 12288
4948 156/300 Read block 716 len 24576
4949 157/300 Read block 3554 len 20480
4950 158/300 Write block 3854 len 28672 data: 2 2 2 2 2 2 2
4951 159/300 Read block 3516 len 12288
4952 160/300 Write block 1120 len 16384 data: 3 3 3 3
4953 161/300 Write block 2254 len 24576 data: 3 3 2 2 3 3
4954 162/300 Read block 211 len 28672
4955 163/300 Read block 1172 len 4096
4956 164/300 Write block 53 len 16384 data: 3 3 3 3
4957 165/300 Read block 4019 len 16384
4958 166/300 Write block 1867 len 32768 data: 3 2 2 3 3 3 3 3
4959 167/300 Read block 2448 len 12288
4960 168/300 Read block 3873 len 36864
4961 169/300 Read block 1843 len 8192
4962 170/300 Write block 39 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4963 171/300 Read block 1884 len 20480
4964 172/300 Read block 3922 len 24576
4965 173/300 Flush
4966 174/300 Write block 1436 len 4096 data: 3
4967 175/300 Write block 4066 len 12288 data: 2 3 3
4968 176/300 Read block 4549 len 16384
4969 177/300 Write block 2481 len 32768 data: 3 3 3 2 2 2 2 2
4970 178/300 Write block 771 len 24576 data: 2 2 2 2 2 2
4971 179/300 Read block 4761 len 40960
4972 180/300 Read block 3840 len 16384
4973 181/300 Read block 1603 len 40960
4974 182/300 Write block 3091 len 36864 data: 2 2 2 2 2 2 2 2 2
4975 183/300 Write block 2369 len 4096 data: 2
4976 184/300 Read block 3176 len 40960
4977 185/300 Read block 1010 len 12288
4978 186/300 Write block 512 len 8192 data: 2 3
4979 187/300 Read block 4022 len 4096
4980 188/300 Write block 430 len 20480 data: 2 2 2 2 2
4981 189/300 Write block 1308 len 24576 data: 3 3 2 2 2 2
4982 190/300 Read block 1570 len 40960
4983 191/300 Write block 995 len 20480 data: 2 2 2 2 2
4984 192/300 Write block 3907 len 36864 data: 4 4 4 5 4 4 3 3 3
4985 193/300 Flush
4986 194/300 Read block 906 len 28672
4987 195/300 Write block 2074 len 8192 data: 3 3
4988 196/300 Write block 4017 len 36864 data: 3 2 2 2 2 3 3 3 3
4989 197/300 Flush
4990 198/300 Read block 1371 len 32768
4991 199/300 Write block 3055 len 36864 data: 3 3 3 2 2 2 2 2 2
4992 200/300 Write block 4384 len 36864 data: 2 2 2 2 2 3 3 3 3
4993 201/300 Read block 1873 len 40960
4994 202/300 Read block 2784 len 32768
4995 203/300 Write block 4123 len 4096 data: 2
4996 204/300 Read block 2249 len 20480
4997 205/300 Write block 2232 len 20480 data: 2 3 3 3 2
4998 206/300 Flush
4999 207/300 Write block 3141 len 36864 data: 2 2 2 2 2 2 2 2 3
5000 208/300 Write block 2739 len 24576 data: 2 2 2 2 2 2
5001 209/300 Read block 650 len 24576
5002 210/300 Write block 2530 len 36864 data: 2 2 2 2 3 3 3 3 3
5003 211/300 Write block 492 len 8192 data: 2 2
5004 212/300 Read block 3836 len 16384
5005 213/300 Write block 1721 len 4096 data: 3
5006 214/300 Write block 1845 len 16384 data: 2 2 2 2
5007 215/300 Write block 4064 len 36864 data: 3 3 3 4 4 3 3 3 4
5008 216/300 Write block 2291 len 20480 data: 2 2 2 2 2
5009 217/300 Write block 3682 len 16384 data: 4 3 2 2
5010 218/300 Write block 4418 len 24576 data: 2 2 2 2 2 3
5011 219/300 Read block 552 len 24576
5012 220/300 Write block 2997 len 24576 data: 3 3 3 3 3 2
5013 221/300 Read block 2519 len 24576
5014 222/300 Write block 3152 len 16384 data: 3 4 4 2
5015 223/300 Write block 4305 len 40960 data: 3 3 2 3 3 3 3 3 3 3
5016 224/300 Flush
5017 225/300 Read block 4702 len 20480
5018 226/300 Write block 54 len 20480 data: 4 4 4 3 2
5019 227/300 Write block 3392 len 16384 data: 2 2 3 3
5020 228/300 Write block 1794 len 12288 data: 2 2 2
5021 229/300 Write block 238 len 12288 data: 2 3 3
5022 230/300 Read block 4913 len 24576
5023 231/300 Write block 655 len 36864 data: 2 2 3 3 3 3 4 4 5
5024 232/300 Write block 4263 len 16384 data: 6 6 5 5
5025 233/300 Write block 307 len 20480 data: 3 4 3 3 3
5026 234/300 Read block 2681 len 28672
5027 235/300 Write block 4743 len 36864 data: 2 2 2 2 2 2 2 2 2
5028 236/300 Read block 2061 len 12288
5029 237/300 Write block 1646 len 4096 data: 4
5030 238/300 Read block 3425 len 24576
5031 239/300 Write block 4518 len 8192 data: 3 3
5032 240/300 Write block 4799 len 36864 data: 3 2 2 2 3 3 3 3 3
5033 241/300 Write block 1900 len 32768 data: 2 2 2 2 3 3 3 3
5034 242/300 Write block 1502 len 12288 data: 2 2 3
5035 243/300 Read block 4253 len 4096
5036 244/300 Read block 3686 len 8192
5037 245/300 Write block 1731 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5038 246/300 Read block 99 len 28672
5039 247/300 Read block 1047 len 4096
5040 248/300 Read block 2258 len 36864
5041 249/300 Flush
5042 250/300 Read block 405 len 16384
5043 251/300 Read block 165 len 16384
5044 252/300 Read block 2601 len 24576
5045 253/300 Read block 2380 len 40960
5046 254/300 Write block 4346 len 4096 data: 3
5047 255/300 Flush
5048 256/300 Write block 1803 len 16384 data: 2 2 2 2
5049 257/300 Write block 733 len 12288 data: 2 2 2
5050 258/300 Read block 794 len 24576
5051 259/300 Write block 4495 len 8192 data: 2 2
5052 260/300 Read block 1289 len 32768
5053 261/300 Flush
5054 262/300 Flush
5055 263/300 Write block 4527 len 28672 data: 3 3 3 3 3 3 3
5056 264/300 Read block 2224 len 36864
5057 265/300 Write block 2710 len 4096 data: 2
5058 266/300 Write block 206 len 40960 data: 2 2 2 2 2 3 2 2 2 2
5059 267/300 Write block 1577 len 28672 data: 2 2 2 2 2 2 2
5060 268/300 Read block 4045 len 40960
5061 269/300 Read block 1721 len 24576
5062 270/300 Read block 1505 len 8192
5063 271/300 Flush
5064 272/300 Read block 3077 len 20480
5065 273/300 Write block 3773 len 32768 data: 2 2 2 3 3 3 3 3
5066 274/300 Flush
5067 275/300 Write block 1118 len 4096 data: 2
5068 276/300 Write block 4422 len 36864 data: 3 4 3 4 4 4 4 4 4
5069 277/300 Write block 1415 len 16384 data: 3 3 3 3
5070 278/300 Read block 3803 len 28672
5071 279/300 Write block 1037 len 28672 data: 2 2 2 2 2 3 3
5072 280/300 Read block 4565 len 24576
5073 281/300 Write block 3401 len 8192 data: 4 4
5074 282/300 Read block 3746 len 4096
5075 283/300 Read block 185 len 40960
5076 284/300 Write block 538 len 20480 data: 2 2 2 2 2
5077 285/300 Write block 3736 len 12288 data: 2 2 2
5078 286/300 Write block 2078 len 12288 data: 2 2 2
5079 287/300 Read block 4787 len 28672
5080 288/300 Read block 2312 len 40960
5081 289/300 Read block 2302 len 8192
5082 290/300 Read block 3296 len 20480
5083 291/300 Flush
5084 292/300 Read block 2592 len 12288
5085 293/300 Read block 3763 len 32768
5086 294/300 Write block 2804 len 16384 data: 2 2 3 3
5087 295/300 Write block 64 len 24576 data: 2 3 3 3 3 3
5088 296/300 Write block 3433 len 16384 data: 3 3 3 3
5089 297/300 Read block 3369 len 12288
5090 298/300 Read block 1600 len 32768
5091 299/300 Write block 3937 len 40960 data: 4 4 4 3 3 3 3 2 2 2
5092 300/300 Write block 263 len 36864 data: 2 2 2 2 2 2 2 2 2
5093 301/300 Read block 3631 len 8192
5094 Main task has work to do, go find it
5095 got dsc Start(1)
5096 start 1
5097 Mar 17 22:35:38.092 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6faf2113-cc19-4471-834c-0ae11c7ba6ef, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
5098 Replay: started 1, returned:Ok(())
5099 ----------------------------------------------------------------
5100 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5101 States: Active Offline Active
5102 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5103 2208 Acked Read 7 Done Sent Done false
5104 2209 Acked Write 10 Done Sent Done false
5105 2210 Acked Read 1 Done Sent Done false
5106 2211 Acked Write 3 Done Sent Done false
5107 2212 Acked Read 8 Done Sent Done false
5108 2213 Acked Read 1 Done Sent Done false
5109 2214 Acked Write 2 Done Sent Done false
5110 2215 Acked Write 7 Done Sent Done false
5111 2216 Acked Write 5 Done Sent Done false
5112 2217 Acked Write 1 Done Sent Done false
5113 2218 Acked Read 6 Done Sent Done false
5114 2219 Acked Write 3 Done Sent Done false
5115 2220 Acked Read 4 Done Sent Done false
5116 2221 Acked Write 4 Done Sent Done false
5117 2222 Acked Write 1 Done Sent Done false
5118 2223 Acked Read 6 Done Sent Done false
5119 2224 Acked Read 9 Done Sent Done false
5120 2225 Acked Write 10 Done Sent Done false
5121 2226 Acked Read 4 Done Sent Done false
5122 2227 Acked Flush 0 Done Sent Done false
5123 2228 Acked Write 3 Done Sent Done false
5124 2229 Acked Read 7 Done Sent Done false
5125 2230 Acked Read 5 Done Sent Done false
5126 2231 Acked Read 9 Done Sent Done false
5127 2232 Acked Write 10 Done Sent Done false
5128 2233 Acked Read 4 Done Sent Done false
5129 2234 Acked Read 5 Done Sent Done false
5130 2235 Acked Read 4 Done Sent Done false
5131 2236 Acked Read 7 Done Sent Done false
5132 2237 Acked Write 9 Done Sent Done false
5133 2238 Acked Read 7 Done Sent Done false
5134 2239 Acked Read 10 Done Sent Done false
5135 2240 Acked Read 3 Done Sent Done false
5136 2241 Acked Write 5 Done Sent Done false
5137 2242 Acked Read 5 Done Sent Done false
5138 2243 Acked Read 4 Done Sent Done false
5139 2244 Acked Read 7 Done Sent Done false
5140 2245 Acked Write 9 Done Sent Done false
5141 2246 Acked Write 5 Done Sent Done false
5142 2247 Acked Read 2 Done Sent Done false
5143 2248 Acked Read 8 Done Sent Done false
5144 2249 Acked Flush 0 Done Sent Done false
5145 2250 Acked Read 9 Done Sent Done false
5146 2251 Acked Flush 0 Done Sent Done false
5147 2252 Acked Write 10 Done Sent Done false
5148 2253 Acked Read 6 Done Sent Done false
5149 2254 Acked Write 10 Done Sent Done false
5150 2255 Acked Read 5 Done Sent Done false
5151 2256 Acked Read 5 Done Sent Done false
5152 2257 Acked Write 10 Done Sent Done false
5153 2258 Acked Write 5 Done Sent Done false
5154 2259 Acked Read 5 Done Sent Done false
5155 2260 Acked Write 1 Done Sent Done false
5156 2261 Acked Write 7 Done Sent Done false
5157 2262 Acked Read 2 Done Sent Done false
5158 2263 Acked Write 3 Done Sent Done false
5159 2264 Acked Flush 0 Done Sent Done false
5160 2265 Acked Read 5 Done Sent Done false
5161 2266 Acked Write 9 Done Sent Done false
5162 2267 Acked Read 4 Done Sent Done false
5163 2268 Acked Read 10 Done Sent Done false
5164 2269 Acked Write 1 Done Sent Done false
5165 2270 Acked Read 6 Done Sent Done false
5166 2271 Acked Read 5 Done Sent Done false
5167 2272 Acked Read 8 Done Sent Done false
5168 2273 Acked Write 10 Done Sent Done false
5169 2274 Acked Write 2 Done Sent Done false
5170 2275 Acked Write 7 Done Sent Done false
5171 2276 Acked Write 3 Done Sent Done false
5172 2277 Acked Write 10 Done Sent Done false
5173 2278 Acked Read 1 Done Sent Done false
5174 2279 Acked Write 6 Done Sent Done false
5175 2280 Acked Read 8 Done Sent Done false
5176 2281 Acked Write 6 Done Sent Done false
5177 2282 Acked Write 1 Done Sent Done false
5178 2283 Acked Read 2 Done Sent Done false
5179 2284 Acked Write 2 Done Sent Done false
5180 2285 Acked Read 3 Done Sent Done false
5181 2286 Acked Read 6 Done Sent Done false
5182 2287 Acked Write 7 Done Sent Done false
5183 2288 Acked Read 2 Done Sent Done false
5184 2289 Acked Read 5 Done Sent Done false
5185 2290 Acked Write 3 Done Sent Done false
5186 2291 Acked Write 5 Done Sent Done false
5187 2292 Acked Flush 0 Done Sent Done false
5188 2293 Acked Read 5 Done Sent Done false
5189 2294 Acked Read 1 Done Sent Done false
5190 2295 Acked Read 6 Done Sent Done false
5191 2296 Acked Write 10 Done Sent Done false
5192 2297 Acked Write 2 Done Sent Done false
5193 2298 Acked Read 7 Done Sent Done false
5194 2299 Acked Write 1 Done Sent Done false
5195 2300 Acked Read 5 Done Sent Done false
5196 2301 Acked Read 9 Done Sent Done false
5197 2302 Acked Read 9 Done Sent Done false
5198 2303 Acked Write 2 Done Sent Done false
5199 2304 Acked Flush 0 Done Sent Done false
5200 2305 Acked Write 10 Done Sent Done false
5201 2306 Acked Read 3 Done Sent Done false
5202 2307 Acked Flush 0 Done Sent Done false
5203 2308 Acked Write 8 Done Sent Done false
5204 2309 Acked Write 9 Done Sent Done false
5205 2310 Acked Read 1 Done Sent Done false
5206 2311 Acked Read 9 Done Sent Done false
5207 2312 Acked Read 10 Done Sent Done false
5208 2313 Acked Read 8 Done Sent Done false
5209 2314 Acked Flush 0 Done Sent Done false
5210 2315 Acked Write 6 Done Sent Done false
5211 2316 Acked Read 6 Done Sent Done false
5212 2317 Acked Read 5 Done Sent Done false
5213 2318 Acked Read 9 Done Sent Done false
5214 2319 Acked Write 3 Done Sent Done false
5215 2320 Acked Read 9 Done Sent Done false
5216 2321 Acked Read 7 Done Sent Done false
5217 2322 Acked Write 6 Done Sent Done false
5218 2323 Acked Write 5 Done Sent Done false
5219 2324 Acked Read 9 Done Sent Done false
5220 2325 Acked Read 5 Done Sent Done false
5221 2326 Acked Write 2 Done Sent Done false
5222 2327 Acked Read 10 Done Sent Done false
5223 2328 Acked Read 5 Done Sent Done false
5224 2329 Acked Read 5 Done Sent Done false
5225 2330 Acked Read 9 Done Sent Done false
5226 2331 Acked Write 3 Done Sent Done false
5227 2332 Acked Write 8 Done Sent Done false
5228 2333 Acked Read 3 Done Sent Done false
5229 2334 Acked Flush 0 Done Sent Done false
5230 2335 Acked Read 4 Done Sent Done false
5231 2336 Acked Write 3 Done Sent Done false
5232 2337 Acked Read 2 Done Sent Done false
5233 2338 Acked Write 4 Done Sent Done false
5234 2339 Acked Write 10 Done Sent Done false
5235 2340 Acked Write 9 Done Sent Done false
5236 2341 Acked Flush 0 Done Sent Done false
5237 2342 Acked Read 7 Done Sent Done false
5238 2343 Acked Write 7 Done Sent Done false
5239 2344 Acked Read 1 Done Sent Done false
5240 2345 Acked Write 8 Done Sent Done false
5241 2346 Acked Write 5 Done Sent Done false
5242 2347 Acked Read 10 Done Sent Done false
5243 2348 Acked Read 10 Done Sent Done false
5244 2349 Acked Read 6 Done Sent Done false
5245 2350 Acked Read 9 Done Sent Done false
5246 2351 Acked Write 9 Done Sent Done false
5247 2352 Acked Write 10 Done Sent Done false
5248 2353 Acked Write 8 Done Sent Done false
5249 2354 Acked Write 10 Done Sent Done false
5250 2355 Acked Write 10 Done Sent Done false
5251 2356 Acked Write 1 Done Sent Done false
5252 2357 Acked Write 1 Done Sent Done false
5253 2358 Acked Read 10 Done Sent Done false
5254 2359 Acked Read 9 Done Sent Done false
5255 2360 Acked Write 6 Done Sent Done false
5256 2361 Acked Write 3 Done Sent Done false
5257 2362 Acked Read 3 Done Sent Done false
5258 2363 Acked Read 6 Done Sent Done false
5259 2364 Acked Read 5 Done Sent Done false
5260 2365 Acked Write 7 Done Sent Done false
5261 2366 Acked Read 3 Done Sent Done false
5262 2367 Acked Write 4 Done Sent Done false
5263 2368 Acked Write 6 Done Sent Done false
5264 2369 Acked Read 7 Done Sent Done false
5265 2370 Acked Read 1 Done Sent Done false
5266 2371 Acked Write 4 Done Sent Done false
5267 2372 Acked Read 4 Done Sent Done false
5268 2373 Acked Write 8 Done Sent Done false
5269 2374 Acked Read 3 Done Sent Done false
5270 2375 Acked Read 9 Done Sent Done false
5271 2376 Acked Read 2 Done Sent Done false
5272 2377 Acked Write 10 Done Sent Done false
5273 2378 Acked Read 5 Done Sent Done false
5274 2379 Acked Read 6 Done Sent Done false
5275 2380 Acked Flush 0 Done Sent Done false
5276 2381 Acked Write 1 Done Sent Done false
5277 2382 Acked Write 3 Done Sent Done false
5278 2383 Acked Read 4 Done Sent Done false
5279 2384 Acked Write 8 Done Sent Done false
5280 2385 Acked Write 6 Done Sent Done false
5281 2386 Acked Read 10 Done Sent Done false
5282 2387 Acked Read 4 Done Sent Done false
5283 2388 Acked Read 10 Done Sent Done false
5284 2389 Acked Write 9 Done Sent Done false
5285 2390 Acked Write 1 Done Sent Done false
5286 2391 Acked Read 10 Done Sent Done false
5287 2392 Acked Read 3 Done Sent Done false
5288 2393 Acked Write 2 Done Sent Done false
5289 2394 Acked Read 1 Done Sent Done false
5290 2395 Acked Write 5 Done Sent Done false
5291 2396 Acked Write 6 Done Sent Done false
5292 2397 Acked Read 10 Done Sent Done false
5293 2398 Acked Write 5 Done Sent Done false
5294 2399 Acked Write 9 Done Sent Done false
5295 2400 Acked Flush 0 Done Sent Done false
5296 2401 Acked Read 7 Done Sent Done false
5297 2402 Acked Write 2 Done Sent Done false
5298 2403 Acked Write 9 Done Sent Done false
5299 2404 Acked Flush 0 Done Sent Done false
5300 2405 Acked Read 8 Done Sent Done false
5301 2406 Acked Write 9 Done Sent Done false
5302 2407 Acked Write 9 Done Sent Done false
5303 2408 Acked Read 10 Done Sent Done false
5304 2409 Acked Read 8 Done Sent Done false
5305 2410 Acked Write 1 Done Sent Done false
5306 2411 Acked Read 5 Done Sent Done false
5307 2412 Acked Write 5 Done Sent Done false
5308 2413 Acked Flush 0 Done Sent Done false
5309 2414 Acked Write 9 Done Sent Done false
5310 2415 Acked Write 6 Done Sent Done false
5311 2416 Acked Read 6 Done Sent Done false
5312 2417 Acked Write 9 Done Sent Done false
5313 2418 Acked Write 2 Done Sent Done false
5314 2419 Acked Read 4 Done Sent Done false
5315 2420 Acked Write 1 Done Sent Done false
5316 2421 Acked Write 4 Done Sent Done false
5317 2422 Acked Write 9 Done Sent Done false
5318 2423 Acked Write 5 Done Sent Done false
5319 2424 Acked Write 4 Done Sent Done false
5320 2425 Acked Write 6 Done Sent Done false
5321 2426 Acked Read 6 Done Sent Done false
5322 2427 Acked Write 6 Done Sent Done false
5323 2428 Acked Read 6 Done Sent Done false
5324 2429 Acked Write 4 Done Sent Done false
5325 2430 Acked Write 10 Done Sent Done false
5326 2431 Acked Flush 0 Done Sent Done false
5327 2432 Acked Read 5 Done Sent Done false
5328 2433 Acked Write 5 Done Sent Done false
5329 2434 Acked Write 4 Done Sent Done false
5330 2435 Acked Write 3 Done Sent Done false
5331 2436 Acked Write 3 Done Sent Done false
5332 2437 Acked Read 6 Done Sent Done false
5333 2438 Acked Write 9 Done Sent Done false
5334 2439 Acked Write 4 Done Sent Done false
5335 2440 Acked Write 5 Done Sent Done false
5336 2441 Acked Read 7 Done Sent Done false
5337 2442 Acked Write 9 Done Sent Done false
5338 2443 Acked Read 3 Done Sent Done false
5339 2444 Acked Write 1 Done Sent Done false
5340 2445 Acked Read 6 Done Sent Done false
5341 2446 Acked Write 2 Done Sent Done false
5342 2447 Acked Write 9 Done Sent Done false
5343 2448 Acked Write 8 Done Sent Done false
5344 2449 Acked Write 3 Done Sent Done false
5345 2450 Acked Read 1 Done Sent Done false
5346 2451 Acked Read 2 Done Sent Done false
5347 2452 Acked Write 10 Done Sent Done false
5348 2453 Acked Read 7 Done Sent Done false
5349 2454 Acked Read 1 Done Sent Done false
5350 2455 Acked Read 9 Done Sent Done false
5351 2456 Acked Flush 0 Done Sent Done false
5352 2457 Acked Read 4 Done Sent Done false
5353 2458 Acked Read 4 Done Sent Done false
5354 2459 Acked Read 6 Done Sent Done false
5355 2460 Acked Read 10 Done Sent Done false
5356 2461 Acked Write 1 Done Sent Done false
5357 2462 Acked Flush 0 Done Sent Done false
5358 2463 Acked Write 4 Done Sent Done false
5359 2464 Acked Write 3 Done Sent Done false
5360 2465 Acked Read 6 Done Sent Done false
5361 2466 Acked Write 2 Done Sent Done false
5362 2467 Acked Read 8 Done Sent Done false
5363 2468 Acked Flush 0 Done Sent Done false
5364 2469 Acked Flush 0 Done Sent Done false
5365 2470 Acked Write 7 Done Sent Done false
5366 2471 Acked Read 9 Done Sent Done false
5367 2472 Acked Write 1 Done Sent Done false
5368 2473 Acked Write 10 Done Sent Done false
5369 2474 Acked Write 7 Done Sent Done false
5370 2475 Acked Read 10 Done Sent Done false
5371 2476 Acked Read 6 Done Sent Done false
5372 2477 Acked Read 2 Done Sent Done false
5373 2478 Acked Flush 0 Done Sent Done false
5374 2479 Acked Read 5 Done Sent Done false
5375 2480 Acked Write 8 Done Sent Done false
5376 2481 Acked Flush 0 Done Sent Done false
5377 2482 Acked Write 1 Done Sent Done false
5378 2483 Acked Write 9 Done Sent Done false
5379 2484 Acked Write 4 Done Sent Done false
5380 2485 Acked Read 7 Done Sent Done false
5381 2486 Acked Write 7 Done Sent Done false
5382 2487 Acked Read 6 Done Sent Done false
5383 2488 Acked Write 2 Done Sent Done false
5384 2489 Acked Read 1 Done Sent Done false
5385 2490 Acked Read 10 Done Sent Done false
5386 2491 Acked Write 5 Done Sent Done false
5387 2492 Acked Write 3 Done Sent Done false
5388 2493 Acked Write 3 Done Sent Done false
5389 2494 Acked Read 7 Done Sent Done false
5390 2495 Acked Read 10 Done Sent Done false
5391 2496 Acked Read 2 Done Sent Done false
5392 2497 Acked Read 5 Done Sent Done false
5393 2498 Acked Flush 0 Done Sent Done false
5394 2499 Acked Read 3 Done Sent Done false
5395 2500 Acked Read 8 Done Sent Done false
5396 2501 Acked Write 4 Done Sent Done false
5397 2502 Acked Write 6 Done Sent Done false
5398 2503 Acked Write 4 Done Sent Done false
5399 2504 Acked Read 3 Done Sent Done false
5400 2505 Acked Read 8 Done Sent Done false
5401 2506 Acked Write 10 Done Sent Done false
5402 2507 Acked Write 9 Done Sent Done false
5403 2508 Acked Read 2 Done Sent Done false
5404 STATES DS:0 DS:1 DS:2 TOTAL
5405 Sent 0 301 0 301
5406 Done 301 0 301 602
5407 Skipped 0 0 0 0
5408 Error 0 0 0 0
5409 Last Flush: Some(JobId(2498)) Some(JobId(2207)) Some(JobId(2498))
5410 Downstairs last five completed: 2207 2206 2205 2204 2203
5411 Upstairs last five completed: 2508 2507 2506 2505 2504
5412 CLIENT: Up:0 ds:301 act:2
54132025-03-17T22:35:38.594ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
54142025-03-17T22:35:39.851ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
54152025-03-17T22:35:40.408ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
54162025-03-17T22:35:40.908ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
54172025-03-17T22:35:41.410ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
5418 ----------------------------------------------------------------
5419 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5420 States: Active Offline Active
5421 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5422 2208 Acked Read 7 Done Sent Done false
5423 2209 Acked Write 10 Done Sent Done false
5424 2210 Acked Read 1 Done Sent Done false
5425 2211 Acked Write 3 Done Sent Done false
5426 2212 Acked Read 8 Done Sent Done false
5427 2213 Acked Read 1 Done Sent Done false
5428 2214 Acked Write 2 Done Sent Done false
5429 2215 Acked Write 7 Done Sent Done false
5430 2216 Acked Write 5 Done Sent Done false
5431 2217 Acked Write 1 Done Sent Done false
5432 2218 Acked Read 6 Done Sent Done false
5433 2219 Acked Write 3 Done Sent Done false
5434 2220 Acked Read 4 Done Sent Done false
5435 2221 Acked Write 4 Done Sent Done false
5436 2222 Acked Write 1 Done Sent Done false
5437 2223 Acked Read 6 Done Sent Done false
5438 2224 Acked Read 9 Done Sent Done false
5439 2225 Acked Write 10 Done Sent Done false
5440 2226 Acked Read 4 Done Sent Done false
5441 2227 Acked Flush 0 Done Sent Done false
5442 2228 Acked Write 3 Done Sent Done false
5443 2229 Acked Read 7 Done Sent Done false
5444 2230 Acked Read 5 Done Sent Done false
5445 2231 Acked Read 9 Done Sent Done false
5446 2232 Acked Write 10 Done Sent Done false
5447 2233 Acked Read 4 Done Sent Done false
5448 2234 Acked Read 5 Done Sent Done false
5449 2235 Acked Read 4 Done Sent Done false
5450 2236 Acked Read 7 Done Sent Done false
5451 2237 Acked Write 9 Done Sent Done false
5452 2238 Acked Read 7 Done Sent Done false
5453 2239 Acked Read 10 Done Sent Done false
5454 2240 Acked Read 3 Done Sent Done false
5455 2241 Acked Write 5 Done Sent Done false
5456 2242 Acked Read 5 Done Sent Done false
5457 2243 Acked Read 4 Done Sent Done false
5458 2244 Acked Read 7 Done Sent Done false
5459 2245 Acked Write 9 Done Sent Done false
5460 2246 Acked Write 5 Done Sent Done false
5461 2247 Acked Read 2 Done Sent Done false
5462 2248 Acked Read 8 Done Sent Done false
5463 2249 Acked Flush 0 Done Sent Done false
5464 2250 Acked Read 9 Done Sent Done false
5465 2251 Acked Flush 0 Done Sent Done false
5466 2252 Acked Write 10 Done Sent Done false
5467 2253 Acked Read 6 Done Sent Done false
5468 2254 Acked Write 10 Done Sent Done false
5469 2255 Acked Read 5 Done Sent Done false
5470 2256 Acked Read 5 Done Sent Done false
5471 2257 Acked Write 10 Done Sent Done false
5472 2258 Acked Write 5 Done Sent Done false
5473 2259 Acked Read 5 Done Sent Done false
5474 2260 Acked Write 1 Done Sent Done false
5475 2261 Acked Write 7 Done Sent Done false
5476 2262 Acked Read 2 Done Sent Done false
5477 2263 Acked Write 3 Done Sent Done false
5478 2264 Acked Flush 0 Done Sent Done false
5479 2265 Acked Read 5 Done Sent Done false
5480 2266 Acked Write 9 Done Sent Done false
5481 2267 Acked Read 4 Done Sent Done false
5482 2268 Acked Read 10 Done Sent Done false
5483 2269 Acked Write 1 Done Sent Done false
5484 2270 Acked Read 6 Done Sent Done false
5485 2271 Acked Read 5 Done Sent Done false
5486 2272 Acked Read 8 Done Sent Done false
5487 2273 Acked Write 10 Done Sent Done false
5488 2274 Acked Write 2 Done Sent Done false
5489 2275 Acked Write 7 Done Sent Done false
5490 2276 Acked Write 3 Done Sent Done false
5491 2277 Acked Write 10 Done Sent Done false
5492 2278 Acked Read 1 Done Sent Done false
5493 2279 Acked Write 6 Done Sent Done false
5494 2280 Acked Read 8 Done Sent Done false
5495 2281 Acked Write 6 Done Sent Done false
5496 2282 Acked Write 1 Done Sent Done false
5497 2283 Acked Read 2 Done Sent Done false
5498 2284 Acked Write 2 Done Sent Done false
5499 2285 Acked Read 3 Done Sent Done false
5500 2286 Acked Read 6 Done Sent Done false
5501 2287 Acked Write 7 Done Sent Done false
5502 2288 Acked Read 2 Done Sent Done false
5503 2289 Acked Read 5 Done Sent Done false
5504 2290 Acked Write 3 Done Sent Done false
5505 2291 Acked Write 5 Done Sent Done false
5506 2292 Acked Flush 0 Done Sent Done false
5507 2293 Acked Read 5 Done Sent Done false
5508 2294 Acked Read 1 Done Sent Done false
5509 2295 Acked Read 6 Done Sent Done false
5510 2296 Acked Write 10 Done Sent Done false
5511 2297 Acked Write 2 Done Sent Done false
5512 2298 Acked Read 7 Done Sent Done false
5513 2299 Acked Write 1 Done Sent Done false
5514 2300 Acked Read 5 Done Sent Done false
5515 2301 Acked Read 9 Done Sent Done false
5516 2302 Acked Read 9 Done Sent Done false
5517 2303 Acked Write 2 Done Sent Done false
5518 2304 Acked Flush 0 Done Sent Done false
5519 2305 Acked Write 10 Done Sent Done false
5520 2306 Acked Read 3 Done Sent Done false
5521 2307 Acked Flush 0 Done Sent Done false
5522 2308 Acked Write 8 Done Sent Done false
5523 2309 Acked Write 9 Done Sent Done false
5524 2310 Acked Read 1 Done Sent Done false
5525 2311 Acked Read 9 Done Sent Done false
5526 2312 Acked Read 10 Done Sent Done false
5527 2313 Acked Read 8 Done Sent Done false
5528 2314 Acked Flush 0 Done Sent Done false
5529 2315 Acked Write 6 Done Sent Done false
5530 2316 Acked Read 6 Done Sent Done false
5531 2317 Acked Read 5 Done Sent Done false
5532 2318 Acked Read 9 Done Sent Done false
5533 2319 Acked Write 3 Done Sent Done false
5534 2320 Acked Read 9 Done Sent Done false
5535 2321 Acked Read 7 Done Sent Done false
5536 2322 Acked Write 6 Done Sent Done false
5537 2323 Acked Write 5 Done Sent Done false
5538 2324 Acked Read 9 Done Sent Done false
5539 2325 Acked Read 5 Done Sent Done false
5540 2326 Acked Write 2 Done Sent Done false
5541 2327 Acked Read 10 Done Sent Done false
5542 2328 Acked Read 5 Done Sent Done false
5543 2329 Acked Read 5 Done Sent Done false
5544 2330 Acked Read 9 Done Sent Done false
5545 2331 Acked Write 3 Done Sent Done false
5546 2332 Acked Write 8 Done Sent Done false
5547 2333 Acked Read 3 Done Sent Done false
5548 2334 Acked Flush 0 Done Sent Done false
5549 2335 Acked Read 4 Done Sent Done false
5550 2336 Acked Write 3 Done Sent Done false
5551 2337 Acked Read 2 Done Sent Done false
5552 2338 Acked Write 4 Done Sent Done false
5553 2339 Acked Write 10 Done Sent Done false
5554 2340 Acked Write 9 Done Sent Done false
5555 2341 Acked Flush 0 Done Sent Done false
5556 2342 Acked Read 7 Done Sent Done false
5557 2343 Acked Write 7 Done Sent Done false
5558 2344 Acked Read 1 Done Sent Done false
5559 2345 Acked Write 8 Done Sent Done false
5560 2346 Acked Write 5 Done Sent Done false
5561 2347 Acked Read 10 Done Sent Done false
5562 2348 Acked Read 10 Done Sent Done false
5563 2349 Acked Read 6 Done Sent Done false
5564 2350 Acked Read 9 Done Sent Done false
5565 2351 Acked Write 9 Done Sent Done false
5566 2352 Acked Write 10 Done Sent Done false
5567 2353 Acked Write 8 Done Sent Done false
5568 2354 Acked Write 10 Done Sent Done false
5569 2355 Acked Write 10 Done Sent Done false
5570 2356 Acked Write 1 Done Sent Done false
5571 2357 Acked Write 1 Done Sent Done false
5572 2358 Acked Read 10 Done Sent Done false
5573 2359 Acked Read 9 Done Sent Done false
5574 2360 Acked Write 6 Done Sent Done false
5575 2361 Acked Write 3 Done Sent Done false
5576 2362 Acked Read 3 Done Sent Done false
5577 2363 Acked Read 6 Done Sent Done false
5578 2364 Acked Read 5 Done Sent Done false
5579 2365 Acked Write 7 Done Sent Done false
5580 2366 Acked Read 3 Done Sent Done false
5581 2367 Acked Write 4 Done Sent Done false
5582 2368 Acked Write 6 Done Sent Done false
5583 2369 Acked Read 7 Done Sent Done false
5584 2370 Acked Read 1 Done Sent Done false
5585 2371 Acked Write 4 Done Sent Done false
5586 2372 Acked Read 4 Done Sent Done false
5587 2373 Acked Write 8 Done Sent Done false
5588 2374 Acked Read 3 Done Sent Done false
5589 2375 Acked Read 9 Done Sent Done false
5590 2376 Acked Read 2 Done Sent Done false
5591 2377 Acked Write 10 Done Sent Done false
5592 2378 Acked Read 5 Done Sent Done false
5593 2379 Acked Read 6 Done Sent Done false
5594 2380 Acked Flush 0 Done Sent Done false
5595 2381 Acked Write 1 Done Sent Done false
5596 2382 Acked Write 3 Done Sent Done false
5597 2383 Acked Read 4 Done Sent Done false
5598 2384 Acked Write 8 Done Sent Done false
5599 2385 Acked Write 6 Done Sent Done false
5600 2386 Acked Read 10 Done Sent Done false
5601 2387 Acked Read 4 Done Sent Done false
5602 2388 Acked Read 10 Done Sent Done false
5603 2389 Acked Write 9 Done Sent Done false
5604 2390 Acked Write 1 Done Sent Done false
5605 2391 Acked Read 10 Done Sent Done false
5606 2392 Acked Read 3 Done Sent Done false
5607 2393 Acked Write 2 Done Sent Done false
5608 2394 Acked Read 1 Done Sent Done false
5609 2395 Acked Write 5 Done Sent Done false
5610 2396 Acked Write 6 Done Sent Done false
5611 2397 Acked Read 10 Done Sent Done false
5612 2398 Acked Write 5 Done Sent Done false
5613 2399 Acked Write 9 Done Sent Done false
5614 2400 Acked Flush 0 Done Sent Done false
5615 2401 Acked Read 7 Done Sent Done false
5616 2402 Acked Write 2 Done Sent Done false
5617 2403 Acked Write 9 Done Sent Done false
5618 2404 Acked Flush 0 Done Sent Done false
5619 2405 Acked Read 8 Done Sent Done false
5620 2406 Acked Write 9 Done Sent Done false
5621 2407 Acked Write 9 Done Sent Done false
5622 2408 Acked Read 10 Done Sent Done false
5623 2409 Acked Read 8 Done Sent Done false
5624 2410 Acked Write 1 Done Sent Done false
5625 2411 Acked Read 5 Done Sent Done false
5626 2412 Acked Write 5 Done Sent Done false
5627 2413 Acked Flush 0 Done Sent Done false
5628 2414 Acked Write 9 Done Sent Done false
5629 2415 Acked Write 6 Done Sent Done false
5630 2416 Acked Read 6 Done Sent Done false
5631 2417 Acked Write 9 Done Sent Done false
5632 2418 Acked Write 2 Done Sent Done false
5633 2419 Acked Read 4 Done Sent Done false
5634 2420 Acked Write 1 Done Sent Done false
5635 2421 Acked Write 4 Done Sent Done false
5636 2422 Acked Write 9 Done Sent Done false
5637 2423 Acked Write 5 Done Sent Done false
5638 2424 Acked Write 4 Done Sent Done false
5639 2425 Acked Write 6 Done Sent Done false
5640 2426 Acked Read 6 Done Sent Done false
5641 2427 Acked Write 6 Done Sent Done false
5642 2428 Acked Read 6 Done Sent Done false
5643 2429 Acked Write 4 Done Sent Done false
5644 2430 Acked Write 10 Done Sent Done false
5645 2431 Acked Flush 0 Done Sent Done false
5646 2432 Acked Read 5 Done Sent Done false
5647 2433 Acked Write 5 Done Sent Done false
5648 2434 Acked Write 4 Done Sent Done false
5649 2435 Acked Write 3 Done Sent Done false
5650 2436 Acked Write 3 Done Sent Done false
5651 2437 Acked Read 6 Done Sent Done false
5652 2438 Acked Write 9 Done Sent Done false
5653 2439 Acked Write 4 Done Sent Done false
5654 2440 Acked Write 5 Done Sent Done false
5655 2441 Acked Read 7 Done Sent Done false
5656 2442 Acked Write 9 Done Sent Done false
5657 2443 Acked Read 3 Done Sent Done false
5658 2444 Acked Write 1 Done Sent Done false
5659 2445 Acked Read 6 Done Sent Done false
5660 2446 Acked Write 2 Done Sent Done false
5661 2447 Acked Write 9 Done Sent Done false
5662 2448 Acked Write 8 Done Sent Done false
5663 2449 Acked Write 3 Done Sent Done false
5664 2450 Acked Read 1 Done Sent Done false
5665 2451 Acked Read 2 Done Sent Done false
5666 2452 Acked Write 10 Done Sent Done false
5667 2453 Acked Read 7 Done Sent Done false
5668 2454 Acked Read 1 Done Sent Done false
5669 2455 Acked Read 9 Done Sent Done false
5670 2456 Acked Flush 0 Done Sent Done false
5671 2457 Acked Read 4 Done Sent Done false
5672 2458 Acked Read 4 Done Sent Done false
5673 2459 Acked Read 6 Done Sent Done false
5674 2460 Acked Read 10 Done Sent Done false
5675 2461 Acked Write 1 Done Sent Done false
5676 2462 Acked Flush 0 Done Sent Done false
5677 2463 Acked Write 4 Done Sent Done false
5678 2464 Acked Write 3 Done Sent Done false
5679 2465 Acked Read 6 Done Sent Done false
5680 2466 Acked Write 2 Done Sent Done false
5681 2467 Acked Read 8 Done Sent Done false
5682 2468 Acked Flush 0 Done Sent Done false
5683 2469 Acked Flush 0 Done Sent Done false
5684 2470 Acked Write 7 Done Sent Done false
5685 2471 Acked Read 9 Done Sent Done false
5686 2472 Acked Write 1 Done Sent Done false
5687 2473 Acked Write 10 Done Sent Done false
5688 2474 Acked Write 7 Done Sent Done false
5689 2475 Acked Read 10 Done Sent Done false
5690 2476 Acked Read 6 Done Sent Done false
5691 2477 Acked Read 2 Done Sent Done false
5692 2478 Acked Flush 0 Done Sent Done false
5693 2479 Acked Read 5 Done Sent Done false
5694 2480 Acked Write 8 Done Sent Done false
5695 2481 Acked Flush 0 Done Sent Done false
5696 2482 Acked Write 1 Done Sent Done false
5697 2483 Acked Write 9 Done Sent Done false
5698 2484 Acked Write 4 Done Sent Done false
5699 2485 Acked Read 7 Done Sent Done false
5700 2486 Acked Write 7 Done Sent Done false
5701 2487 Acked Read 6 Done Sent Done false
5702 2488 Acked Write 2 Done Sent Done false
5703 2489 Acked Read 1 Done Sent Done false
5704 2490 Acked Read 10 Done Sent Done false
5705 2491 Acked Write 5 Done Sent Done false
5706 2492 Acked Write 3 Done Sent Done false
5707 2493 Acked Write 3 Done Sent Done false
5708 2494 Acked Read 7 Done Sent Done false
5709 2495 Acked Read 10 Done Sent Done false
5710 2496 Acked Read 2 Done Sent Done false
5711 2497 Acked Read 5 Done Sent Done false
5712 2498 Acked Flush 0 Done Sent Done false
5713 2499 Acked Read 3 Done Sent Done false
5714 2500 Acked Read 8 Done Sent Done false
5715 2501 Acked Write 4 Done Sent Done false
5716 2502 Acked Write 6 Done Sent Done false
5717 2503 Acked Write 4 Done Sent Done false
5718 2504 Acked Read 3 Done Sent Done false
5719 2505 Acked Read 8 Done Sent Done false
5720 2506 Acked Write 10 Done Sent Done false
5721 2507 Acked Write 9 Done Sent Done false
5722 2508 Acked Read 2 Done Sent Done false
5723 STATES DS:0 DS:1 DS:2 TOTAL
5724 Sent 0 301 0 301
5725 Done 301 0 301 602
5726 Skipped 0 0 0 0
5727 Error 0 0 0 0
5728 Last Flush: Some(JobId(2498)) Some(JobId(2207)) Some(JobId(2498))
5729 Downstairs last five completed: 2207 2206 2205 2204 2203
5730 Upstairs last five completed: 2508 2507 2506 2505 2504
5731 CLIENT: Up:0 ds:301 act:2
57322025-03-17T22:35:42.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57332025-03-17T22:35:43.412ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57342025-03-17T22:35:43.849ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57352025-03-17T22:35:43.849ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57362025-03-17T22:35:43.849ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57372025-03-17T22:35:43.849ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57382025-03-17T22:35:43.849ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57392025-03-17T22:35:44.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
57402025-03-17T22:35:45.415ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
5741 ----------------------------------------------------------------
5742 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5743 States: Active Offline Active
5744 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5745 2208 Acked Read 7 Done Sent Done false
5746 2209 Acked Write 10 Done Sent Done false
5747 2210 Acked Read 1 Done Sent Done false
5748 2211 Acked Write 3 Done Sent Done false
5749 2212 Acked Read 8 Done Sent Done false
5750 2213 Acked Read 1 Done Sent Done false
5751 2214 Acked Write 2 Done Sent Done false
5752 2215 Acked Write 7 Done Sent Done false
5753 2216 Acked Write 5 Done Sent Done false
5754 2217 Acked Write 1 Done Sent Done false
5755 2218 Acked Read 6 Done Sent Done false
5756 2219 Acked Write 3 Done Sent Done false
5757 2220 Acked Read 4 Done Sent Done false
5758 2221 Acked Write 4 Done Sent Done false
5759 2222 Acked Write 1 Done Sent Done false
5760 2223 Acked Read 6 Done Sent Done false
5761 2224 Acked Read 9 Done Sent Done false
5762 2225 Acked Write 10 Done Sent Done false
5763 2226 Acked Read 4 Done Sent Done false
5764 2227 Acked Flush 0 Done Sent Done false
5765 2228 Acked Write 3 Done Sent Done false
5766 2229 Acked Read 7 Done Sent Done false
5767 2230 Acked Read 5 Done Sent Done false
5768 2231 Acked Read 9 Done Sent Done false
5769 2232 Acked Write 10 Done Sent Done false
5770 2233 Acked Read 4 Done Sent Done false
5771 2234 Acked Read 5 Done Sent Done false
5772 2235 Acked Read 4 Done Sent Done false
5773 2236 Acked Read 7 Done Sent Done false
5774 2237 Acked Write 9 Done Sent Done false
5775 2238 Acked Read 7 Done Sent Done false
5776 2239 Acked Read 10 Done Sent Done false
5777 2240 Acked Read 3 Done Sent Done false
5778 2241 Acked Write 5 Done Sent Done false
5779 2242 Acked Read 5 Done Sent Done false
5780 2243 Acked Read 4 Done Sent Done false
5781 2244 Acked Read 7 Done Sent Done false
5782 2245 Acked Write 9 Done Sent Done false
5783 2246 Acked Write 5 Done Sent Done false
5784 2247 Acked Read 2 Done Sent Done false
5785 2248 Acked Read 8 Done Sent Done false
5786 2249 Acked Flush 0 Done Sent Done false
5787 2250 Acked Read 9 Done Sent Done false
5788 2251 Acked Flush 0 Done Sent Done false
5789 2252 Acked Write 10 Done Sent Done false
5790 2253 Acked Read 6 Done Sent Done false
5791 2254 Acked Write 10 Done Sent Done false
5792 2255 Acked Read 5 Done Sent Done false
5793 2256 Acked Read 5 Done Sent Done false
5794 2257 Acked Write 10 Done Sent Done false
5795 2258 Acked Write 5 Done Sent Done false
5796 2259 Acked Read 5 Done Sent Done false
5797 2260 Acked Write 1 Done Sent Done false
5798 2261 Acked Write 7 Done Sent Done false
5799 2262 Acked Read 2 Done Sent Done false
5800 2263 Acked Write 3 Done Sent Done false
5801 2264 Acked Flush 0 Done Sent Done false
5802 2265 Acked Read 5 Done Sent Done false
5803 2266 Acked Write 9 Done Sent Done false
5804 2267 Acked Read 4 Done Sent Done false
5805 2268 Acked Read 10 Done Sent Done false
5806 2269 Acked Write 1 Done Sent Done false
5807 2270 Acked Read 6 Done Sent Done false
5808 2271 Acked Read 5 Done Sent Done false
5809 2272 Acked Read 8 Done Sent Done false
5810 2273 Acked Write 10 Done Sent Done false
5811 2274 Acked Write 2 Done Sent Done false
5812 2275 Acked Write 7 Done Sent Done false
5813 2276 Acked Write 3 Done Sent Done false
5814 2277 Acked Write 10 Done Sent Done false
5815 2278 Acked Read 1 Done Sent Done false
5816 2279 Acked Write 6 Done Sent Done false
5817 2280 Acked Read 8 Done Sent Done false
5818 2281 Acked Write 6 Done Sent Done false
5819 2282 Acked Write 1 Done Sent Done false
5820 2283 Acked Read 2 Done Sent Done false
5821 2284 Acked Write 2 Done Sent Done false
5822 2285 Acked Read 3 Done Sent Done false
5823 2286 Acked Read 6 Done Sent Done false
5824 2287 Acked Write 7 Done Sent Done false
5825 2288 Acked Read 2 Done Sent Done false
5826 2289 Acked Read 5 Done Sent Done false
5827 2290 Acked Write 3 Done Sent Done false
5828 2291 Acked Write 5 Done Sent Done false
5829 2292 Acked Flush 0 Done Sent Done false
5830 2293 Acked Read 5 Done Sent Done false
5831 2294 Acked Read 1 Done Sent Done false
5832 2295 Acked Read 6 Done Sent Done false
5833 2296 Acked Write 10 Done Sent Done false
5834 2297 Acked Write 2 Done Sent Done false
5835 2298 Acked Read 7 Done Sent Done false
5836 2299 Acked Write 1 Done Sent Done false
5837 2300 Acked Read 5 Done Sent Done false
5838 2301 Acked Read 9 Done Sent Done false
5839 2302 Acked Read 9 Done Sent Done false
5840 2303 Acked Write 2 Done Sent Done false
5841 2304 Acked Flush 0 Done Sent Done false
5842 2305 Acked Write 10 Done Sent Done false
5843 2306 Acked Read 3 Done Sent Done false
5844 2307 Acked Flush 0 Done Sent Done false
5845 2308 Acked Write 8 Done Sent Done false
5846 2309 Acked Write 9 Done Sent Done false
5847 2310 Acked Read 1 Done Sent Done false
5848 2311 Acked Read 9 Done Sent Done false
5849 2312 Acked Read 10 Done Sent Done false
5850 2313 Acked Read 8 Done Sent Done false
5851 2314 Acked Flush 0 Done Sent Done false
5852 2315 Acked Write 6 Done Sent Done false
5853 2316 Acked Read 6 Done Sent Done false
5854 2317 Acked Read 5 Done Sent Done false
5855 2318 Acked Read 9 Done Sent Done false
5856 2319 Acked Write 3 Done Sent Done false
5857 2320 Acked Read 9 Done Sent Done false
5858 2321 Acked Read 7 Done Sent Done false
5859 2322 Acked Write 6 Done Sent Done false
5860 2323 Acked Write 5 Done Sent Done false
5861 2324 Acked Read 9 Done Sent Done false
5862 2325 Acked Read 5 Done Sent Done false
5863 2326 Acked Write 2 Done Sent Done false
5864 2327 Acked Read 10 Done Sent Done false
5865 2328 Acked Read 5 Done Sent Done false
5866 2329 Acked Read 5 Done Sent Done false
5867 2330 Acked Read 9 Done Sent Done false
5868 2331 Acked Write 3 Done Sent Done false
5869 2332 Acked Write 8 Done Sent Done false
5870 2333 Acked Read 3 Done Sent Done false
5871 2334 Acked Flush 0 Done Sent Done false
5872 2335 Acked Read 4 Done Sent Done false
5873 2336 Acked Write 3 Done Sent Done false
5874 2337 Acked Read 2 Done Sent Done false
5875 2338 Acked Write 4 Done Sent Done false
5876 2339 Acked Write 10 Done Sent Done false
5877 2340 Acked Write 9 Done Sent Done false
5878 2341 Acked Flush 0 Done Sent Done false
5879 2342 Acked Read 7 Done Sent Done false
5880 2343 Acked Write 7 Done Sent Done false
5881 2344 Acked Read 1 Done Sent Done false
5882 2345 Acked Write 8 Done Sent Done false
5883 2346 Acked Write 5 Done Sent Done false
5884 2347 Acked Read 10 Done Sent Done false
5885 2348 Acked Read 10 Done Sent Done false
5886 2349 Acked Read 6 Done Sent Done false
5887 2350 Acked Read 9 Done Sent Done false
5888 2351 Acked Write 9 Done Sent Done false
5889 2352 Acked Write 10 Done Sent Done false
5890 2353 Acked Write 8 Done Sent Done false
5891 2354 Acked Write 10 Done Sent Done false
5892 2355 Acked Write 10 Done Sent Done false
5893 2356 Acked Write 1 Done Sent Done false
5894 2357 Acked Write 1 Done Sent Done false
5895 2358 Acked Read 10 Done Sent Done false
5896 2359 Acked Read 9 Done Sent Done false
5897 2360 Acked Write 6 Done Sent Done false
5898 2361 Acked Write 3 Done Sent Done false
5899 2362 Acked Read 3 Done Sent Done false
5900 2363 Acked Read 6 Done Sent Done false
5901 2364 Acked Read 5 Done Sent Done false
5902 2365 Acked Write 7 Done Sent Done false
5903 2366 Acked Read 3 Done Sent Done false
5904 2367 Acked Write 4 Done Sent Done false
5905 2368 Acked Write 6 Done Sent Done false
5906 2369 Acked Read 7 Done Sent Done false
5907 2370 Acked Read 1 Done Sent Done false
5908 2371 Acked Write 4 Done Sent Done false
5909 2372 Acked Read 4 Done Sent Done false
5910 2373 Acked Write 8 Done Sent Done false
5911 2374 Acked Read 3 Done Sent Done false
5912 2375 Acked Read 9 Done Sent Done false
5913 2376 Acked Read 2 Done Sent Done false
5914 2377 Acked Write 10 Done Sent Done false
5915 2378 Acked Read 5 Done Sent Done false
5916 2379 Acked Read 6 Done Sent Done false
5917 2380 Acked Flush 0 Done Sent Done false
5918 2381 Acked Write 1 Done Sent Done false
5919 2382 Acked Write 3 Done Sent Done false
5920 2383 Acked Read 4 Done Sent Done false
5921 2384 Acked Write 8 Done Sent Done false
5922 2385 Acked Write 6 Done Sent Done false
5923 2386 Acked Read 10 Done Sent Done false
5924 2387 Acked Read 4 Done Sent Done false
5925 2388 Acked Read 10 Done Sent Done false
5926 2389 Acked Write 9 Done Sent Done false
5927 2390 Acked Write 1 Done Sent Done false
5928 2391 Acked Read 10 Done Sent Done false
5929 2392 Acked Read 3 Done Sent Done false
5930 2393 Acked Write 2 Done Sent Done false
5931 2394 Acked Read 1 Done Sent Done false
5932 2395 Acked Write 5 Done Sent Done false
5933 2396 Acked Write 6 Done Sent Done false
5934 2397 Acked Read 10 Done Sent Done false
5935 2398 Acked Write 5 Done Sent Done false
5936 2399 Acked Write 9 Done Sent Done false
5937 2400 Acked Flush 0 Done Sent Done false
5938 2401 Acked Read 7 Done Sent Done false
5939 2402 Acked Write 2 Done Sent Done false
5940 2403 Acked Write 9 Done Sent Done false
5941 2404 Acked Flush 0 Done Sent Done false
5942 2405 Acked Read 8 Done Sent Done false
5943 2406 Acked Write 9 Done Sent Done false
5944 2407 Acked Write 9 Done Sent Done false
5945 2408 Acked Read 10 Done Sent Done false
5946 2409 Acked Read 8 Done Sent Done false
5947 2410 Acked Write 1 Done Sent Done false
5948 2411 Acked Read 5 Done Sent Done false
5949 2412 Acked Write 5 Done Sent Done false
5950 2413 Acked Flush 0 Done Sent Done false
5951 2414 Acked Write 9 Done Sent Done false
5952 2415 Acked Write 6 Done Sent Done false
5953 2416 Acked Read 6 Done Sent Done false
5954 2417 Acked Write 9 Done Sent Done false
5955 2418 Acked Write 2 Done Sent Done false
5956 2419 Acked Read 4 Done Sent Done false
5957 2420 Acked Write 1 Done Sent Done false
5958 2421 Acked Write 4 Done Sent Done false
5959 2422 Acked Write 9 Done Sent Done false
5960 2423 Acked Write 5 Done Sent Done false
5961 2424 Acked Write 4 Done Sent Done false
5962 2425 Acked Write 6 Done Sent Done false
5963 2426 Acked Read 6 Done Sent Done false
5964 2427 Acked Write 6 Done Sent Done false
5965 2428 Acked Read 6 Done Sent Done false
5966 2429 Acked Write 4 Done Sent Done false
5967 2430 Acked Write 10 Done Sent Done false
5968 2431 Acked Flush 0 Done Sent Done false
5969 2432 Acked Read 5 Done Sent Done false
5970 2433 Acked Write 5 Done Sent Done false
5971 2434 Acked Write 4 Done Sent Done false
5972 2435 Acked Write 3 Done Sent Done false
5973 2436 Acked Write 3 Done Sent Done false
5974 2437 Acked Read 6 Done Sent Done false
5975 2438 Acked Write 9 Done Sent Done false
5976 2439 Acked Write 4 Done Sent Done false
5977 2440 Acked Write 5 Done Sent Done false
5978 2441 Acked Read 7 Done Sent Done false
5979 2442 Acked Write 9 Done Sent Done false
5980 2443 Acked Read 3 Done Sent Done false
5981 2444 Acked Write 1 Done Sent Done false
5982 2445 Acked Read 6 Done Sent Done false
5983 2446 Acked Write 2 Done Sent Done false
5984 2447 Acked Write 9 Done Sent Done false
5985 2448 Acked Write 8 Done Sent Done false
5986 2449 Acked Write 3 Done Sent Done false
5987 2450 Acked Read 1 Done Sent Done false
5988 2451 Acked Read 2 Done Sent Done false
5989 2452 Acked Write 10 Done Sent Done false
5990 2453 Acked Read 7 Done Sent Done false
5991 2454 Acked Read 1 Done Sent Done false
5992 2455 Acked Read 9 Done Sent Done false
5993 2456 Acked Flush 0 Done Sent Done false
5994 2457 Acked Read 4 Done Sent Done false
5995 2458 Acked Read 4 Done Sent Done false
5996 2459 Acked Read 6 Done Sent Done false
5997 2460 Acked Read 10 Done Sent Done false
5998 2461 Acked Write 1 Done Sent Done false
5999 2462 Acked Flush 0 Done Sent Done false
6000 2463 Acked Write 4 Done Sent Done false
6001 2464 Acked Write 3 Done Sent Done false
6002 2465 Acked Read 6 Done Sent Done false
6003 2466 Acked Write 2 Done Sent Done false
6004 2467 Acked Read 8 Done Sent Done false
6005 2468 Acked Flush 0 Done Sent Done false
6006 2469 Acked Flush 0 Done Sent Done false
6007 2470 Acked Write 7 Done Sent Done false
6008 2471 Acked Read 9 Done Sent Done false
6009 2472 Acked Write 1 Done Sent Done false
6010 2473 Acked Write 10 Done Sent Done false
6011 2474 Acked Write 7 Done Sent Done false
6012 2475 Acked Read 10 Done Sent Done false
6013 2476 Acked Read 6 Done Sent Done false
6014 2477 Acked Read 2 Done Sent Done false
6015 2478 Acked Flush 0 Done Sent Done false
6016 2479 Acked Read 5 Done Sent Done false
6017 2480 Acked Write 8 Done Sent Done false
6018 2481 Acked Flush 0 Done Sent Done false
6019 2482 Acked Write 1 Done Sent Done false
6020 2483 Acked Write 9 Done Sent Done false
6021 2484 Acked Write 4 Done Sent Done false
6022 2485 Acked Read 7 Done Sent Done false
6023 2486 Acked Write 7 Done Sent Done false
6024 2487 Acked Read 6 Done Sent Done false
6025 2488 Acked Write 2 Done Sent Done false
6026 2489 Acked Read 1 Done Sent Done false
6027 2490 Acked Read 10 Done Sent Done false
6028 2491 Acked Write 5 Done Sent Done false
6029 2492 Acked Write 3 Done Sent Done false
6030 2493 Acked Write 3 Done Sent Done false
6031 2494 Acked Read 7 Done Sent Done false
6032 2495 Acked Read 10 Done Sent Done false
6033 2496 Acked Read 2 Done Sent Done false
6034 2497 Acked Read 5 Done Sent Done false
6035 2498 Acked Flush 0 Done Sent Done false
6036 2499 Acked Read 3 Done Sent Done false
6037 2500 Acked Read 8 Done Sent Done false
6038 2501 Acked Write 4 Done Sent Done false
6039 2502 Acked Write 6 Done Sent Done false
6040 2503 Acked Write 4 Done Sent Done false
6041 2504 Acked Read 3 Done Sent Done false
6042 2505 Acked Read 8 Done Sent Done false
6043 2506 Acked Write 10 Done Sent Done false
6044 2507 Acked Write 9 Done Sent Done false
6045 2508 Acked Read 2 Done Sent Done false
6046 STATES DS:0 DS:1 DS:2 TOTAL
6047 Sent 0 301 0 301
6048 Done 301 0 301 602
6049 Skipped 0 0 0 0
6050 Error 0 0 0 0
6051 Last Flush: Some(JobId(2498)) Some(JobId(2207)) Some(JobId(2498))
6052 Downstairs last five completed: 2207 2206 2205 2204 2203
6053 Upstairs last five completed: 2508 2507 2506 2505 2504
6054 CLIENT: Up:0 ds:301 act:2
60552025-03-17T22:35:46.598ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
60562025-03-17T22:35:47.419ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
60572025-03-17T22:35:48.421ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
60582025-03-17T22:35:49.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
6059 [8820] got start action
6060 [8820] I am going to restart
6061 Starting downstairs at port 8820
6062 Make output file at "/tmp/dsc/downstairs-8820.txt"
6063 [8820][1] reports Starting
6064 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
6065 [8820][1] reports Running
6066 ----------------------------------------------------------------
6067 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6068 States: Active Offline Active
6069 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6070 2208 Acked Read 7 Done Sent Done false
6071 2209 Acked Write 10 Done Sent Done false
6072 2210 Acked Read 1 Done Sent Done false
6073 2211 Acked Write 3 Done Sent Done false
6074 2212 Acked Read 8 Done Sent Done false
6075 2213 Acked Read 1 Done Sent Done false
6076 2214 Acked Write 2 Done Sent Done false
6077 2215 Acked Write 7 Done Sent Done false
6078 2216 Acked Write 5 Done Sent Done false
6079 2217 Acked Write 1 Done Sent Done false
6080 2218 Acked Read 6 Done Sent Done false
6081 2219 Acked Write 3 Done Sent Done false
6082 2220 Acked Read 4 Done Sent Done false
6083 2221 Acked Write 4 Done Sent Done false
6084 2222 Acked Write 1 Done Sent Done false
6085 2223 Acked Read 6 Done Sent Done false
6086 2224 Acked Read 9 Done Sent Done false
6087 2225 Acked Write 10 Done Sent Done false
6088 2226 Acked Read 4 Done Sent Done false
6089 2227 Acked Flush 0 Done Sent Done false
6090 2228 Acked Write 3 Done Sent Done false
6091 2229 Acked Read 7 Done Sent Done false
6092 2230 Acked Read 5 Done Sent Done false
6093 2231 Acked Read 9 Done Sent Done false
6094 2232 Acked Write 10 Done Sent Done false
6095 2233 Acked Read 4 Done Sent Done false
6096 2234 Acked Read 5 Done Sent Done false
6097 2235 Acked Read 4 Done Sent Done false
6098 2236 Acked Read 7 Done Sent Done false
6099 2237 Acked Write 9 Done Sent Done false
6100 2238 Acked Read 7 Done Sent Done false
6101 2239 Acked Read 10 Done Sent Done false
6102 2240 Acked Read 3 Done Sent Done false
6103 2241 Acked Write 5 Done Sent Done false
6104 2242 Acked Read 5 Done Sent Done false
6105 2243 Acked Read 4 Done Sent Done false
6106 2244 Acked Read 7 Done Sent Done false
6107 2245 Acked Write 9 Done Sent Done false
6108 2246 Acked Write 5 Done Sent Done false
6109 2247 Acked Read 2 Done Sent Done false
6110 2248 Acked Read 8 Done Sent Done false
6111 2249 Acked Flush 0 Done Sent Done false
6112 2250 Acked Read 9 Done Sent Done false
6113 2251 Acked Flush 0 Done Sent Done false
6114 2252 Acked Write 10 Done Sent Done false
6115 2253 Acked Read 6 Done Sent Done false
6116 2254 Acked Write 10 Done Sent Done false
6117 2255 Acked Read 5 Done Sent Done false
6118 2256 Acked Read 5 Done Sent Done false
6119 2257 Acked Write 10 Done Sent Done false
6120 2258 Acked Write 5 Done Sent Done false
6121 2259 Acked Read 5 Done Sent Done false
6122 2260 Acked Write 1 Done Sent Done false
6123 2261 Acked Write 7 Done Sent Done false
6124 2262 Acked Read 2 Done Sent Done false
6125 2263 Acked Write 3 Done Sent Done false
6126 2264 Acked Flush 0 Done Sent Done false
6127 2265 Acked Read 5 Done Sent Done false
6128 2266 Acked Write 9 Done Sent Done false
6129 2267 Acked Read 4 Done Sent Done false
6130 2268 Acked Read 10 Done Sent Done false
6131 2269 Acked Write 1 Done Sent Done false
6132 2270 Acked Read 6 Done Sent Done false
6133 2271 Acked Read 5 Done Sent Done false
6134 2272 Acked Read 8 Done Sent Done false
6135 2273 Acked Write 10 Done Sent Done false
6136 2274 Acked Write 2 Done Sent Done false
6137 2275 Acked Write 7 Done Sent Done false
6138 2276 Acked Write 3 Done Sent Done false
6139 2277 Acked Write 10 Done Sent Done false
6140 2278 Acked Read 1 Done Sent Done false
6141 2279 Acked Write 6 Done Sent Done false
6142 2280 Acked Read 8 Done Sent Done false
6143 2281 Acked Write 6 Done Sent Done false
6144 2282 Acked Write 1 Done Sent Done false
6145 2283 Acked Read 2 Done Sent Done false
6146 2284 Acked Write 2 Done Sent Done false
6147 2285 Acked Read 3 Done Sent Done false
6148 2286 Acked Read 6 Done Sent Done false
6149 2287 Acked Write 7 Done Sent Done false
6150 2288 Acked Read 2 Done Sent Done false
6151 2289 Acked Read 5 Done Sent Done false
6152 2290 Acked Write 3 Done Sent Done false
6153 2291 Acked Write 5 Done Sent Done false
6154 2292 Acked Flush 0 Done Sent Done false
6155 2293 Acked Read 5 Done Sent Done false
6156 2294 Acked Read 1 Done Sent Done false
6157 2295 Acked Read 6 Done Sent Done false
6158 2296 Acked Write 10 Done Sent Done false
6159 2297 Acked Write 2 Done Sent Done false
6160 2298 Acked Read 7 Done Sent Done false
6161 2299 Acked Write 1 Done Sent Done false
6162 2300 Acked Read 5 Done Sent Done false
6163 2301 Acked Read 9 Done Sent Done false
6164 2302 Acked Read 9 Done Sent Done false
6165 2303 Acked Write 2 Done Sent Done false
6166 2304 Acked Flush 0 Done Sent Done false
6167 2305 Acked Write 10 Done Sent Done false
6168 2306 Acked Read 3 Done Sent Done false
6169 2307 Acked Flush 0 Done Sent Done false
6170 2308 Acked Write 8 Done Sent Done false
6171 2309 Acked Write 9 Done Sent Done false
6172 2310 Acked Read 1 Done Sent Done false
6173 2311 Acked Read 9 Done Sent Done false
6174 2312 Acked Read 10 Done Sent Done false
6175 2313 Acked Read 8 Done Sent Done false
6176 2314 Acked Flush 0 Done Sent Done false
6177 2315 Acked Write 6 Done Sent Done false
6178 2316 Acked Read 6 Done Sent Done false
6179 2317 Acked Read 5 Done Sent Done false
6180 2318 Acked Read 9 Done Sent Done false
6181 2319 Acked Write 3 Done Sent Done false
6182 2320 Acked Read 9 Done Sent Done false
6183 2321 Acked Read 7 Done Sent Done false
6184 2322 Acked Write 6 Done Sent Done false
6185 2323 Acked Write 5 Done Sent Done false
6186 2324 Acked Read 9 Done Sent Done false
6187 2325 Acked Read 5 Done Sent Done false
6188 2326 Acked Write 2 Done Sent Done false
6189 2327 Acked Read 10 Done Sent Done false
6190 2328 Acked Read 5 Done Sent Done false
6191 2329 Acked Read 5 Done Sent Done false
6192 2330 Acked Read 9 Done Sent Done false
6193 2331 Acked Write 3 Done Sent Done false
6194 2332 Acked Write 8 Done Sent Done false
6195 2333 Acked Read 3 Done Sent Done false
6196 2334 Acked Flush 0 Done Sent Done false
6197 2335 Acked Read 4 Done Sent Done false
6198 2336 Acked Write 3 Done Sent Done false
6199 2337 Acked Read 2 Done Sent Done false
6200 2338 Acked Write 4 Done Sent Done false
6201 2339 Acked Write 10 Done Sent Done false
6202 2340 Acked Write 9 Done Sent Done false
6203 2341 Acked Flush 0 Done Sent Done false
6204 2342 Acked Read 7 Done Sent Done false
6205 2343 Acked Write 7 Done Sent Done false
6206 2344 Acked Read 1 Done Sent Done false
6207 2345 Acked Write 8 Done Sent Done false
6208 2346 Acked Write 5 Done Sent Done false
6209 2347 Acked Read 10 Done Sent Done false
6210 2348 Acked Read 10 Done Sent Done false
6211 2349 Acked Read 6 Done Sent Done false
6212 2350 Acked Read 9 Done Sent Done false
6213 2351 Acked Write 9 Done Sent Done false
6214 2352 Acked Write 10 Done Sent Done false
6215 2353 Acked Write 8 Done Sent Done false
6216 2354 Acked Write 10 Done Sent Done false
6217 2355 Acked Write 10 Done Sent Done false
6218 2356 Acked Write 1 Done Sent Done false
6219 2357 Acked Write 1 Done Sent Done false
6220 2358 Acked Read 10 Done Sent Done false
6221 2359 Acked Read 9 Done Sent Done false
6222 2360 Acked Write 6 Done Sent Done false
6223 2361 Acked Write 3 Done Sent Done false
6224 2362 Acked Read 3 Done Sent Done false
6225 2363 Acked Read 6 Done Sent Done false
6226 2364 Acked Read 5 Done Sent Done false
6227 2365 Acked Write 7 Done Sent Done false
6228 2366 Acked Read 3 Done Sent Done false
6229 2367 Acked Write 4 Done Sent Done false
6230 2368 Acked Write 6 Done Sent Done false
6231 2369 Acked Read 7 Done Sent Done false
6232 2370 Acked Read 1 Done Sent Done false
6233 2371 Acked Write 4 Done Sent Done false
6234 2372 Acked Read 4 Done Sent Done false
6235 2373 Acked Write 8 Done Sent Done false
6236 2374 Acked Read 3 Done Sent Done false
6237 2375 Acked Read 9 Done Sent Done false
6238 2376 Acked Read 2 Done Sent Done false
6239 2377 Acked Write 10 Done Sent Done false
6240 2378 Acked Read 5 Done Sent Done false
6241 2379 Acked Read 6 Done Sent Done false
6242 2380 Acked Flush 0 Done Sent Done false
6243 2381 Acked Write 1 Done Sent Done false
6244 2382 Acked Write 3 Done Sent Done false
6245 2383 Acked Read 4 Done Sent Done false
6246 2384 Acked Write 8 Done Sent Done false
6247 2385 Acked Write 6 Done Sent Done false
6248 2386 Acked Read 10 Done Sent Done false
6249 2387 Acked Read 4 Done Sent Done false
6250 2388 Acked Read 10 Done Sent Done false
6251 2389 Acked Write 9 Done Sent Done false
6252 2390 Acked Write 1 Done Sent Done false
6253 2391 Acked Read 10 Done Sent Done false
6254 2392 Acked Read 3 Done Sent Done false
6255 2393 Acked Write 2 Done Sent Done false
6256 2394 Acked Read 1 Done Sent Done false
6257 2395 Acked Write 5 Done Sent Done false
6258 2396 Acked Write 6 Done Sent Done false
6259 2397 Acked Read 10 Done Sent Done false
6260 2398 Acked Write 5 Done Sent Done false
6261 2399 Acked Write 9 Done Sent Done false
6262 2400 Acked Flush 0 Done Sent Done false
6263 2401 Acked Read 7 Done Sent Done false
6264 2402 Acked Write 2 Done Sent Done false
6265 2403 Acked Write 9 Done Sent Done false
6266 2404 Acked Flush 0 Done Sent Done false
6267 2405 Acked Read 8 Done Sent Done false
6268 2406 Acked Write 9 Done Sent Done false
6269 2407 Acked Write 9 Done Sent Done false
6270 2408 Acked Read 10 Done Sent Done false
6271 2409 Acked Read 8 Done Sent Done false
6272 2410 Acked Write 1 Done Sent Done false
6273 2411 Acked Read 5 Done Sent Done false
6274 2412 Acked Write 5 Done Sent Done false
6275 2413 Acked Flush 0 Done Sent Done false
6276 2414 Acked Write 9 Done Sent Done false
6277 2415 Acked Write 6 Done Sent Done false
6278 2416 Acked Read 6 Done Sent Done false
6279 2417 Acked Write 9 Done Sent Done false
6280 2418 Acked Write 2 Done Sent Done false
6281 2419 Acked Read 4 Done Sent Done false
6282 2420 Acked Write 1 Done Sent Done false
6283 2421 Acked Write 4 Done Sent Done false
6284 2422 Acked Write 9 Done Sent Done false
6285 2423 Acked Write 5 Done Sent Done false
6286 2424 Acked Write 4 Done Sent Done false
6287 2425 Acked Write 6 Done Sent Done false
6288 2426 Acked Read 6 Done Sent Done false
6289 2427 Acked Write 6 Done Sent Done false
6290 2428 Acked Read 6 Done Sent Done false
6291 2429 Acked Write 4 Done Sent Done false
6292 2430 Acked Write 10 Done Sent Done false
6293 2431 Acked Flush 0 Done Sent Done false
6294 2432 Acked Read 5 Done Sent Done false
6295 2433 Acked Write 5 Done Sent Done false
6296 2434 Acked Write 4 Done Sent Done false
6297 2435 Acked Write 3 Done Sent Done false
6298 2436 Acked Write 3 Done Sent Done false
6299 2437 Acked Read 6 Done Sent Done false
6300 2438 Acked Write 9 Done Sent Done false
6301 2439 Acked Write 4 Done Sent Done false
6302 2440 Acked Write 5 Done Sent Done false
6303 2441 Acked Read 7 Done Sent Done false
6304 2442 Acked Write 9 Done Sent Done false
6305 2443 Acked Read 3 Done Sent Done false
6306 2444 Acked Write 1 Done Sent Done false
6307 2445 Acked Read 6 Done Sent Done false
6308 2446 Acked Write 2 Done Sent Done false
6309 2447 Acked Write 9 Done Sent Done false
6310 2448 Acked Write 8 Done Sent Done false
6311 2449 Acked Write 3 Done Sent Done false
6312 2450 Acked Read 1 Done Sent Done false
6313 2451 Acked Read 2 Done Sent Done false
6314 2452 Acked Write 10 Done Sent Done false
6315 2453 Acked Read 7 Done Sent Done false
6316 2454 Acked Read 1 Done Sent Done false
6317 2455 Acked Read 9 Done Sent Done false
6318 2456 Acked Flush 0 Done Sent Done false
6319 2457 Acked Read 4 Done Sent Done false
6320 2458 Acked Read 4 Done Sent Done false
6321 2459 Acked Read 6 Done Sent Done false
6322 2460 Acked Read 10 Done Sent Done false
6323 2461 Acked Write 1 Done Sent Done false
6324 2462 Acked Flush 0 Done Sent Done false
6325 2463 Acked Write 4 Done Sent Done false
6326 2464 Acked Write 3 Done Sent Done false
6327 2465 Acked Read 6 Done Sent Done false
6328 2466 Acked Write 2 Done Sent Done false
6329 2467 Acked Read 8 Done Sent Done false
6330 2468 Acked Flush 0 Done Sent Done false
6331 2469 Acked Flush 0 Done Sent Done false
6332 2470 Acked Write 7 Done Sent Done false
6333 2471 Acked Read 9 Done Sent Done false
6334 2472 Acked Write 1 Done Sent Done false
6335 2473 Acked Write 10 Done Sent Done false
6336 2474 Acked Write 7 Done Sent Done false
6337 2475 Acked Read 10 Done Sent Done false
6338 2476 Acked Read 6 Done Sent Done false
6339 2477 Acked Read 2 Done Sent Done false
6340 2478 Acked Flush 0 Done Sent Done false
6341 2479 Acked Read 5 Done Sent Done false
6342 2480 Acked Write 8 Done Sent Done false
6343 2481 Acked Flush 0 Done Sent Done false
6344 2482 Acked Write 1 Done Sent Done false
6345 2483 Acked Write 9 Done Sent Done false
6346 2484 Acked Write 4 Done Sent Done false
6347 2485 Acked Read 7 Done Sent Done false
6348 2486 Acked Write 7 Done Sent Done false
6349 2487 Acked Read 6 Done Sent Done false
6350 2488 Acked Write 2 Done Sent Done false
6351 2489 Acked Read 1 Done Sent Done false
6352 2490 Acked Read 10 Done Sent Done false
6353 2491 Acked Write 5 Done Sent Done false
6354 2492 Acked Write 3 Done Sent Done false
6355 2493 Acked Write 3 Done Sent Done false
6356 2494 Acked Read 7 Done Sent Done false
6357 2495 Acked Read 10 Done Sent Done false
6358 2496 Acked Read 2 Done Sent Done false
6359 2497 Acked Read 5 Done Sent Done false
6360 2498 Acked Flush 0 Done Sent Done false
6361 2499 Acked Read 3 Done Sent Done false
6362 2500 Acked Read 8 Done Sent Done false
6363 2501 Acked Write 4 Done Sent Done false
6364 2502 Acked Write 6 Done Sent Done false
6365 2503 Acked Write 4 Done Sent Done false
6366 2504 Acked Read 3 Done Sent Done false
6367 2505 Acked Read 8 Done Sent Done false
6368 2506 Acked Write 10 Done Sent Done false
6369 2507 Acked Write 9 Done Sent Done false
6370 2508 Acked Read 2 Done Sent Done false
6371 STATES DS:0 DS:1 DS:2 TOTAL
6372 Sent 0 301 0 301
6373 Done 301 0 301 602
6374 Skipped 0 0 0 0
6375 Error 0 0 0 0
6376 Last Flush: Some(JobId(2498)) Some(JobId(2207)) Some(JobId(2498))
6377 Downstairs last five completed: 2207 2206 2205 2204 2203
6378 Upstairs last five completed: 2508 2507 2506 2505 2504
6379 CLIENT: Up:0 ds:301 act:2
63802025-03-17T22:35:50.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63812025-03-17T22:35:51.426ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63822025-03-17T22:35:52.428ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63832025-03-17T22:35:53.429ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63842025-03-17T22:35:53.850ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63852025-03-17T22:35:53.850ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42666) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63862025-03-17T22:35:53.851ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63872025-03-17T22:35:53.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63882025-03-17T22:35:53.852ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63892025-03-17T22:35:53.852ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63902025-03-17T22:35:53.852ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63912025-03-17T22:35:53.852ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
63922025-03-17T22:35:53.852ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
6393 ----------------------------------------------------------------
6394 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
6395 States: Active Active Active
6396 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6397 2499 Acked Read 3 Done Done Done true
6398 2500 Acked Read 8 Done Done Done true
6399 2501 Acked Write 4 Done Done Done true
6400 2502 Acked Write 6 Done Done Done true
6401 2503 Acked Write 4 Done Done Done true
6402 2504 Acked Read 3 Done Done Done true
6403 2505 Acked Read 8 Done Done Done true
6404 2506 Acked Write 10 Done Done Done true
6405 2507 Acked Write 9 Done Done Done true
6406 2508 Acked Read 2 Done Done Done true
6407 STATES DS:0 DS:1 DS:2 TOTAL
6408 Sent 0 0 0 0
6409 Done 10 10 10 30
6410 Skipped 0 0 0 0
6411 Error 0 0 0 0
6412 Last Flush: Some(JobId(2498)) Some(JobId(2498)) Some(JobId(2498))
6413 Downstairs last five completed: 2498 2497 2496 2495 2494
6414 Upstairs last five completed: 2508 2507 2506 2505 2504
6415 CLIENT: Up:0 ds:10 act:3
6416 ----------------------------------------------------------------
6417 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6418 Downstairs last five completed: 2509 2508 2507 2506 2505
6419 Upstairs last five completed: 2509 2508 2507 2506 2505
6420 CLIENT: Up:0 ds:0 act:3
6421 Replay: All jobs finished
6422 Main task has work to do, go find it
6423 got dsc Stop(1)
6424 stop 1
6425 Mar 17 22:35:58.103 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
6426 630, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3a575b52-350a-4f5f-ad02-d802c27e00ad, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
64272025-03-17T22:35:58.104ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
64282025-03-17T22:35:58.104ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
64292025-03-17T22:35:58.104ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
64302025-03-17T22:35:58.104ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
6431 Mar 17 22:35:58.104 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2bf38457-f2f1-44e0-b3fe-2d9ee92f2409, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
6432 [8820] Exited with: ExitStatus(unix_wait_status(9))
6433 [8820][1] reports Exit
6434 Mar 17 22:36:02.107 INFO request completed, latency_us: 284, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3cea4b26-8d41-45f1-b79c-36c26ab461b7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
6435 001/300 Write block 2140 len 36864 data: 2 2 2 2 3 3 3 2 2
6436 002/300 Read block 2897 len 32768
6437 003/300 Read block 2028 len 40960
6438 004/300 Read block 3093 len 20480
6439 005/300 Read block 2202 len 40960
6440 006/300 Write block 1598 len 16384 data: 3 3 3 3
6441 007/300 Flush
6442 008/300 Flush
6443 009/300 Read block 1251 len 28672
6444 010/300 Read block 3734 len 24576
6445 011/300 Write block 4407 len 16384 data: 3 3 3 3
6446 012/300 Read block 3585 len 24576
6447 013/300 Write block 4877 len 4096 data: 2
6448 014/300 Write block 3873 len 12288 data: 2 3 3
6449 015/300 Read block 1342 len 24576
6450 016/300 Read block 2714 len 16384
6451 017/300 Read block 2187 len 40960
6452 018/300 Read block 711 len 20480
6453 019/300 Read block 1228 len 4096
6454 020/300 Read block 2043 len 32768
6455 021/300 Read block 2485 len 20480
6456 022/300 Read block 2986 len 28672
6457 023/300 Write block 2294 len 12288 data: 3 3 2
6458 024/300 Read block 3769 len 4096
6459 025/300 Write block 383 len 12288 data: 2 2 2
6460 026/300 Write block 533 len 20480 data: 2 3 3 2 2
6461 027/300 Read block 4873 len 40960
6462 028/300 Write block 135 len 8192 data: 4 4
6463 029/300 Read block 1101 len 40960
6464 030/300 Write block 654 len 16384 data: 3 3 3 4
6465 031/300 Read block 2252 len 24576
6466 032/300 Read block 2275 len 36864
6467 033/300 Write block 675 len 20480 data: 4 4 5 4 3
6468 034/300 Read block 1796 len 8192
6469 035/300 Read block 4442 len 16384
6470 036/300 Read block 2253 len 16384
6471 037/300 Write block 3642 len 12288 data: 3 2 2
6472 038/300 Write block 3245 len 32768 data: 2 2 2 2 2 2 2 2
6473 039/300 Read block 2980 len 36864
6474 040/300 Write block 2635 len 20480 data: 2 2 3 3 3
6475 041/300 Write block 2027 len 36864 data: 5 4 4 3 3 3 3 3 4
6476 042/300 Write block 4910 len 20480 data: 4 4 3 3 3
6477 043/300 Flush
6478 044/300 Read block 3516 len 36864
6479 045/300 Read block 2224 len 24576
6480 046/300 Flush
6481 047/300 Read block 1149 len 20480
6482 048/300 Write block 3659 len 4096 data: 2
6483 049/300 Write block 117 len 40960 data: 2 2 2 3 3 3 3 3 3 3
6484 050/300 Flush
6485 051/300 Read block 1873 len 8192
6486 052/300 Flush
6487 053/300 Flush
6488 054/300 Read block 1481 len 8192
6489 055/300 Read block 4174 len 32768
6490 056/300 Read block 4414 len 24576
6491 057/300 Read block 284 len 36864
6492 058/300 Write block 1074 len 12288 data: 4 3 3
6493 059/300 Read block 140 len 40960
6494 060/300 Read block 4962 len 28672
6495 061/300 Read block 2860 len 24576
6496 062/300 Write block 3417 len 16384 data: 3 4 3 4
6497 063/300 Read block 258 len 28672
6498 064/300 Read block 146 len 20480
6499 065/300 Write block 4365 len 32768 data: 2 2 3 3 2 2 2 2
6500 066/300 Read block 595 len 32768
6501 067/300 Write block 4428 len 40960 data: 5 5 5 3 3 3 3 3 3 3
6502 068/300 Read block 1162 len 24576
6503 069/300 Flush
6504 070/300 Read block 2294 len 4096
6505 071/300 Write block 2201 len 24576 data: 2 2 2 2 3 2
6506 072/300 Flush
6507 073/300 Write block 2507 len 24576 data: 2 2 2 2 2 2
6508 074/300 Write block 3112 len 36864 data: 2 2 2 2 2 2 2 2 2
6509 075/300 Read block 1572 len 16384
6510 076/300 Flush
6511 077/300 Read block 4424 len 16384
6512 078/300 Read block 4732 len 24576
6513 079/300 Read block 937 len 40960
6514 080/300 Read block 3211 len 12288
6515 081/300 Flush
6516 082/300 Read block 764 len 12288
6517 083/300 Read block 4506 len 24576
6518 084/300 Read block 4301 len 28672
6519 085/300 Write block 1534 len 12288 data: 4 3 3
6520 086/300 Read block 1555 len 12288
6521 087/300 Read block 4774 len 4096
6522 088/300 Write block 2377 len 24576 data: 2 2 3 3 2 2
6523 089/300 Read block 3516 len 8192
6524 090/300 Read block 550 len 40960
6525 091/300 Write block 572 len 16384 data: 2 2 2 2
6526 092/300 Write block 549 len 32768 data: 2 2 2 2 2 2 2 2
6527 093/300 Write block 4915 len 40960 data: 3 3 3 3 3 2 3 3 4 4
6528 094/300 Write block 1807 len 32768 data: 2 2 2 2 2 2 2 2
6529 095/300 Read block 1544 len 24576
6530 096/300 Read block 3225 len 16384
6531 097/300 Read block 1849 len 28672
6532 098/300 Read block 4321 len 8192
6533 099/300 Read block 1031 len 4096
6534 100/300 Write block 3209 len 20480 data: 3 4 4 4 4
6535 101/300 Write block 80 len 32768 data: 2 3 3 4 2 2 2 2
6536 102/300 Read block 1559 len 32768
6537 103/300 Flush
6538 104/300 Read block 3198 len 24576
6539 105/300 Flush
6540 106/300 Read block 3528 len 28672
6541 107/300 Read block 1561 len 36864
6542 108/300 Write block 1319 len 40960 data: 2 2 2 2 2 2 2 3 3 3
6543 109/300 Read block 1815 len 12288
6544 110/300 Read block 4488 len 28672
6545 111/300 Write block 3193 len 32768 data: 2 3 3 3 3 2 2 2
6546 112/300 Read block 4756 len 16384
6547 113/300 Read block 1969 len 4096
6548 114/300 Read block 525 len 24576
6549 115/300 Read block 1145 len 16384
6550 116/300 Read block 2580 len 12288
6551 117/300 Flush
6552 118/300 Write block 3100 len 24576 data: 2 2 2 2 2 2
6553 119/300 Write block 289 len 28672 data: 2 2 2 3 4 4 4
6554 120/300 Read block 3182 len 20480
6555 121/300 Write block 4527 len 12288 data: 4 4 4
6556 122/300 Flush
6557 123/300 Flush
6558 124/300 Write block 3655 len 12288 data: 2 2 2
6559 125/300 Write block 4351 len 32768 data: 4 4 4 4 3 3 3 3
6560 126/300 Read block 196 len 32768
6561 127/300 Read block 3753 len 8192
6562 128/300 Write block 3128 len 32768 data: 3 3 3 3 2 3 3 2
6563 129/300 Flush
6564 130/300 Write block 3409 len 4096 data: 4
6565 131/300 Write block 4908 len 40960 data: 3 3 5 5 4 4 4 4 4 4
6566 132/300 Write block 3471 len 8192 data: 2 2
6567 133/300 Flush
6568 134/300 Read block 1560 len 4096
6569 135/300 Read block 3515 len 16384
6570 136/300 Write block 4809 len 20480 data: 3 3 3 2 2
6571 137/300 Read block 3749 len 8192
6572 138/300 Write block 3568 len 12288 data: 2 2 4
6573 139/300 Write block 2632 len 4096 data: 2
6574 140/300 Write block 3381 len 4096 data: 2
6575 141/300 Write block 4413 len 12288 data: 2 2 2
6576 142/300 Read block 2199 len 32768
6577 143/300 Read block 4424 len 32768
6578 144/300 Read block 3483 len 28672
6579 145/300 Write block 2633 len 4096 data: 2
6580 146/300 Read block 4976 len 40960
6581 147/300 Write block 2470 len 24576 data: 3 2 2 2 3 3
6582 148/300 Read block 2668 len 20480
6583 149/300 Write block 4323 len 40960 data: 2 2 2 2 3 3 3 3 2 2
6584 150/300 Write block 3877 len 12288 data: 3 3 3
6585 151/300 Write block 264 len 4096 data: 3
6586 152/300 Read block 1727 len 40960
6587 153/300 Read block 1356 len 28672
6588 154/300 Read block 593 len 12288
6589 155/300 Flush
6590 156/300 Write block 1562 len 16384 data: 3 2 2 2
6591 157/300 Read block 2055 len 32768
6592 158/300 Flush
6593 159/300 Read block 4983 len 4096
6594 160/300 Read block 4970 len 28672
6595 161/300 Write block 1262 len 24576 data: 2 2 2 2 2 2
6596 162/300 Write block 938 len 32768 data: 3 3 3 4 4 4 4 4
6597 163/300 Read block 4181 len 36864
6598 164/300 Read block 3665 len 28672
6599 165/300 Write block 2945 len 40960 data: 2 2 2 2 2 2 3 3 3 3
6600 166/300 Flush
6601 167/300 Read block 2965 len 40960
6602 168/300 Write block 2831 len 20480 data: 2 2 2 2 2
6603 169/300 Write block 645 len 8192 data: 3 3
6604 170/300 Write block 1784 len 32768 data: 4 4 4 4 3 3 2 2
6605 171/300 Write block 1301 len 8192 data: 2 2
6606 172/300 Write block 3653 len 40960 data: 2 2 3 3 3 2 3 2 2 3
6607 173/300 Write block 3474 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6608 174/300 Read block 497 len 28672
6609 175/300 Write block 3016 len 16384 data: 3 3 2 2
6610 176/300 Write block 296 len 36864 data: 4 5 3 3 3 3 2 2 2
6611 177/300 Read block 1293 len 4096
6612 178/300 Read block 917 len 24576
6613 179/300 Read block 2269 len 16384
6614 180/300 Flush
6615 181/300 Write block 748 len 12288 data: 5 3 3
6616 182/300 Read block 3485 len 16384
6617 183/300 Write block 3357 len 28672 data: 3 3 3 2 2 3 3
6618 184/300 Write block 2986 len 36864 data: 3 2 2 2 3 3 4 3 3
6619 185/300 Read block 3196 len 12288
6620 186/300 Read block 2177 len 32768
6621 187/300 Flush
6622 188/300 Write block 991 len 12288 data: 2 2 2
6623 189/300 Read block 1809 len 4096
6624 190/300 Write block 4167 len 32768 data: 4 4 4 3 3 2 2 2
6625 191/300 Flush
6626 192/300 Write block 3323 len 36864 data: 3 3 3 2 2 2 2 2 2
6627 193/300 Write block 3932 len 16384 data: 3 3 3 3
6628 194/300 Read block 2734 len 20480
6629 195/300 Flush
6630 196/300 Read block 4942 len 16384
6631 197/300 Read block 3172 len 8192
6632 198/300 Read block 2275 len 4096
6633 199/300 Write block 2947 len 4096 data: 3
6634 200/300 Write block 1998 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6635 201/300 Write block 2193 len 4096 data: 2
6636 202/300 Read block 1183 len 8192
6637 203/300 Write block 1507 len 40960 data: 3 3 3 2 2 2 2 2 2 2
6638 204/300 Write block 2902 len 16384 data: 3 2 2 2
6639 205/300 Read block 942 len 24576
6640 206/300 Flush
6641 207/300 Read block 408 len 32768
6642 208/300 Read block 4610 len 28672
6643 209/300 Read block 2095 len 12288
6644 210/300 Write block 1156 len 12288 data: 2 2 2
6645 211/300 Flush
6646 212/300 Write block 3945 len 40960 data: 3 3 2 2 2 2 2 2 2 2
6647 213/300 Write block 3551 len 24576 data: 2 2 2 2 2 3
6648 214/300 Read block 1719 len 24576
6649 215/300 Write block 4930 len 32768 data: 3 3 2 2 2 2 2 2
6650 216/300 Write block 122 len 36864 data: 4 4 4 4 4 3 2 3 4
6651 217/300 Read block 627 len 12288
6652 218/300 Read block 4095 len 20480
6653 219/300 Write block 1391 len 12288 data: 2 3 3
6654 220/300 Flush
6655 221/300 Write block 4385 len 16384 data: 3 3 3 3
6656 222/300 Write block 369 len 12288 data: 2 2 2
6657 223/300 Read block 120 len 36864
6658 224/300 Write block 4673 len 32768 data: 3 2 2 2 2 2 2 2
6659 225/300 Write block 3526 len 20480 data: 2 2 2 2 2
6660 226/300 Write block 2043 len 32768 data: 4 4 3 3 3 4 4 4
6661 227/300 Read block 694 len 12288
6662 228/300 Read block 4852 len 24576
6663 229/300 Read block 4988 len 40960
6664 230/300 Write block 2149 len 4096 data: 4
6665 231/300 Read block 4666 len 16384
6666 232/300 Read block 2855 len 40960
6667 233/300 Write block 4934 len 28672 data: 3 3 3 3 2 2 3
6668 234/300 Read block 2003 len 36864
6669 235/300 Write block 4216 len 20480 data: 3 3 3 3 3
6670 236/300 Write block 664 len 36864 data: 5 4 3 2 2 2 2 3 3
6671 237/300 Read block 2341 len 36864
6672 238/300 Write block 422 len 4096 data: 2
6673 239/300 Write block 3023 len 32768 data: 2 2 2 2 2 2 3 3
6674 240/300 Write block 563 len 8192 data: 3 3
6675 241/300 Flush
6676 242/300 Write block 405 len 8192 data: 3 3
6677 243/300 Write block 1454 len 36864 data: 3 3 3 3 3 3 2 2 2
6678 244/300 Read block 2277 len 40960
6679 245/300 Read block 462 len 36864
6680 246/300 Write block 3346 len 28672 data: 2 2 2 2 2 2 2
6681 247/300 Read block 235 len 28672
6682 248/300 Read block 3794 len 4096
6683 249/300 Write block 431 len 8192 data: 3 3
6684 250/300 Flush
6685 251/300 Read block 2362 len 36864
6686 252/300 Read block 2516 len 8192
6687 253/300 Write block 2144 len 20480 data: 4 4 4 3 3
6688 254/300 Write block 753 len 8192 data: 2 2
6689 255/300 Read block 3644 len 4096
6690 256/300 Write block 2832 len 28672 data: 3 3 3 3 2 2 2
6691 257/300 Read block 3592 len 28672
6692 258/300 Read block 3197 len 40960
6693 259/300 Write block 4627 len 40960 data: 2 2 2 3 3 2 3 4 4 4
6694 260/300 Flush
6695 261/300 Flush
6696 262/300 Read block 1395 len 40960
6697 263/300 Read block 2852 len 16384
6698 264/300 Read block 3345 len 20480
6699 265/300 Read block 482 len 4096
6700 266/300 Write block 2818 len 16384 data: 2 2 2 2
6701 267/300 Read block 3164 len 32768
6702 268/300 Read block 1203 len 36864
6703 269/300 Write block 1622 len 16384 data: 3 3 3 3
6704 270/300 Write block 1254 len 36864 data: 2 2 2 2 2 2 2 2 3
6705 271/300 Write block 32 len 24576 data: 2 2 2 3 2 2
6706 272/300 Write block 3586 len 8192 data: 3 3
6707 273/300 Write block 3726 len 8192 data: 2 3
6708 274/300 Write block 978 len 32768 data: 2 3 3 4 4 4 4 2
6709 275/300 Read block 222 len 24576
6710 276/300 Read block 2122 len 40960
6711 277/300 Read block 3325 len 28672
6712 278/300 Write block 341 len 16384 data: 3 3 3 3
6713 279/300 Read block 2758 len 36864
6714 280/300 Read block 2947 len 12288
6715 281/300 Flush
6716 282/300 Read block 1206 len 8192
6717 283/300 Write block 3413 len 16384 data: 3 3 3 3
6718 284/300 Read block 2406 len 8192
6719 285/300 Read block 2969 len 36864
6720 286/300 Read block 18 len 16384
6721 287/300 Read block 165 len 16384
6722 288/300 Read block 3755 len 40960
6723 289/300 Write block 680 len 24576 data: 3 2 2 2 2 2
6724 290/300 Read block 2203 len 40960
6725 291/300 Write block 93 len 40960 data: 3 3 3 3 3 3 3 3 3 3
6726 292/300 Write block 687 len 16384 data: 2 2 2 2
6727 293/300 Read block 4274 len 20480
6728 294/300 Write block 4632 len 8192 data: 3 4
6729 295/300 Read block 2312 len 20480
6730 296/300 Write block 2990 len 24576 data: 4 4 5 4 4 3
6731 297/300 Read block 324 len 32768
6732 298/300 Write block 3565 len 32768 data: 2 2 2 3 3 5 4 4
6733 299/300 Read block 3948 len 4096
6734 300/300 Flush
6735 301/300 Write block 4028 len 8192 data: 3 2
6736 Main task has work to do, go find it
6737 got dsc Start(1)
6738 start 1
6739 Mar 17 22:36:02.353 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: a18ba029-0749-4c11-96ba-b3b739c165e1, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
6740 Replay: started 1, returned:Ok(())
6741 ----------------------------------------------------------------
6742 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6743 States: Active Offline Active
6744 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6745 2510 Acked Write 9 Done Sent Done false
6746 2511 Acked Read 8 Done Sent Done false
6747 2512 Acked Read 10 Done Sent Done false
6748 2513 Acked Read 5 Done Sent Done false
6749 2514 Acked Read 10 Done Sent Done false
6750 2515 Acked Write 4 Done Sent Done false
6751 2516 Acked Flush 0 Done Sent Done false
6752 2517 Acked Flush 0 Done Sent Done false
6753 2518 Acked Read 7 Done Sent Done false
6754 2519 Acked Read 6 Done Sent Done false
6755 2520 Acked Write 4 Done Sent Done false
6756 2521 Acked Read 6 Done Sent Done false
6757 2522 Acked Write 1 Done Sent Done false
6758 2523 Acked Write 3 Done Sent Done false
6759 2524 Acked Read 6 Done Sent Done false
6760 2525 Acked Read 4 Done Sent Done false
6761 2526 Acked Read 10 Done Sent Done false
6762 2527 Acked Read 5 Done Sent Done false
6763 2528 Acked Read 1 Done Sent Done false
6764 2529 Acked Read 8 Done Sent Done false
6765 2530 Acked Read 5 Done Sent Done false
6766 2531 Acked Read 7 Done Sent Done false
6767 2532 Acked Write 3 Done Sent Done false
6768 2533 Acked Read 1 Done Sent Done false
6769 2534 Acked Write 3 Done Sent Done false
6770 2535 Acked Write 5 Done Sent Done false
6771 2536 Acked Read 10 Done Sent Done false
6772 2537 Acked Write 2 Done Sent Done false
6773 2538 Acked Read 10 Done Sent Done false
6774 2539 Acked Write 4 Done Sent Done false
6775 2540 Acked Read 6 Done Sent Done false
6776 2541 Acked Read 9 Done Sent Done false
6777 2542 Acked Write 5 Done Sent Done false
6778 2543 Acked Read 2 Done Sent Done false
6779 2544 Acked Read 4 Done Sent Done false
6780 2545 Acked Read 4 Done Sent Done false
6781 2546 Acked Write 3 Done Sent Done false
6782 2547 Acked Write 8 Done Sent Done false
6783 2548 Acked Read 9 Done Sent Done false
6784 2549 Acked Write 5 Done Sent Done false
6785 2550 Acked Write 9 Done Sent Done false
6786 2551 Acked Write 5 Done Sent Done false
6787 2552 Acked Flush 0 Done Sent Done false
6788 2553 Acked Read 9 Done Sent Done false
6789 2554 Acked Read 6 Done Sent Done false
6790 2555 Acked Flush 0 Done Sent Done false
6791 2556 Acked Read 5 Done Sent Done false
6792 2557 Acked Write 1 Done Sent Done false
6793 2558 Acked Write 10 Done Sent Done false
6794 2559 Acked Flush 0 Done Sent Done false
6795 2560 Acked Read 2 Done Sent Done false
6796 2561 Acked Flush 0 Done Sent Done false
6797 2562 Acked Flush 0 Done Sent Done false
6798 2563 Acked Read 2 Done Sent Done false
6799 2564 Acked Read 8 Done Sent Done false
6800 2565 Acked Read 6 Done Sent Done false
6801 2566 Acked Read 9 Done Sent Done false
6802 2567 Acked Write 3 Done Sent Done false
6803 2568 Acked Read 10 Done Sent Done false
6804 2569 Acked Read 7 Done Sent Done false
6805 2570 Acked Read 6 Done Sent Done false
6806 2571 Acked Write 4 Done Sent Done false
6807 2572 Acked Read 7 Done Sent Done false
6808 2573 Acked Read 5 Done Sent Done false
6809 2574 Acked Write 8 Done Sent Done false
6810 2575 Acked Read 8 Done Sent Done false
6811 2576 Acked Write 10 Done Sent Done false
6812 2577 Acked Read 6 Done Sent Done false
6813 2578 Acked Flush 0 Done Sent Done false
6814 2579 Acked Read 1 Done Sent Done false
6815 2580 Acked Write 6 Done Sent Done false
6816 2581 Acked Flush 0 Done Sent Done false
6817 2582 Acked Write 6 Done Sent Done false
6818 2583 Acked Write 9 Done Sent Done false
6819 2584 Acked Read 4 Done Sent Done false
6820 2585 Acked Flush 0 Done Sent Done false
6821 2586 Acked Read 4 Done Sent Done false
6822 2587 Acked Read 6 Done Sent Done false
6823 2588 Acked Read 10 Done Sent Done false
6824 2589 Acked Read 3 Done Sent Done false
6825 2590 Acked Flush 0 Done Sent Done false
6826 2591 Acked Read 3 Done Sent Done false
6827 2592 Acked Read 6 Done Sent Done false
6828 2593 Acked Read 7 Done Sent Done false
6829 2594 Acked Write 3 Done Sent Done false
6830 2595 Acked Read 3 Done Sent Done false
6831 2596 Acked Read 1 Done Sent Done false
6832 2597 Acked Write 6 Done Sent Done false
6833 2598 Acked Read 2 Done Sent Done false
6834 2599 Acked Read 10 Done Sent Done false
6835 2600 Acked Write 4 Done Sent Done false
6836 2601 Acked Write 8 Done Sent Done false
6837 2602 Acked Write 10 Done Sent Done false
6838 2603 Acked Write 8 Done Sent Done false
6839 2604 Acked Read 6 Done Sent Done false
6840 2605 Acked Read 4 Done Sent Done false
6841 2606 Acked Read 7 Done Sent Done false
6842 2607 Acked Read 2 Done Sent Done false
6843 2608 Acked Read 1 Done Sent Done false
6844 2609 Acked Write 5 Done Sent Done false
6845 2610 Acked Write 8 Done Sent Done false
6846 2611 Acked Read 8 Done Sent Done false
6847 2612 Acked Flush 0 Done Sent Done false
6848 2613 Acked Read 6 Done Sent Done false
6849 2614 Acked Flush 0 Done Sent Done false
6850 2615 Acked Read 7 Done Sent Done false
6851 2616 Acked Read 9 Done Sent Done false
6852 2617 Acked Write 10 Done Sent Done false
6853 2618 Acked Read 3 Done Sent Done false
6854 2619 Acked Read 7 Done Sent Done false
6855 2620 Acked Write 8 Done Sent Done false
6856 2621 Acked Read 4 Done Sent Done false
6857 2622 Acked Read 1 Done Sent Done false
6858 2623 Acked Read 6 Done Sent Done false
6859 2624 Acked Read 4 Done Sent Done false
6860 2625 Acked Read 3 Done Sent Done false
6861 2626 Acked Flush 0 Done Sent Done false
6862 2627 Acked Write 6 Done Sent Done false
6863 2628 Acked Write 7 Done Sent Done false
6864 2629 Acked Read 5 Done Sent Done false
6865 2630 Acked Write 3 Done Sent Done false
6866 2631 Acked Flush 0 Done Sent Done false
6867 2632 Acked Flush 0 Done Sent Done false
6868 2633 Acked Write 3 Done Sent Done false
6869 2634 Acked Write 8 Done Sent Done false
6870 2635 Acked Read 8 Done Sent Done false
6871 2636 Acked Read 2 Done Sent Done false
6872 2637 Acked Write 8 Done Sent Done false
6873 2638 Acked Flush 0 Done Sent Done false
6874 2639 Acked Write 1 Done Sent Done false
6875 2640 Acked Write 10 Done Sent Done false
6876 2641 Acked Write 2 Done Sent Done false
6877 2642 Acked Flush 0 Done Sent Done false
6878 2643 Acked Read 1 Done Sent Done false
6879 2644 Acked Read 4 Done Sent Done false
6880 2645 Acked Write 5 Done Sent Done false
6881 2646 Acked Read 2 Done Sent Done false
6882 2647 Acked Write 3 Done Sent Done false
6883 2648 Acked Write 1 Done Sent Done false
6884 2649 Acked Write 1 Done Sent Done false
6885 2650 Acked Write 3 Done Sent Done false
6886 2651 Acked Read 8 Done Sent Done false
6887 2652 Acked Read 8 Done Sent Done false
6888 2653 Acked Read 7 Done Sent Done false
6889 2654 Acked Write 1 Done Sent Done false
6890 2655 Acked Read 10 Done Sent Done false
6891 2656 Acked Write 6 Done Sent Done false
6892 2657 Acked Read 5 Done Sent Done false
6893 2658 Acked Write 10 Done Sent Done false
6894 2659 Acked Write 3 Done Sent Done false
6895 2660 Acked Write 1 Done Sent Done false
6896 2661 Acked Read 10 Done Sent Done false
6897 2662 Acked Read 7 Done Sent Done false
6898 2663 Acked Read 3 Done Sent Done false
6899 2664 Acked Flush 0 Done Sent Done false
6900 2665 Acked Write 4 Done Sent Done false
6901 2666 Acked Read 8 Done Sent Done false
6902 2667 Acked Flush 0 Done Sent Done false
6903 2668 Acked Read 1 Done Sent Done false
6904 2669 Acked Read 7 Done Sent Done false
6905 2670 Acked Write 6 Done Sent Done false
6906 2671 Acked Write 8 Done Sent Done false
6907 2672 Acked Read 9 Done Sent Done false
6908 2673 Acked Read 7 Done Sent Done false
6909 2674 Acked Write 10 Done Sent Done false
6910 2675 Acked Flush 0 Done Sent Done false
6911 2676 Acked Read 10 Done Sent Done false
6912 2677 Acked Write 5 Done Sent Done false
6913 2678 Acked Write 2 Done Sent Done false
6914 2679 Acked Write 8 Done Sent Done false
6915 2680 Acked Write 2 Done Sent Done false
6916 2681 Acked Write 10 Done Sent Done false
6917 2682 Acked Write 10 Done Sent Done false
6918 2683 Acked Read 7 Done Sent Done false
6919 2684 Acked Write 4 Done Sent Done false
6920 2685 Acked Write 9 Done Sent Done false
6921 2686 Acked Read 1 Done Sent Done false
6922 2687 Acked Read 6 Done Sent Done false
6923 2688 Acked Read 4 Done Sent Done false
6924 2689 Acked Flush 0 Done Sent Done false
6925 2690 Acked Write 3 Done Sent Done false
6926 2691 Acked Read 4 Done Sent Done false
6927 2692 Acked Write 7 Done Sent Done false
6928 2693 Acked Write 9 Done Sent Done false
6929 2694 Acked Read 3 Done Sent Done false
6930 2695 Acked Read 8 Done Sent Done false
6931 2696 Acked Flush 0 Done Sent Done false
6932 2697 Acked Write 3 Done Sent Done false
6933 2698 Acked Read 1 Done Sent Done false
6934 2699 Acked Write 8 Done Sent Done false
6935 2700 Acked Flush 0 Done Sent Done false
6936 2701 Acked Write 9 Done Sent Done false
6937 2702 Acked Write 4 Done Sent Done false
6938 2703 Acked Read 5 Done Sent Done false
6939 2704 Acked Flush 0 Done Sent Done false
6940 2705 Acked Read 4 Done Sent Done false
6941 2706 Acked Read 2 Done Sent Done false
6942 2707 Acked Read 1 Done Sent Done false
6943 2708 Acked Write 1 Done Sent Done false
6944 2709 Acked Write 10 Done Sent Done false
6945 2710 Acked Write 1 Done Sent Done false
6946 2711 Acked Read 2 Done Sent Done false
6947 2712 Acked Write 10 Done Sent Done false
6948 2713 Acked Write 4 Done Sent Done false
6949 2714 Acked Read 6 Done Sent Done false
6950 2715 Acked Flush 0 Done Sent Done false
6951 2716 Acked Read 8 Done Sent Done false
6952 2717 Acked Read 7 Done Sent Done false
6953 2718 Acked Read 3 Done Sent Done false
6954 2719 Acked Write 3 Done Sent Done false
6955 2720 Acked Flush 0 Done Sent Done false
6956 2721 Acked Write 10 Done Sent Done false
6957 2722 Acked Write 6 Done Sent Done false
6958 2723 Acked Read 6 Done Sent Done false
6959 2724 Acked Write 8 Done Sent Done false
6960 2725 Acked Write 9 Done Sent Done false
6961 2726 Acked Read 3 Done Sent Done false
6962 2727 Acked Read 5 Done Sent Done false
6963 2728 Acked Write 3 Done Sent Done false
6964 2729 Acked Flush 0 Done Sent Done false
6965 2730 Acked Write 4 Done Sent Done false
6966 2731 Acked Write 3 Done Sent Done false
6967 2732 Acked Read 9 Done Sent Done false
6968 2733 Acked Write 8 Done Sent Done false
6969 2734 Acked Write 5 Done Sent Done false
6970 2735 Acked Write 8 Done Sent Done false
6971 2736 Acked Read 3 Done Sent Done false
6972 2737 Acked Read 6 Done Sent Done false
6973 2738 Acked Read 10 Done Sent Done false
6974 2739 Acked Write 1 Done Sent Done false
6975 2740 Acked Read 4 Done Sent Done false
6976 2741 Acked Read 10 Done Sent Done false
6977 2742 Acked Write 7 Done Sent Done false
6978 2743 Acked Read 9 Done Sent Done false
6979 2744 Acked Write 5 Done Sent Done false
6980 2745 Acked Write 9 Done Sent Done false
6981 2746 Acked Read 9 Done Sent Done false
6982 2747 Acked Write 1 Done Sent Done false
6983 2748 Acked Write 8 Done Sent Done false
6984 2749 Acked Write 2 Done Sent Done false
6985 2750 Acked Flush 0 Done Sent Done false
6986 2751 Acked Write 2 Done Sent Done false
6987 2752 Acked Write 9 Done Sent Done false
6988 2753 Acked Read 10 Done Sent Done false
6989 2754 Acked Read 9 Done Sent Done false
6990 2755 Acked Write 7 Done Sent Done false
6991 2756 Acked Read 7 Done Sent Done false
6992 2757 Acked Read 1 Done Sent Done false
6993 2758 Acked Write 2 Done Sent Done false
6994 2759 Acked Flush 0 Done Sent Done false
6995 2760 Acked Read 9 Done Sent Done false
6996 2761 Acked Read 2 Done Sent Done false
6997 2762 Acked Write 5 Done Sent Done false
6998 2763 Acked Write 2 Done Sent Done false
6999 2764 Acked Read 1 Done Sent Done false
7000 2765 Acked Write 7 Done Sent Done false
7001 2766 Acked Read 7 Done Sent Done false
7002 2767 Acked Read 10 Done Sent Done false
7003 2768 Acked Write 10 Done Sent Done false
7004 2769 Acked Flush 0 Done Sent Done false
7005 2770 Acked Flush 0 Done Sent Done false
7006 2771 Acked Read 10 Done Sent Done false
7007 2772 Acked Read 4 Done Sent Done false
7008 2773 Acked Read 5 Done Sent Done false
7009 2774 Acked Read 1 Done Sent Done false
7010 2775 Acked Write 4 Done Sent Done false
7011 2776 Acked Read 8 Done Sent Done false
7012 2777 Acked Read 9 Done Sent Done false
7013 2778 Acked Write 4 Done Sent Done false
7014 2779 Acked Write 9 Done Sent Done false
7015 2780 Acked Write 6 Done Sent Done false
7016 2781 Acked Write 2 Done Sent Done false
7017 2782 Acked Write 2 Done Sent Done false
7018 2783 Acked Write 8 Done Sent Done false
7019 2784 Acked Read 6 Done Sent Done false
7020 2785 Acked Read 10 Done Sent Done false
7021 2786 Acked Read 7 Done Sent Done false
7022 2787 Acked Write 4 Done Sent Done false
7023 2788 Acked Read 9 Done Sent Done false
7024 2789 Acked Read 3 Done Sent Done false
7025 2790 Acked Flush 0 Done Sent Done false
7026 2791 Acked Read 2 Done Sent Done false
7027 2792 Acked Write 4 Done Sent Done false
7028 2793 Acked Read 2 Done Sent Done false
7029 2794 Acked Read 9 Done Sent Done false
7030 2795 Acked Read 4 Done Sent Done false
7031 2796 Acked Read 4 Done Sent Done false
7032 2797 Acked Read 10 Done Sent Done false
7033 2798 Acked Write 6 Done Sent Done false
7034 2799 Acked Read 10 Done Sent Done false
7035 2800 Acked Write 10 Done Sent Done false
7036 2801 Acked Write 4 Done Sent Done false
7037 2802 Acked Read 5 Done Sent Done false
7038 2803 Acked Write 2 Done Sent Done false
7039 2804 Acked Read 5 Done Sent Done false
7040 2805 Acked Write 6 Done Sent Done false
7041 2806 Acked Read 8 Done Sent Done false
7042 2807 Acked Write 8 Done Sent Done false
7043 2808 Acked Read 1 Done Sent Done false
7044 2809 Acked Flush 0 Done Sent Done false
7045 2810 Acked Write 2 Done Sent Done false
7046 STATES DS:0 DS:1 DS:2 TOTAL
7047 Sent 0 301 0 301
7048 Done 301 0 301 602
7049 Skipped 0 0 0 0
7050 Error 0 0 0 0
7051 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
7052 Downstairs last five completed: 2509 2508 2507 2506 2505
7053 Upstairs last five completed: 2810 2809 2808 2807 2806
7054 CLIENT: Up:0 ds:301 act:2
70552025-03-17T22:36:03.444ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
70562025-03-17T22:36:04.857ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
70572025-03-17T22:36:05.447ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
7058 [8820] got start action
7059 [8820] I am going to restart
7060 Starting downstairs at port 8820
7061 Make output file at "/tmp/dsc/downstairs-8820.txt"
7062 [8820][1] reports Starting
7063 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
7064 [8820][1] reports Running
7065 ----------------------------------------------------------------
7066 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7067 States: Active Offline Active
7068 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7069 2510 Acked Write 9 Done Sent Done false
7070 2511 Acked Read 8 Done Sent Done false
7071 2512 Acked Read 10 Done Sent Done false
7072 2513 Acked Read 5 Done Sent Done false
7073 2514 Acked Read 10 Done Sent Done false
7074 2515 Acked Write 4 Done Sent Done false
7075 2516 Acked Flush 0 Done Sent Done false
7076 2517 Acked Flush 0 Done Sent Done false
7077 2518 Acked Read 7 Done Sent Done false
7078 2519 Acked Read 6 Done Sent Done false
7079 2520 Acked Write 4 Done Sent Done false
7080 2521 Acked Read 6 Done Sent Done false
7081 2522 Acked Write 1 Done Sent Done false
7082 2523 Acked Write 3 Done Sent Done false
7083 2524 Acked Read 6 Done Sent Done false
7084 2525 Acked Read 4 Done Sent Done false
7085 2526 Acked Read 10 Done Sent Done false
7086 2527 Acked Read 5 Done Sent Done false
7087 2528 Acked Read 1 Done Sent Done false
7088 2529 Acked Read 8 Done Sent Done false
7089 2530 Acked Read 5 Done Sent Done false
7090 2531 Acked Read 7 Done Sent Done false
7091 2532 Acked Write 3 Done Sent Done false
7092 2533 Acked Read 1 Done Sent Done false
7093 2534 Acked Write 3 Done Sent Done false
7094 2535 Acked Write 5 Done Sent Done false
7095 2536 Acked Read 10 Done Sent Done false
7096 2537 Acked Write 2 Done Sent Done false
7097 2538 Acked Read 10 Done Sent Done false
7098 2539 Acked Write 4 Done Sent Done false
7099 2540 Acked Read 6 Done Sent Done false
7100 2541 Acked Read 9 Done Sent Done false
7101 2542 Acked Write 5 Done Sent Done false
7102 2543 Acked Read 2 Done Sent Done false
7103 2544 Acked Read 4 Done Sent Done false
7104 2545 Acked Read 4 Done Sent Done false
7105 2546 Acked Write 3 Done Sent Done false
7106 2547 Acked Write 8 Done Sent Done false
7107 2548 Acked Read 9 Done Sent Done false
7108 2549 Acked Write 5 Done Sent Done false
7109 2550 Acked Write 9 Done Sent Done false
7110 2551 Acked Write 5 Done Sent Done false
7111 2552 Acked Flush 0 Done Sent Done false
7112 2553 Acked Read 9 Done Sent Done false
7113 2554 Acked Read 6 Done Sent Done false
7114 2555 Acked Flush 0 Done Sent Done false
7115 2556 Acked Read 5 Done Sent Done false
7116 2557 Acked Write 1 Done Sent Done false
7117 2558 Acked Write 10 Done Sent Done false
7118 2559 Acked Flush 0 Done Sent Done false
7119 2560 Acked Read 2 Done Sent Done false
7120 2561 Acked Flush 0 Done Sent Done false
7121 2562 Acked Flush 0 Done Sent Done false
7122 2563 Acked Read 2 Done Sent Done false
7123 2564 Acked Read 8 Done Sent Done false
7124 2565 Acked Read 6 Done Sent Done false
7125 2566 Acked Read 9 Done Sent Done false
7126 2567 Acked Write 3 Done Sent Done false
7127 2568 Acked Read 10 Done Sent Done false
7128 2569 Acked Read 7 Done Sent Done false
7129 2570 Acked Read 6 Done Sent Done false
7130 2571 Acked Write 4 Done Sent Done false
7131 2572 Acked Read 7 Done Sent Done false
7132 2573 Acked Read 5 Done Sent Done false
7133 2574 Acked Write 8 Done Sent Done false
7134 2575 Acked Read 8 Done Sent Done false
7135 2576 Acked Write 10 Done Sent Done false
7136 2577 Acked Read 6 Done Sent Done false
7137 2578 Acked Flush 0 Done Sent Done false
7138 2579 Acked Read 1 Done Sent Done false
7139 2580 Acked Write 6 Done Sent Done false
7140 2581 Acked Flush 0 Done Sent Done false
7141 2582 Acked Write 6 Done Sent Done false
7142 2583 Acked Write 9 Done Sent Done false
7143 2584 Acked Read 4 Done Sent Done false
7144 2585 Acked Flush 0 Done Sent Done false
7145 2586 Acked Read 4 Done Sent Done false
7146 2587 Acked Read 6 Done Sent Done false
7147 2588 Acked Read 10 Done Sent Done false
7148 2589 Acked Read 3 Done Sent Done false
7149 2590 Acked Flush 0 Done Sent Done false
7150 2591 Acked Read 3 Done Sent Done false
7151 2592 Acked Read 6 Done Sent Done false
7152 2593 Acked Read 7 Done Sent Done false
7153 2594 Acked Write 3 Done Sent Done false
7154 2595 Acked Read 3 Done Sent Done false
7155 2596 Acked Read 1 Done Sent Done false
7156 2597 Acked Write 6 Done Sent Done false
7157 2598 Acked Read 2 Done Sent Done false
7158 2599 Acked Read 10 Done Sent Done false
7159 2600 Acked Write 4 Done Sent Done false
7160 2601 Acked Write 8 Done Sent Done false
7161 2602 Acked Write 10 Done Sent Done false
7162 2603 Acked Write 8 Done Sent Done false
7163 2604 Acked Read 6 Done Sent Done false
7164 2605 Acked Read 4 Done Sent Done false
7165 2606 Acked Read 7 Done Sent Done false
7166 2607 Acked Read 2 Done Sent Done false
7167 2608 Acked Read 1 Done Sent Done false
7168 2609 Acked Write 5 Done Sent Done false
7169 2610 Acked Write 8 Done Sent Done false
7170 2611 Acked Read 8 Done Sent Done false
7171 2612 Acked Flush 0 Done Sent Done false
7172 2613 Acked Read 6 Done Sent Done false
7173 2614 Acked Flush 0 Done Sent Done false
7174 2615 Acked Read 7 Done Sent Done false
7175 2616 Acked Read 9 Done Sent Done false
7176 2617 Acked Write 10 Done Sent Done false
7177 2618 Acked Read 3 Done Sent Done false
7178 2619 Acked Read 7 Done Sent Done false
7179 2620 Acked Write 8 Done Sent Done false
7180 2621 Acked Read 4 Done Sent Done false
7181 2622 Acked Read 1 Done Sent Done false
7182 2623 Acked Read 6 Done Sent Done false
7183 2624 Acked Read 4 Done Sent Done false
7184 2625 Acked Read 3 Done Sent Done false
7185 2626 Acked Flush 0 Done Sent Done false
7186 2627 Acked Write 6 Done Sent Done false
7187 2628 Acked Write 7 Done Sent Done false
7188 2629 Acked Read 5 Done Sent Done false
7189 2630 Acked Write 3 Done Sent Done false
7190 2631 Acked Flush 0 Done Sent Done false
7191 2632 Acked Flush 0 Done Sent Done false
7192 2633 Acked Write 3 Done Sent Done false
7193 2634 Acked Write 8 Done Sent Done false
7194 2635 Acked Read 8 Done Sent Done false
7195 2636 Acked Read 2 Done Sent Done false
7196 2637 Acked Write 8 Done Sent Done false
7197 2638 Acked Flush 0 Done Sent Done false
7198 2639 Acked Write 1 Done Sent Done false
7199 2640 Acked Write 10 Done Sent Done false
7200 2641 Acked Write 2 Done Sent Done false
7201 2642 Acked Flush 0 Done Sent Done false
7202 2643 Acked Read 1 Done Sent Done false
7203 2644 Acked Read 4 Done Sent Done false
7204 2645 Acked Write 5 Done Sent Done false
7205 2646 Acked Read 2 Done Sent Done false
7206 2647 Acked Write 3 Done Sent Done false
7207 2648 Acked Write 1 Done Sent Done false
7208 2649 Acked Write 1 Done Sent Done false
7209 2650 Acked Write 3 Done Sent Done false
7210 2651 Acked Read 8 Done Sent Done false
7211 2652 Acked Read 8 Done Sent Done false
7212 2653 Acked Read 7 Done Sent Done false
7213 2654 Acked Write 1 Done Sent Done false
7214 2655 Acked Read 10 Done Sent Done false
7215 2656 Acked Write 6 Done Sent Done false
7216 2657 Acked Read 5 Done Sent Done false
7217 2658 Acked Write 10 Done Sent Done false
7218 2659 Acked Write 3 Done Sent Done false
7219 2660 Acked Write 1 Done Sent Done false
7220 2661 Acked Read 10 Done Sent Done false
7221 2662 Acked Read 7 Done Sent Done false
7222 2663 Acked Read 3 Done Sent Done false
7223 2664 Acked Flush 0 Done Sent Done false
7224 2665 Acked Write 4 Done Sent Done false
7225 2666 Acked Read 8 Done Sent Done false
7226 2667 Acked Flush 0 Done Sent Done false
7227 2668 Acked Read 1 Done Sent Done false
7228 2669 Acked Read 7 Done Sent Done false
7229 2670 Acked Write 6 Done Sent Done false
7230 2671 Acked Write 8 Done Sent Done false
7231 2672 Acked Read 9 Done Sent Done false
7232 2673 Acked Read 7 Done Sent Done false
7233 2674 Acked Write 10 Done Sent Done false
7234 2675 Acked Flush 0 Done Sent Done false
7235 2676 Acked Read 10 Done Sent Done false
7236 2677 Acked Write 5 Done Sent Done false
7237 2678 Acked Write 2 Done Sent Done false
7238 2679 Acked Write 8 Done Sent Done false
7239 2680 Acked Write 2 Done Sent Done false
7240 2681 Acked Write 10 Done Sent Done false
7241 2682 Acked Write 10 Done Sent Done false
7242 2683 Acked Read 7 Done Sent Done false
7243 2684 Acked Write 4 Done Sent Done false
7244 2685 Acked Write 9 Done Sent Done false
7245 2686 Acked Read 1 Done Sent Done false
7246 2687 Acked Read 6 Done Sent Done false
7247 2688 Acked Read 4 Done Sent Done false
7248 2689 Acked Flush 0 Done Sent Done false
7249 2690 Acked Write 3 Done Sent Done false
7250 2691 Acked Read 4 Done Sent Done false
7251 2692 Acked Write 7 Done Sent Done false
7252 2693 Acked Write 9 Done Sent Done false
7253 2694 Acked Read 3 Done Sent Done false
7254 2695 Acked Read 8 Done Sent Done false
7255 2696 Acked Flush 0 Done Sent Done false
7256 2697 Acked Write 3 Done Sent Done false
7257 2698 Acked Read 1 Done Sent Done false
7258 2699 Acked Write 8 Done Sent Done false
7259 2700 Acked Flush 0 Done Sent Done false
7260 2701 Acked Write 9 Done Sent Done false
7261 2702 Acked Write 4 Done Sent Done false
7262 2703 Acked Read 5 Done Sent Done false
7263 2704 Acked Flush 0 Done Sent Done false
7264 2705 Acked Read 4 Done Sent Done false
7265 2706 Acked Read 2 Done Sent Done false
7266 2707 Acked Read 1 Done Sent Done false
7267 2708 Acked Write 1 Done Sent Done false
7268 2709 Acked Write 10 Done Sent Done false
7269 2710 Acked Write 1 Done Sent Done false
7270 2711 Acked Read 2 Done Sent Done false
7271 2712 Acked Write 10 Done Sent Done false
7272 2713 Acked Write 4 Done Sent Done false
7273 2714 Acked Read 6 Done Sent Done false
7274 2715 Acked Flush 0 Done Sent Done false
7275 2716 Acked Read 8 Done Sent Done false
7276 2717 Acked Read 7 Done Sent Done false
7277 2718 Acked Read 3 Done Sent Done false
7278 2719 Acked Write 3 Done Sent Done false
7279 2720 Acked Flush 0 Done Sent Done false
7280 2721 Acked Write 10 Done Sent Done false
7281 2722 Acked Write 6 Done Sent Done false
7282 2723 Acked Read 6 Done Sent Done false
7283 2724 Acked Write 8 Done Sent Done false
7284 2725 Acked Write 9 Done Sent Done false
7285 2726 Acked Read 3 Done Sent Done false
7286 2727 Acked Read 5 Done Sent Done false
7287 2728 Acked Write 3 Done Sent Done false
7288 2729 Acked Flush 0 Done Sent Done false
7289 2730 Acked Write 4 Done Sent Done false
7290 2731 Acked Write 3 Done Sent Done false
7291 2732 Acked Read 9 Done Sent Done false
7292 2733 Acked Write 8 Done Sent Done false
7293 2734 Acked Write 5 Done Sent Done false
7294 2735 Acked Write 8 Done Sent Done false
7295 2736 Acked Read 3 Done Sent Done false
7296 2737 Acked Read 6 Done Sent Done false
7297 2738 Acked Read 10 Done Sent Done false
7298 2739 Acked Write 1 Done Sent Done false
7299 2740 Acked Read 4 Done Sent Done false
7300 2741 Acked Read 10 Done Sent Done false
7301 2742 Acked Write 7 Done Sent Done false
7302 2743 Acked Read 9 Done Sent Done false
7303 2744 Acked Write 5 Done Sent Done false
7304 2745 Acked Write 9 Done Sent Done false
7305 2746 Acked Read 9 Done Sent Done false
7306 2747 Acked Write 1 Done Sent Done false
7307 2748 Acked Write 8 Done Sent Done false
7308 2749 Acked Write 2 Done Sent Done false
7309 2750 Acked Flush 0 Done Sent Done false
7310 2751 Acked Write 2 Done Sent Done false
7311 2752 Acked Write 9 Done Sent Done false
7312 2753 Acked Read 10 Done Sent Done false
7313 2754 Acked Read 9 Done Sent Done false
7314 2755 Acked Write 7 Done Sent Done false
7315 2756 Acked Read 7 Done Sent Done false
7316 2757 Acked Read 1 Done Sent Done false
7317 2758 Acked Write 2 Done Sent Done false
7318 2759 Acked Flush 0 Done Sent Done false
7319 2760 Acked Read 9 Done Sent Done false
7320 2761 Acked Read 2 Done Sent Done false
7321 2762 Acked Write 5 Done Sent Done false
7322 2763 Acked Write 2 Done Sent Done false
7323 2764 Acked Read 1 Done Sent Done false
7324 2765 Acked Write 7 Done Sent Done false
7325 2766 Acked Read 7 Done Sent Done false
7326 2767 Acked Read 10 Done Sent Done false
7327 2768 Acked Write 10 Done Sent Done false
7328 2769 Acked Flush 0 Done Sent Done false
7329 2770 Acked Flush 0 Done Sent Done false
7330 2771 Acked Read 10 Done Sent Done false
7331 2772 Acked Read 4 Done Sent Done false
7332 2773 Acked Read 5 Done Sent Done false
7333 2774 Acked Read 1 Done Sent Done false
7334 2775 Acked Write 4 Done Sent Done false
7335 2776 Acked Read 8 Done Sent Done false
7336 2777 Acked Read 9 Done Sent Done false
7337 2778 Acked Write 4 Done Sent Done false
7338 2779 Acked Write 9 Done Sent Done false
7339 2780 Acked Write 6 Done Sent Done false
7340 2781 Acked Write 2 Done Sent Done false
7341 2782 Acked Write 2 Done Sent Done false
7342 2783 Acked Write 8 Done Sent Done false
7343 2784 Acked Read 6 Done Sent Done false
7344 2785 Acked Read 10 Done Sent Done false
7345 2786 Acked Read 7 Done Sent Done false
7346 2787 Acked Write 4 Done Sent Done false
7347 2788 Acked Read 9 Done Sent Done false
7348 2789 Acked Read 3 Done Sent Done false
7349 2790 Acked Flush 0 Done Sent Done false
7350 2791 Acked Read 2 Done Sent Done false
7351 2792 Acked Write 4 Done Sent Done false
7352 2793 Acked Read 2 Done Sent Done false
7353 2794 Acked Read 9 Done Sent Done false
7354 2795 Acked Read 4 Done Sent Done false
7355 2796 Acked Read 4 Done Sent Done false
7356 2797 Acked Read 10 Done Sent Done false
7357 2798 Acked Write 6 Done Sent Done false
7358 2799 Acked Read 10 Done Sent Done false
7359 2800 Acked Write 10 Done Sent Done false
7360 2801 Acked Write 4 Done Sent Done false
7361 2802 Acked Read 5 Done Sent Done false
7362 2803 Acked Write 2 Done Sent Done false
7363 2804 Acked Read 5 Done Sent Done false
7364 2805 Acked Write 6 Done Sent Done false
7365 2806 Acked Read 8 Done Sent Done false
7366 2807 Acked Write 8 Done Sent Done false
7367 2808 Acked Read 1 Done Sent Done false
7368 2809 Acked Flush 0 Done Sent Done false
7369 2810 Acked Write 2 Done Sent Done false
7370 STATES DS:0 DS:1 DS:2 TOTAL
7371 Sent 0 301 0 301
7372 Done 301 0 301 602
7373 Skipped 0 0 0 0
7374 Error 0 0 0 0
7375 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
7376 Downstairs last five completed: 2509 2508 2507 2506 2505
7377 Upstairs last five completed: 2810 2809 2808 2807 2806
7378 CLIENT: Up:0 ds:301 act:2
73792025-03-17T22:36:06.857ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73802025-03-17T22:36:07.450ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73812025-03-17T22:36:08.106ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73822025-03-17T22:36:08.106ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39746) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73832025-03-17T22:36:08.107ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73842025-03-17T22:36:08.107ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73852025-03-17T22:36:08.107ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73862025-03-17T22:36:08.107ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73872025-03-17T22:36:08.107ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73882025-03-17T22:36:08.107ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
73892025-03-17T22:36:08.108ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
7390 ----------------------------------------------------------------
7391 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7392 Downstairs last five completed: 2811 2810 2809 2808 2807
7393 Upstairs last five completed: 2811 2810 2809 2808 2807
7394 CLIENT: Up:0 ds:0 act:3
7395 Replay: All jobs finished
7396 Main task has work to do, go find it
7397 got dsc Stop(2)
7398 stop 2
7399 [8830] Got stop action so:false kr:false
7400 Mar 17 22:36:10.358 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f972473b-9647-4e52-8339-a203fdbda6e7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
7401 Mar 17 22:36:10.359 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:36:10.359486388Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
7402 INFO request completed, latency_us: 116, response_code: 200, uri: /state/cid/2, method: GET, req_id: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:36:10.359559974Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
7403 8c4fa580-252b-40aa-93f3-fdc0b3233fd7, remote_addr: 127.0.0.1:35180{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:36:10.359607088Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
7404 , local_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:36:10.359644811Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
7405 127.0.0.1:9998
7406 [8830] Exited with: ExitStatus(unix_wait_status(9))
7407 [8830][2] reports Exit
7408 Mar 17 22:36:14.362 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 73509ec4-26e2-49bc-8688-ed330fa2072c, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
7409 001/300 Read block 3691 len 36864
7410 002/300 Write block 329 len 16384 data: 2 2 2 2
7411 003/300 Flush
7412 004/300 Write block 3212 len 32768 data: 5 5 4 4 4 4 4 3
7413 005/300 Write block 1152 len 4096 data: 3
7414 006/300 Write block 4265 len 12288 data: 6 6 5
7415 007/300 Write block 2859 len 24576 data: 2 2 2 2 2 2
7416 008/300 Read block 4364 len 32768
7417 009/300 Write block 3343 len 20480 data: 2 2 2 3 3
7418 010/300 Read block 3610 len 32768
7419 011/300 Read block 4343 len 24576
7420 012/300 Write block 1918 len 40960 data: 3 3 3 3 2 3 3 3 3 3
7421 013/300 Write block 4138 len 36864 data: 2 2 2 2 3 3 3 3 3
7422 014/300 Write block 943 len 28672 data: 5 5 5 4 4 3 3
7423 015/300 Write block 3574 len 36864 data: 4 3 2 2 2 2 2 3 3
7424 016/300 Write block 1512 len 8192 data: 3 3
7425 017/300 Read block 4352 len 4096
7426 018/300 Read block 1233 len 36864
7427 019/300 Write block 3483 len 24576 data: 3 2 3 3 3 3
7428 020/300 Flush
7429 021/300 Read block 4435 len 12288
7430 022/300 Write block 1414 len 20480 data: 3 4 4 4 4
7431 023/300 Write block 3975 len 24576 data: 2 3 3 3 2 2
7432 024/300 Write block 2460 len 36864 data: 2 2 2 2 2 3 3 3 3
7433 025/300 Read block 2172 len 36864
7434 026/300 Write block 108 len 16384 data: 3 3 3 3
7435 027/300 Write block 125 len 32768 data: 5 5 4 3 4 5 4 4
7436 028/300 Write block 494 len 4096 data: 2
7437 029/300 Write block 1593 len 40960 data: 3 3 4 4 4 4 4 4 4 3
7438 030/300 Write block 3299 len 28672 data: 2 2 2 2 2 3 4
7439 031/300 Read block 1235 len 28672
7440 032/300 Write block 1261 len 8192 data: 3 4
7441 033/300 Flush
7442 034/300 Flush
7443 035/300 Read block 4277 len 8192
7444 036/300 Write block 1425 len 12288 data: 2 2 2
7445 037/300 Read block 2808 len 32768
7446 038/300 Read block 2832 len 16384
7447 039/300 Read block 1833 len 32768
7448 040/300 Read block 3346 len 24576
7449 041/300 Write block 3384 len 32768 data: 3 3 3 3 3 2 2 2
7450 042/300 Read block 3400 len 28672
7451 043/300 Flush
7452 044/300 Read block 3039 len 12288
7453 045/300 Write block 2550 len 28672 data: 3 3 2 2 2 2 3
7454 046/300 Read block 1111 len 32768
7455 047/300 Read block 4046 len 12288
7456 048/300 Read block 640 len 4096
7457 049/300 Write block 4114 len 36864 data: 4 4 4 3 4 4 3 2 2
7458 050/300 Write block 527 len 36864 data: 2 2 2 2 2 2 3 4 4
7459 051/300 Flush
7460 052/300 Read block 2608 len 4096
7461 053/300 Read block 4942 len 24576
7462 054/300 Read block 469 len 12288
7463 055/300 Write block 2040 len 4096 data: 4
7464 056/300 Flush
7465 057/300 Flush
7466 058/300 Write block 1533 len 20480 data: 5 5 4 4 2
7467 059/300 Read block 105 len 8192
7468 060/300 Read block 1480 len 4096
7469 061/300 Write block 925 len 24576 data: 3 3 3 3 4 3
7470 062/300 Write block 2887 len 28672 data: 2 2 2 2 2 2 2
7471 063/300 Write block 1880 len 28672 data: 2 2 3 2 3 3 3
7472 064/300 Write block 2423 len 20480 data: 2 2 2 2 2
7473 065/300 Write block 1175 len 20480 data: 2 2 2 3 3
7474 066/300 Read block 2708 len 20480
7475 067/300 Read block 3440 len 32768
7476 068/300 Write block 1751 len 4096 data: 2
7477 069/300 Flush
7478 070/300 Write block 3874 len 16384 data: 4 4 3 4
7479 071/300 Flush
7480 072/300 Write block 4500 len 40960 data: 3 5 5 5 4 4 4 3 2 2
7481 073/300 Read block 2040 len 12288
7482 074/300 Flush
7483 075/300 Flush
7484 076/300 Write block 1226 len 20480 data: 2 2 2 3 3
7485 077/300 Read block 311 len 16384
7486 078/300 Write block 486 len 28672 data: 3 3 3 2 2 2 3
7487 079/300 Read block 237 len 16384
7488 080/300 Flush
7489 081/300 Write block 736 len 28672 data: 2 2 2 2 2 3 3
7490 082/300 Write block 967 len 4096 data: 3
7491 083/300 Read block 2031 len 12288
7492 084/300 Read block 4722 len 12288
7493 085/300 Read block 438 len 20480
7494 086/300 Read block 2246 len 36864
7495 087/300 Read block 2290 len 20480
7496 088/300 Write block 174 len 24576 data: 3 4 4 4 5 3
7497 089/300 Read block 719 len 32768
7498 090/300 Write block 3686 len 28672 data: 2 2 2 2 2 2 2
7499 091/300 Read block 1986 len 12288
7500 092/300 Write block 4987 len 12288 data: 2 2 2
7501 093/300 Write block 2020 len 28672 data: 2 2 2 2 2 2 3
7502 094/300 Read block 2991 len 4096
7503 095/300 Read block 1780 len 20480
7504 096/300 Flush
7505 097/300 Read block 1773 len 12288
7506 098/300 Read block 3594 len 4096
7507 099/300 Write block 2516 len 40960 data: 3 3 3 3 2 2 2 2 2 2
7508 100/300 Write block 1761 len 8192 data: 2 2
7509 101/300 Read block 2386 len 28672
7510 102/300 Write block 4596 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7511 103/300 Read block 4667 len 16384
7512 104/300 Read block 2542 len 16384
7513 105/300 Write block 460 len 20480 data: 2 2 2 2 2
7514 106/300 Write block 793 len 4096 data: 3
7515 107/300 Flush
7516 108/300 Read block 1486 len 8192
7517 109/300 Write block 302 len 20480 data: 3 3 3 3 3
7518 110/300 Read block 4307 len 40960
7519 111/300 Flush
7520 112/300 Read block 1 len 28672
7521 113/300 Read block 3454 len 24576
7522 114/300 Write block 4351 len 28672 data: 5 5 5 5 4 4 4
7523 115/300 Read block 3227 len 8192
7524 116/300 Read block 3540 len 36864
7525 117/300 Read block 349 len 16384
7526 118/300 Flush
7527 119/300 Read block 278 len 4096
7528 120/300 Read block 2205 len 36864
7529 121/300 Write block 1279 len 8192 data: 2 2
7530 122/300 Write block 1877 len 8192 data: 3 3
7531 123/300 Flush
7532 124/300 Read block 4099 len 8192
7533 125/300 Write block 3742 len 8192 data: 2 3
7534 126/300 Write block 2804 len 8192 data: 3 3
7535 127/300 Write block 330 len 36864 data: 3 3 3 2 2 3 3 3 3
7536 128/300 Read block 2198 len 24576
7537 129/300 Flush
7538 130/300 Write block 2086 len 24576 data: 3 3 3 3 3 2
7539 131/300 Read block 1839 len 8192
7540 132/300 Flush
7541 133/300 Write block 1706 len 24576 data: 2 2 2 2 2 2
7542 134/300 Read block 4494 len 8192
7543 135/300 Read block 4823 len 8192
7544 136/300 Write block 286 len 24576 data: 2 2 2 3 3 3
7545 137/300 Write block 3671 len 36864 data: 3 2 2 2 2 2 2 3 4
7546 138/300 Write block 1340 len 28672 data: 2 2 2 2 2 2 2
7547 139/300 Read block 4937 len 16384
7548 140/300 Read block 4538 len 8192
7549 141/300 Write block 550 len 16384 data: 3 3 3 3
7550 142/300 Write block 698 len 36864 data: 2 2 2 2 2 2 2 2 2
7551 143/300 Read block 1693 len 16384
7552 144/300 Write block 2756 len 32768 data: 2 2 2 2 2 2 2 2
7553 145/300 Read block 3525 len 8192
7554 146/300 Read block 1842 len 32768
7555 147/300 Read block 4115 len 32768
7556 148/300 Read block 434 len 36864
7557 149/300 Read block 2662 len 24576
7558 150/300 Write block 2978 len 36864 data: 2 2 2 2 2 2 2 2 4
7559 151/300 Read block 968 len 4096
7560 152/300 Write block 2830 len 32768 data: 2 3 4 4 4 4 3 3
7561 153/300 Write block 1394 len 20480 data: 3 3 4 4 3
7562 154/300 Read block 1885 len 16384
7563 155/300 Write block 3175 len 36864 data: 4 4 3 4 4 4 5 5 4
7564 156/300 Read block 1124 len 40960
7565 157/300 Read block 2226 len 36864
7566 158/300 Read block 2259 len 40960
7567 159/300 Read block 1470 len 20480
7568 160/300 Write block 4026 len 8192 data: 3 3
7569 161/300 Flush
7570 162/300 Read block 4486 len 24576
7571 163/300 Flush
7572 164/300 Write block 2146 len 4096 data: 5
7573 165/300 Read block 4344 len 4096
7574 166/300 Write block 877 len 36864 data: 3 2 2 2 3 3 3 4 3
7575 167/300 Read block 4922 len 8192
7576 168/300 Read block 4340 len 40960
7577 169/300 Read block 1185 len 12288
7578 170/300 Write block 2329 len 40960 data: 2 2 2 2 2 2 3 3 3 3
7579 171/300 Read block 1908 len 12288
7580 172/300 Flush
7581 173/300 Read block 56 len 4096
7582 174/300 Read block 3729 len 40960
7583 175/300 Write block 2572 len 12288 data: 3 3 3
7584 176/300 Read block 2375 len 40960
7585 177/300 Read block 2919 len 8192
7586 178/300 Read block 3355 len 32768
7587 179/300 Read block 2020 len 12288
7588 180/300 Write block 4728 len 12288 data: 3 3 3
7589 181/300 Read block 4025 len 20480
7590 182/300 Write block 1551 len 32768 data: 2 2 2 2 2 2 2 2
7591 183/300 Flush
7592 184/300 Write block 4363 len 24576 data: 2 2 3 3 4 4
7593 185/300 Write block 4206 len 20480 data: 3 3 3 2 2
7594 186/300 Read block 420 len 20480
7595 187/300 Write block 4578 len 40960 data: 2 2 2 2 2 3 3 3 3 3
7596 188/300 Flush
7597 189/300 Read block 1588 len 36864
7598 190/300 Read block 1215 len 24576
7599 191/300 Write block 230 len 36864 data: 3 3 3 2 2 2 2 2 3
7600 192/300 Read block 1977 len 20480
7601 193/300 Write block 3340 len 24576 data: 2 2 2 3 3 3
7602 194/300 Read block 4470 len 20480
7603 195/300 Read block 4870 len 32768
7604 196/300 Read block 3747 len 12288
7605 197/300 Read block 2246 len 36864
7606 198/300 Write block 2295 len 36864 data: 4 3 2 2 2 2 2 2 2
7607 199/300 Read block 3985 len 8192
7608 200/300 Read block 3791 len 12288
7609 201/300 Read block 1215 len 12288
7610 202/300 Write block 244 len 20480 data: 2 2 2 2 2
7611 203/300 Read block 1792 len 12288
7612 204/300 Write block 4709 len 16384 data: 3 3 3 3
7613 205/300 Write block 1719 len 12288 data: 2 2 4
7614 206/300 Read block 2633 len 28672
7615 207/300 Write block 1099 len 40960 data: 2 3 3 3 3 3 2 2 2 2
7616 208/300 Read block 1991 len 28672
7617 209/300 Read block 1839 len 20480
7618 210/300 Write block 3173 len 32768 data: 2 3 5 5 4 5 5 5
7619 211/300 Read block 2941 len 32768
7620 212/300 Write block 1252 len 8192 data: 2 2
7621 213/300 Write block 4391 len 20480 data: 4 4 3 3 2
7622 214/300 Read block 3048 len 36864
7623 215/300 Write block 257 len 36864 data: 3 3 2 2 2 2 3 4 3
7624 216/300 Write block 2038 len 28672 data: 4 4 5 4 5 5 5
7625 217/300 Write block 2281 len 32768 data: 2 2 2 2 2 2 3 2
7626 218/300 Write block 4096 len 20480 data: 4 4 3 3 3
7627 219/300 Read block 1013 len 36864
7628 220/300 Write block 3060 len 16384 data: 3 3 3 3
7629 221/300 Flush
7630 222/300 Flush
7631 223/300 Read block 3570 len 32768
7632 224/300 Write block 1656 len 20480 data: 3 3 3 3 3
7633 225/300 Flush
7634 226/300 Write block 4493 len 32768 data: 2 2 3 3 2 2 2 4
7635 227/300 Read block 2046 len 4096
7636 228/300 Read block 4908 len 12288
7637 229/300 Read block 1226 len 8192
7638 230/300 Read block 1781 len 8192
7639 231/300 Read block 3187 len 20480
7640 232/300 Read block 4366 len 40960
7641 233/300 Flush
7642 234/300 Write block 2601 len 20480 data: 2 2 2 2 2
7643 235/300 Write block 4681 len 12288 data: 2 2 2
7644 236/300 Write block 95 len 8192 data: 4 4
7645 237/300 Read block 3540 len 16384
7646 238/300 Read block 2532 len 24576
7647 239/300 Write block 415 len 32768 data: 5 4 4 4 4 4 3 3
7648 240/300 Read block 2789 len 16384
7649 241/300 Read block 4531 len 4096
7650 242/300 Flush
7651 243/300 Read block 2330 len 28672
7652 244/300 Write block 400 len 24576 data: 2 2 3 3 3 4
7653 245/300 Read block 2172 len 32768
7654 246/300 Write block 1357 len 16384 data: 3 3 3 3
7655 247/300 Flush
7656 248/300 Write block 3871 len 32768 data: 2 2 3 5 5 4 5 4
7657 249/300 Write block 1581 len 4096 data: 3
7658 250/300 Write block 4724 len 8192 data: 2 3
7659 251/300 Flush
7660 252/300 Read block 824 len 32768
7661 253/300 Read block 2790 len 36864
7662 254/300 Write block 935 len 40960 data: 3 3 3 4 4 4 5 5 6 6
7663 255/300 Flush
7664 256/300 Write block 4163 len 32768 data: 3 3 3 4 5 5 5 4
7665 257/300 Write block 3625 len 20480 data: 2 2 3 3 3
7666 258/300 Read block 4945 len 8192
7667 259/300 Write block 4798 len 4096 data: 4
7668 260/300 Flush
7669 261/300 Write block 2526 len 28672 data: 2 2 2 2 3 3 3
7670 262/300 Write block 897 len 20480 data: 2 2 2 2 3
7671 263/300 Read block 2065 len 24576
7672 264/300 Read block 200 len 16384
7673 265/300 Flush
7674 266/300 Write block 522 len 24576 data: 3 3 3 3 3 3
7675 267/300 Read block 1574 len 32768
7676 268/300 Read block 2618 len 16384
7677 269/300 Write block 2728 len 20480 data: 2 4 4 4 4
7678 270/300 Flush
7679 271/300 Write block 1292 len 16384 data: 4 4 4 4
7680 272/300 Write block 4216 len 20480 data: 4 4 4 4 4
7681 273/300 Write block 4411 len 24576 data: 2 2 3 3 3 2
7682 274/300 Read block 3400 len 16384
7683 275/300 Write block 2463 len 16384 data: 3 3 4 4
7684 276/300 Write block 3726 len 36864 data: 3 4 3 4 3 3 3 4 4
7685 277/300 Read block 994 len 12288
7686 278/300 Write block 709 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7687 279/300 Read block 2512 len 4096
7688 280/300 Read block 4583 len 20480
7689 281/300 Read block 502 len 36864
7690 282/300 Read block 2560 len 8192
7691 283/300 Write block 2872 len 8192 data: 3 3
7692 284/300 Read block 2451 len 12288
7693 285/300 Read block 2001 len 16384
7694 286/300 Write block 3713 len 16384 data: 3 2 2 2
7695 287/300 Write block 1388 len 8192 data: 3 3
7696 288/300 Write block 830 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7697 289/300 Read block 3042 len 24576
7698 290/300 Write block 423 len 12288 data: 2 2 2
7699 291/300 Read block 1875 len 8192
7700 292/300 Read block 3244 len 20480
7701 293/300 Write block 253 len 16384 data: 4 5 4 3
7702 294/300 Read block 1979 len 32768
7703 295/300 Read block 367 len 8192
7704 296/300 Write block 1432 len 24576 data: 2 2 2 3 4 3
7705 297/300 Read block 2140 len 24576
7706 298/300 Write block 1804 len 40960 data: 3 3 3 3 3 3 3 3 3 3
7707 299/300 Flush
7708 300/300 Read block 304 len 8192
7709 301/300 Read block 2162 len 36864
7710 Main task has work to do, go find it
7711 got dsc Start(2)
7712 start 2
7713 Mar 17 22:36:14.598 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: da0141bc-28ce-4936-8dec-7633170b17a6, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
7714 Replay: started 2, returned:Ok(())
7715 ----------------------------------------------------------------
7716 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7717 States: Active Active Offline
7718 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7719 2812 Acked Read 9 Done Done Sent false
7720 2813 Acked Write 4 Done Done Sent false
7721 2814 Acked Flush 0 Done Done Sent false
7722 2815 Acked Write 8 Done Done Sent false
7723 2816 Acked Write 1 Done Done Sent false
7724 2817 Acked Write 3 Done Done Sent false
7725 2818 Acked Write 6 Done Done Sent false
7726 2819 Acked Read 8 Done Done Sent false
7727 2820 Acked Write 5 Done Done Sent false
7728 2821 Acked Read 8 Done Done Sent false
7729 2822 Acked Read 6 Done Done Sent false
7730 2823 Acked Write 10 Done Done Sent false
7731 2824 Acked Write 9 Done Done Sent false
7732 2825 Acked Write 7 Done Done Sent false
7733 2826 Acked Write 9 Done Done Sent false
7734 2827 Acked Write 2 Done Done Sent false
7735 2828 Acked Read 1 Done Done Sent false
7736 2829 Acked Read 9 Done Done Sent false
7737 2830 Acked Write 6 Done Done Sent false
7738 2831 Acked Flush 0 Done Done Sent false
7739 2832 Acked Read 3 Done Done Sent false
7740 2833 Acked Write 5 Done Done Sent false
7741 2834 Acked Write 6 Done Done Sent false
7742 2835 Acked Write 9 Done Done Sent false
7743 2836 Acked Read 9 Done Done Sent false
7744 2837 Acked Write 4 Done Done Sent false
7745 2838 Acked Write 8 Done Done Sent false
7746 2839 Acked Write 1 Done Done Sent false
7747 2840 Acked Write 10 Done Done Sent false
7748 2841 Acked Write 7 Done Done Sent false
7749 2842 Acked Read 7 Done Done Sent false
7750 2843 Acked Write 2 Done Done Sent false
7751 2844 Acked Flush 0 Done Done Sent false
7752 2845 Acked Flush 0 Done Done Sent false
7753 2846 Acked Read 2 Done Done Sent false
7754 2847 Acked Write 3 Done Done Sent false
7755 2848 Acked Read 8 Done Done Sent false
7756 2849 Acked Read 4 Done Done Sent false
7757 2850 Acked Read 8 Done Done Sent false
7758 2851 Acked Read 6 Done Done Sent false
7759 2852 Acked Write 8 Done Done Sent false
7760 2853 Acked Read 7 Done Done Sent false
7761 2854 Acked Flush 0 Done Done Sent false
7762 2855 Acked Read 3 Done Done Sent false
7763 2856 Acked Write 7 Done Done Sent false
7764 2857 Acked Read 8 Done Done Sent false
7765 2858 Acked Read 3 Done Done Sent false
7766 2859 Acked Read 1 Done Done Sent false
7767 2860 Acked Write 9 Done Done Sent false
7768 2861 Acked Write 9 Done Done Sent false
7769 2862 Acked Flush 0 Done Done Sent false
7770 2863 Acked Read 1 Done Done Sent false
7771 2864 Acked Read 6 Done Done Sent false
7772 2865 Acked Read 3 Done Done Sent false
7773 2866 Acked Write 1 Done Done Sent false
7774 2867 Acked Flush 0 Done Done Sent false
7775 2868 Acked Flush 0 Done Done Sent false
7776 2869 Acked Write 5 Done Done Sent false
7777 2870 Acked Read 2 Done Done Sent false
7778 2871 Acked Read 1 Done Done Sent false
7779 2872 Acked Write 6 Done Done Sent false
7780 2873 Acked Write 7 Done Done Sent false
7781 2874 Acked Write 7 Done Done Sent false
7782 2875 Acked Write 5 Done Done Sent false
7783 2876 Acked Write 5 Done Done Sent false
7784 2877 Acked Read 5 Done Done Sent false
7785 2878 Acked Read 8 Done Done Sent false
7786 2879 Acked Write 1 Done Done Sent false
7787 2880 Acked Flush 0 Done Done Sent false
7788 2881 Acked Write 4 Done Done Sent false
7789 2882 Acked Flush 0 Done Done Sent false
7790 2883 Acked Write 10 Done Done Sent false
7791 2884 Acked Read 3 Done Done Sent false
7792 2885 Acked Flush 0 Done Done Sent false
7793 2886 Acked Flush 0 Done Done Sent false
7794 2887 Acked Write 5 Done Done Sent false
7795 2888 Acked Read 4 Done Done Sent false
7796 2889 Acked Write 7 Done Done Sent false
7797 2890 Acked Read 4 Done Done Sent false
7798 2891 Acked Flush 0 Done Done Sent false
7799 2892 Acked Write 7 Done Done Sent false
7800 2893 Acked Write 1 Done Done Sent false
7801 2894 Acked Read 3 Done Done Sent false
7802 2895 Acked Read 3 Done Done Sent false
7803 2896 Acked Read 5 Done Done Sent false
7804 2897 Acked Read 9 Done Done Sent false
7805 2898 Acked Read 5 Done Done Sent false
7806 2899 Acked Write 6 Done Done Sent false
7807 2900 Acked Read 8 Done Done Sent false
7808 2901 Acked Write 7 Done Done Sent false
7809 2902 Acked Read 3 Done Done Sent false
7810 2903 Acked Write 3 Done Done Sent false
7811 2904 Acked Write 7 Done Done Sent false
7812 2905 Acked Read 1 Done Done Sent false
7813 2906 Acked Read 5 Done Done Sent false
7814 2907 Acked Flush 0 Done Done Sent false
7815 2908 Acked Read 3 Done Done Sent false
7816 2909 Acked Read 1 Done Done Sent false
7817 2910 Acked Write 10 Done Done Sent false
7818 2911 Acked Write 2 Done Done Sent false
7819 2912 Acked Read 7 Done Done Sent false
7820 2913 Acked Write 10 Done Done Sent false
7821 2914 Acked Read 4 Done Done Sent false
7822 2915 Acked Read 4 Done Done Sent false
7823 2916 Acked Write 5 Done Done Sent false
7824 2917 Acked Write 1 Done Done Sent false
7825 2918 Acked Flush 0 Done Done Sent false
7826 2919 Acked Read 2 Done Done Sent false
7827 2920 Acked Write 5 Done Done Sent false
7828 2921 Acked Read 10 Done Done Sent false
7829 2922 Acked Flush 0 Done Done Sent false
7830 2923 Acked Read 7 Done Done Sent false
7831 2924 Acked Read 6 Done Done Sent false
7832 2925 Acked Write 7 Done Done Sent false
7833 2926 Acked Read 2 Done Done Sent false
7834 2927 Acked Read 9 Done Done Sent false
7835 2928 Acked Read 4 Done Done Sent false
7836 2929 Acked Flush 0 Done Done Sent false
7837 2930 Acked Read 1 Done Done Sent false
7838 2931 Acked Read 9 Done Done Sent false
7839 2932 Acked Write 2 Done Done Sent false
7840 2933 Acked Write 2 Done Done Sent false
7841 2934 Acked Flush 0 Done Done Sent false
7842 2935 Acked Read 2 Done Done Sent false
7843 2936 Acked Write 2 Done Done Sent false
7844 2937 Acked Write 2 Done Done Sent false
7845 2938 Acked Write 9 Done Done Sent false
7846 2939 Acked Read 6 Done Done Sent false
7847 2940 Acked Flush 0 Done Done Sent false
7848 2941 Acked Write 6 Done Done Sent false
7849 2942 Acked Read 2 Done Done Sent false
7850 2943 Acked Flush 0 Done Done Sent false
7851 2944 Acked Write 6 Done Done Sent false
7852 2945 Acked Read 2 Done Done Sent false
7853 2946 Acked Read 2 Done Done Sent false
7854 2947 Acked Write 6 Done Done Sent false
7855 2948 Acked Write 9 Done Done Sent false
7856 2949 Acked Write 7 Done Done Sent false
7857 2950 Acked Read 4 Done Done Sent false
7858 2951 Acked Read 2 Done Done Sent false
7859 2952 Acked Write 4 Done Done Sent false
7860 2953 Acked Write 9 Done Done Sent false
7861 2954 Acked Read 4 Done Done Sent false
7862 2955 Acked Write 8 Done Done Sent false
7863 2956 Acked Read 2 Done Done Sent false
7864 2957 Acked Read 8 Done Done Sent false
7865 2958 Acked Read 8 Done Done Sent false
7866 2959 Acked Read 9 Done Done Sent false
7867 2960 Acked Read 6 Done Done Sent false
7868 2961 Acked Write 9 Done Done Sent false
7869 2962 Acked Read 1 Done Done Sent false
7870 2963 Acked Write 8 Done Done Sent false
7871 2964 Acked Write 5 Done Done Sent false
7872 2965 Acked Read 4 Done Done Sent false
7873 2966 Acked Write 9 Done Done Sent false
7874 2967 Acked Read 10 Done Done Sent false
7875 2968 Acked Read 9 Done Done Sent false
7876 2969 Acked Read 10 Done Done Sent false
7877 2970 Acked Read 5 Done Done Sent false
7878 2971 Acked Write 2 Done Done Sent false
7879 2972 Acked Flush 0 Done Done Sent false
7880 2973 Acked Read 6 Done Done Sent false
7881 2974 Acked Flush 0 Done Done Sent false
7882 2975 Acked Write 1 Done Done Sent false
7883 2976 Acked Read 1 Done Done Sent false
7884 2977 Acked Write 9 Done Done Sent false
7885 2978 Acked Read 2 Done Done Sent false
7886 2979 Acked Read 10 Done Done Sent false
7887 2980 Acked Read 3 Done Done Sent false
7888 2981 Acked Write 10 Done Done Sent false
7889 2982 Acked Read 3 Done Done Sent false
7890 2983 Acked Flush 0 Done Done Sent false
7891 2984 Acked Read 1 Done Done Sent false
7892 2985 Acked Read 10 Done Done Sent false
7893 2986 Acked Write 3 Done Done Sent false
7894 2987 Acked Read 10 Done Done Sent false
7895 2988 Acked Read 2 Done Done Sent false
7896 2989 Acked Read 8 Done Done Sent false
7897 2990 Acked Read 3 Done Done Sent false
7898 2991 Acked Write 3 Done Done Sent false
7899 2992 Acked Read 5 Done Done Sent false
7900 2993 Acked Write 8 Done Done Sent false
7901 2994 Acked Flush 0 Done Done Sent false
7902 2995 Acked Write 6 Done Done Sent false
7903 2996 Acked Write 5 Done Done Sent false
7904 2997 Acked Read 5 Done Done Sent false
7905 2998 Acked Write 10 Done Done Sent false
7906 2999 Acked Flush 0 Done Done Sent false
7907 3000 Acked Read 9 Done Done Sent false
7908 3001 Acked Read 6 Done Done Sent false
7909 3002 Acked Write 9 Done Done Sent false
7910 3003 Acked Read 5 Done Done Sent false
7911 3004 Acked Write 6 Done Done Sent false
7912 3005 Acked Read 5 Done Done Sent false
7913 3006 Acked Read 8 Done Done Sent false
7914 3007 Acked Read 3 Done Done Sent false
7915 3008 Acked Read 9 Done Done Sent false
7916 3009 Acked Write 9 Done Done Sent false
7917 3010 Acked Read 2 Done Done Sent false
7918 3011 Acked Read 3 Done Done Sent false
7919 3012 Acked Read 3 Done Done Sent false
7920 3013 Acked Write 5 Done Done Sent false
7921 3014 Acked Read 3 Done Done Sent false
7922 3015 Acked Write 4 Done Done Sent false
7923 3016 Acked Write 3 Done Done Sent false
7924 3017 Acked Read 7 Done Done Sent false
7925 3018 Acked Write 10 Done Done Sent false
7926 3019 Acked Read 7 Done Done Sent false
7927 3020 Acked Read 5 Done Done Sent false
7928 3021 Acked Write 8 Done Done Sent false
7929 3022 Acked Read 8 Done Done Sent false
7930 3023 Acked Write 2 Done Done Sent false
7931 3024 Acked Write 5 Done Done Sent false
7932 3025 Acked Read 9 Done Done Sent false
7933 3026 Acked Write 9 Done Done Sent false
7934 3027 Acked Write 7 Done Done Sent false
7935 3028 Acked Write 8 Done Done Sent false
7936 3029 Acked Write 5 Done Done Sent false
7937 3030 Acked Read 9 Done Done Sent false
7938 3031 Acked Write 4 Done Done Sent false
7939 3032 Acked Flush 0 Done Done Sent false
7940 3033 Acked Flush 0 Done Done Sent false
7941 3034 Acked Read 8 Done Done Sent false
7942 3035 Acked Write 5 Done Done Sent false
7943 3036 Acked Flush 0 Done Done Sent false
7944 3037 Acked Write 8 Done Done Sent false
7945 3038 Acked Read 1 Done Done Sent false
7946 3039 Acked Read 3 Done Done Sent false
7947 3040 Acked Read 2 Done Done Sent false
7948 3041 Acked Read 2 Done Done Sent false
7949 3042 Acked Read 5 Done Done Sent false
7950 3043 Acked Read 10 Done Done Sent false
7951 3044 Acked Flush 0 Done Done Sent false
7952 3045 Acked Write 5 Done Done Sent false
7953 3046 Acked Write 3 Done Done Sent false
7954 3047 Acked Write 2 Done Done Sent false
7955 3048 Acked Read 4 Done Done Sent false
7956 3049 Acked Read 6 Done Done Sent false
7957 3050 Acked Write 8 Done Done Sent false
7958 3051 Acked Read 4 Done Done Sent false
7959 3052 Acked Read 1 Done Done Sent false
7960 3053 Acked Flush 0 Done Done Sent false
7961 3054 Acked Read 7 Done Done Sent false
7962 3055 Acked Write 6 Done Done Sent false
7963 3056 Acked Read 8 Done Done Sent false
7964 3057 Acked Write 4 Done Done Sent false
7965 3058 Acked Flush 0 Done Done Sent false
7966 3059 Acked Write 8 Done Done Sent false
7967 3060 Acked Write 1 Done Done Sent false
7968 3061 Acked Write 2 Done Done Sent false
7969 3062 Acked Flush 0 Done Done Sent false
7970 3063 Acked Read 8 Done Done Sent false
7971 3064 Acked Read 9 Done Done Sent false
7972 3065 Acked Write 10 Done Done Sent false
7973 3066 Acked Flush 0 Done Done Sent false
7974 3067 Acked Write 8 Done Done Sent false
7975 3068 Acked Write 5 Done Done Sent false
7976 3069 Acked Read 2 Done Done Sent false
7977 3070 Acked Write 1 Done Done Sent false
7978 3071 Acked Flush 0 Done Done Sent false
7979 3072 Acked Write 7 Done Done Sent false
7980 3073 Acked Write 5 Done Done Sent false
7981 3074 Acked Read 6 Done Done Sent false
7982 3075 Acked Read 4 Done Done Sent false
7983 3076 Acked Flush 0 Done Done Sent false
7984 3077 Acked Write 6 Done Done Sent false
7985 3078 Acked Read 8 Done Done Sent false
7986 3079 Acked Read 4 Done Done Sent false
7987 3080 Acked Write 5 Done Done Sent false
7988 3081 Acked Flush 0 Done Done Sent false
7989 3082 Acked Write 4 Done Done Sent false
7990 3083 Acked Write 5 Done Done Sent false
7991 3084 Acked Write 6 Done Done Sent false
7992 3085 Acked Read 4 Done Done Sent false
7993 3086 Acked Write 4 Done Done Sent false
7994 3087 Acked Write 9 Done Done Sent false
7995 3088 Acked Read 3 Done Done Sent false
7996 3089 Acked Write 10 Done Done Sent false
7997 3090 Acked Read 1 Done Done Sent false
7998 3091 Acked Read 5 Done Done Sent false
7999 3092 Acked Read 9 Done Done Sent false
8000 3093 Acked Read 2 Done Done Sent false
8001 3094 Acked Write 2 Done Done Sent false
8002 3095 Acked Read 3 Done Done Sent false
8003 3096 Acked Read 4 Done Done Sent false
8004 3097 Acked Write 4 Done Done Sent false
8005 3098 Acked Write 2 Done Done Sent false
8006 3099 Acked Write 10 Done Done Sent false
8007 3100 Acked Read 6 Done Done Sent false
8008 3101 Acked Write 3 Done Done Sent false
8009 3102 Acked Read 2 Done Done Sent false
8010 3103 Acked Read 5 Done Done Sent false
8011 3104 Acked Write 4 Done Done Sent false
8012 3105 Acked Read 8 Done Done Sent false
8013 3106 Acked Read 2 Done Done Sent false
8014 3107 Acked Write 6 Done Done Sent false
8015 3108 Acked Read 6 Done Done Sent false
8016 3109 Acked Write 10 Done Done Sent false
8017 3110 Acked Flush 0 Done Done Sent false
8018 3111 Acked Read 2 Done Done Sent false
8019 3112 Acked Read 9 Done Done Sent false
8020 STATES DS:0 DS:1 DS:2 TOTAL
8021 Sent 0 0 301 301
8022 Done 301 301 0 602
8023 Skipped 0 0 0 0
8024 Error 0 0 0 0
8025 Last Flush: Some(JobId(3110)) Some(JobId(3110)) Some(JobId(2811))
8026 Downstairs last five completed: 2811 2810 2809 2808 2807
8027 Upstairs last five completed: 3112 3111 3110 3109 3108
8028 CLIENT: Up:0 ds:301 act:2
80292025-03-17T22:36:15.465ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8030 [8830] got start action
8031 [8830] I am going to restart
8032 Starting downstairs at port 8830
8033 Make output file at "/tmp/dsc/downstairs-8830.txt"
8034 [8830][2] reports Starting
8035 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
8036 [8830][2] reports Running
80372025-03-17T22:36:16.467ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
80382025-03-17T22:36:17.469ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8039 ----------------------------------------------------------------
8040 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8041 States: Active Active Offline
8042 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8043 2812 Acked Read 9 Done Done Sent false
8044 2813 Acked Write 4 Done Done Sent false
8045 2814 Acked Flush 0 Done Done Sent false
8046 2815 Acked Write 8 Done Done Sent false
8047 2816 Acked Write 1 Done Done Sent false
8048 2817 Acked Write 3 Done Done Sent false
8049 2818 Acked Write 6 Done Done Sent false
8050 2819 Acked Read 8 Done Done Sent false
8051 2820 Acked Write 5 Done Done Sent false
8052 2821 Acked Read 8 Done Done Sent false
8053 2822 Acked Read 6 Done Done Sent false
8054 2823 Acked Write 10 Done Done Sent false
8055 2824 Acked Write 9 Done Done Sent false
8056 2825 Acked Write 7 Done Done Sent false
8057 2826 Acked Write 9 Done Done Sent false
8058 2827 Acked Write 2 Done Done Sent false
8059 2828 Acked Read 1 Done Done Sent false
8060 2829 Acked Read 9 Done Done Sent false
8061 2830 Acked Write 6 Done Done Sent false
8062 2831 Acked Flush 0 Done Done Sent false
8063 2832 Acked Read 3 Done Done Sent false
8064 2833 Acked Write 5 Done Done Sent false
8065 2834 Acked Write 6 Done Done Sent false
8066 2835 Acked Write 9 Done Done Sent false
8067 2836 Acked Read 9 Done Done Sent false
8068 2837 Acked Write 4 Done Done Sent false
8069 2838 Acked Write 8 Done Done Sent false
8070 2839 Acked Write 1 Done Done Sent false
8071 2840 Acked Write 10 Done Done Sent false
8072 2841 Acked Write 7 Done Done Sent false
8073 2842 Acked Read 7 Done Done Sent false
8074 2843 Acked Write 2 Done Done Sent false
8075 2844 Acked Flush 0 Done Done Sent false
8076 2845 Acked Flush 0 Done Done Sent false
8077 2846 Acked Read 2 Done Done Sent false
8078 2847 Acked Write 3 Done Done Sent false
8079 2848 Acked Read 8 Done Done Sent false
8080 2849 Acked Read 4 Done Done Sent false
8081 2850 Acked Read 8 Done Done Sent false
8082 2851 Acked Read 6 Done Done Sent false
8083 2852 Acked Write 8 Done Done Sent false
8084 2853 Acked Read 7 Done Done Sent false
8085 2854 Acked Flush 0 Done Done Sent false
8086 2855 Acked Read 3 Done Done Sent false
8087 2856 Acked Write 7 Done Done Sent false
8088 2857 Acked Read 8 Done Done Sent false
8089 2858 Acked Read 3 Done Done Sent false
8090 2859 Acked Read 1 Done Done Sent false
8091 2860 Acked Write 9 Done Done Sent false
8092 2861 Acked Write 9 Done Done Sent false
8093 2862 Acked Flush 0 Done Done Sent false
8094 2863 Acked Read 1 Done Done Sent false
8095 2864 Acked Read 6 Done Done Sent false
8096 2865 Acked Read 3 Done Done Sent false
8097 2866 Acked Write 1 Done Done Sent false
8098 2867 Acked Flush 0 Done Done Sent false
8099 2868 Acked Flush 0 Done Done Sent false
8100 2869 Acked Write 5 Done Done Sent false
8101 2870 Acked Read 2 Done Done Sent false
8102 2871 Acked Read 1 Done Done Sent false
8103 2872 Acked Write 6 Done Done Sent false
8104 2873 Acked Write 7 Done Done Sent false
8105 2874 Acked Write 7 Done Done Sent false
8106 2875 Acked Write 5 Done Done Sent false
8107 2876 Acked Write 5 Done Done Sent false
8108 2877 Acked Read 5 Done Done Sent false
8109 2878 Acked Read 8 Done Done Sent false
8110 2879 Acked Write 1 Done Done Sent false
8111 2880 Acked Flush 0 Done Done Sent false
8112 2881 Acked Write 4 Done Done Sent false
8113 2882 Acked Flush 0 Done Done Sent false
8114 2883 Acked Write 10 Done Done Sent false
8115 2884 Acked Read 3 Done Done Sent false
8116 2885 Acked Flush 0 Done Done Sent false
8117 2886 Acked Flush 0 Done Done Sent false
8118 2887 Acked Write 5 Done Done Sent false
8119 2888 Acked Read 4 Done Done Sent false
8120 2889 Acked Write 7 Done Done Sent false
8121 2890 Acked Read 4 Done Done Sent false
8122 2891 Acked Flush 0 Done Done Sent false
8123 2892 Acked Write 7 Done Done Sent false
8124 2893 Acked Write 1 Done Done Sent false
8125 2894 Acked Read 3 Done Done Sent false
8126 2895 Acked Read 3 Done Done Sent false
8127 2896 Acked Read 5 Done Done Sent false
8128 2897 Acked Read 9 Done Done Sent false
8129 2898 Acked Read 5 Done Done Sent false
8130 2899 Acked Write 6 Done Done Sent false
8131 2900 Acked Read 8 Done Done Sent false
8132 2901 Acked Write 7 Done Done Sent false
8133 2902 Acked Read 3 Done Done Sent false
8134 2903 Acked Write 3 Done Done Sent false
8135 2904 Acked Write 7 Done Done Sent false
8136 2905 Acked Read 1 Done Done Sent false
8137 2906 Acked Read 5 Done Done Sent false
8138 2907 Acked Flush 0 Done Done Sent false
8139 2908 Acked Read 3 Done Done Sent false
8140 2909 Acked Read 1 Done Done Sent false
8141 2910 Acked Write 10 Done Done Sent false
8142 2911 Acked Write 2 Done Done Sent false
8143 2912 Acked Read 7 Done Done Sent false
8144 2913 Acked Write 10 Done Done Sent false
8145 2914 Acked Read 4 Done Done Sent false
8146 2915 Acked Read 4 Done Done Sent false
8147 2916 Acked Write 5 Done Done Sent false
8148 2917 Acked Write 1 Done Done Sent false
8149 2918 Acked Flush 0 Done Done Sent false
8150 2919 Acked Read 2 Done Done Sent false
8151 2920 Acked Write 5 Done Done Sent false
8152 2921 Acked Read 10 Done Done Sent false
8153 2922 Acked Flush 0 Done Done Sent false
8154 2923 Acked Read 7 Done Done Sent false
8155 2924 Acked Read 6 Done Done Sent false
8156 2925 Acked Write 7 Done Done Sent false
8157 2926 Acked Read 2 Done Done Sent false
8158 2927 Acked Read 9 Done Done Sent false
8159 2928 Acked Read 4 Done Done Sent false
8160 2929 Acked Flush 0 Done Done Sent false
8161 2930 Acked Read 1 Done Done Sent false
8162 2931 Acked Read 9 Done Done Sent false
8163 2932 Acked Write 2 Done Done Sent false
8164 2933 Acked Write 2 Done Done Sent false
8165 2934 Acked Flush 0 Done Done Sent false
8166 2935 Acked Read 2 Done Done Sent false
8167 2936 Acked Write 2 Done Done Sent false
8168 2937 Acked Write 2 Done Done Sent false
8169 2938 Acked Write 9 Done Done Sent false
8170 2939 Acked Read 6 Done Done Sent false
8171 2940 Acked Flush 0 Done Done Sent false
8172 2941 Acked Write 6 Done Done Sent false
8173 2942 Acked Read 2 Done Done Sent false
8174 2943 Acked Flush 0 Done Done Sent false
8175 2944 Acked Write 6 Done Done Sent false
8176 2945 Acked Read 2 Done Done Sent false
8177 2946 Acked Read 2 Done Done Sent false
8178 2947 Acked Write 6 Done Done Sent false
8179 2948 Acked Write 9 Done Done Sent false
8180 2949 Acked Write 7 Done Done Sent false
8181 2950 Acked Read 4 Done Done Sent false
8182 2951 Acked Read 2 Done Done Sent false
8183 2952 Acked Write 4 Done Done Sent false
8184 2953 Acked Write 9 Done Done Sent false
8185 2954 Acked Read 4 Done Done Sent false
8186 2955 Acked Write 8 Done Done Sent false
8187 2956 Acked Read 2 Done Done Sent false
8188 2957 Acked Read 8 Done Done Sent false
8189 2958 Acked Read 8 Done Done Sent false
8190 2959 Acked Read 9 Done Done Sent false
8191 2960 Acked Read 6 Done Done Sent false
8192 2961 Acked Write 9 Done Done Sent false
8193 2962 Acked Read 1 Done Done Sent false
8194 2963 Acked Write 8 Done Done Sent false
8195 2964 Acked Write 5 Done Done Sent false
8196 2965 Acked Read 4 Done Done Sent false
8197 2966 Acked Write 9 Done Done Sent false
8198 2967 Acked Read 10 Done Done Sent false
8199 2968 Acked Read 9 Done Done Sent false
8200 2969 Acked Read 10 Done Done Sent false
8201 2970 Acked Read 5 Done Done Sent false
8202 2971 Acked Write 2 Done Done Sent false
8203 2972 Acked Flush 0 Done Done Sent false
8204 2973 Acked Read 6 Done Done Sent false
8205 2974 Acked Flush 0 Done Done Sent false
8206 2975 Acked Write 1 Done Done Sent false
8207 2976 Acked Read 1 Done Done Sent false
8208 2977 Acked Write 9 Done Done Sent false
8209 2978 Acked Read 2 Done Done Sent false
8210 2979 Acked Read 10 Done Done Sent false
8211 2980 Acked Read 3 Done Done Sent false
8212 2981 Acked Write 10 Done Done Sent false
8213 2982 Acked Read 3 Done Done Sent false
8214 2983 Acked Flush 0 Done Done Sent false
8215 2984 Acked Read 1 Done Done Sent false
8216 2985 Acked Read 10 Done Done Sent false
8217 2986 Acked Write 3 Done Done Sent false
8218 2987 Acked Read 10 Done Done Sent false
8219 2988 Acked Read 2 Done Done Sent false
8220 2989 Acked Read 8 Done Done Sent false
8221 2990 Acked Read 3 Done Done Sent false
8222 2991 Acked Write 3 Done Done Sent false
8223 2992 Acked Read 5 Done Done Sent false
8224 2993 Acked Write 8 Done Done Sent false
8225 2994 Acked Flush 0 Done Done Sent false
8226 2995 Acked Write 6 Done Done Sent false
8227 2996 Acked Write 5 Done Done Sent false
8228 2997 Acked Read 5 Done Done Sent false
8229 2998 Acked Write 10 Done Done Sent false
8230 2999 Acked Flush 0 Done Done Sent false
8231 3000 Acked Read 9 Done Done Sent false
8232 3001 Acked Read 6 Done Done Sent false
8233 3002 Acked Write 9 Done Done Sent false
8234 3003 Acked Read 5 Done Done Sent false
8235 3004 Acked Write 6 Done Done Sent false
8236 3005 Acked Read 5 Done Done Sent false
8237 3006 Acked Read 8 Done Done Sent false
8238 3007 Acked Read 3 Done Done Sent false
8239 3008 Acked Read 9 Done Done Sent false
8240 3009 Acked Write 9 Done Done Sent false
8241 3010 Acked Read 2 Done Done Sent false
8242 3011 Acked Read 3 Done Done Sent false
8243 3012 Acked Read 3 Done Done Sent false
8244 3013 Acked Write 5 Done Done Sent false
8245 3014 Acked Read 3 Done Done Sent false
8246 3015 Acked Write 4 Done Done Sent false
8247 3016 Acked Write 3 Done Done Sent false
8248 3017 Acked Read 7 Done Done Sent false
8249 3018 Acked Write 10 Done Done Sent false
8250 3019 Acked Read 7 Done Done Sent false
8251 3020 Acked Read 5 Done Done Sent false
8252 3021 Acked Write 8 Done Done Sent false
8253 3022 Acked Read 8 Done Done Sent false
8254 3023 Acked Write 2 Done Done Sent false
8255 3024 Acked Write 5 Done Done Sent false
8256 3025 Acked Read 9 Done Done Sent false
8257 3026 Acked Write 9 Done Done Sent false
8258 3027 Acked Write 7 Done Done Sent false
8259 3028 Acked Write 8 Done Done Sent false
8260 3029 Acked Write 5 Done Done Sent false
8261 3030 Acked Read 9 Done Done Sent false
8262 3031 Acked Write 4 Done Done Sent false
8263 3032 Acked Flush 0 Done Done Sent false
8264 3033 Acked Flush 0 Done Done Sent false
8265 3034 Acked Read 8 Done Done Sent false
8266 3035 Acked Write 5 Done Done Sent false
8267 3036 Acked Flush 0 Done Done Sent false
8268 3037 Acked Write 8 Done Done Sent false
8269 3038 Acked Read 1 Done Done Sent false
8270 3039 Acked Read 3 Done Done Sent false
8271 3040 Acked Read 2 Done Done Sent false
8272 3041 Acked Read 2 Done Done Sent false
8273 3042 Acked Read 5 Done Done Sent false
8274 3043 Acked Read 10 Done Done Sent false
8275 3044 Acked Flush 0 Done Done Sent false
8276 3045 Acked Write 5 Done Done Sent false
8277 3046 Acked Write 3 Done Done Sent false
8278 3047 Acked Write 2 Done Done Sent false
8279 3048 Acked Read 4 Done Done Sent false
8280 3049 Acked Read 6 Done Done Sent false
8281 3050 Acked Write 8 Done Done Sent false
8282 3051 Acked Read 4 Done Done Sent false
8283 3052 Acked Read 1 Done Done Sent false
8284 3053 Acked Flush 0 Done Done Sent false
8285 3054 Acked Read 7 Done Done Sent false
8286 3055 Acked Write 6 Done Done Sent false
8287 3056 Acked Read 8 Done Done Sent false
8288 3057 Acked Write 4 Done Done Sent false
8289 3058 Acked Flush 0 Done Done Sent false
8290 3059 Acked Write 8 Done Done Sent false
8291 3060 Acked Write 1 Done Done Sent false
8292 3061 Acked Write 2 Done Done Sent false
8293 3062 Acked Flush 0 Done Done Sent false
8294 3063 Acked Read 8 Done Done Sent false
8295 3064 Acked Read 9 Done Done Sent false
8296 3065 Acked Write 10 Done Done Sent false
8297 3066 Acked Flush 0 Done Done Sent false
8298 3067 Acked Write 8 Done Done Sent false
8299 3068 Acked Write 5 Done Done Sent false
8300 3069 Acked Read 2 Done Done Sent false
8301 3070 Acked Write 1 Done Done Sent false
8302 3071 Acked Flush 0 Done Done Sent false
8303 3072 Acked Write 7 Done Done Sent false
8304 3073 Acked Write 5 Done Done Sent false
8305 3074 Acked Read 6 Done Done Sent false
8306 3075 Acked Read 4 Done Done Sent false
8307 3076 Acked Flush 0 Done Done Sent false
8308 3077 Acked Write 6 Done Done Sent false
8309 3078 Acked Read 8 Done Done Sent false
8310 3079 Acked Read 4 Done Done Sent false
8311 3080 Acked Write 5 Done Done Sent false
8312 3081 Acked Flush 0 Done Done Sent false
8313 3082 Acked Write 4 Done Done Sent false
8314 3083 Acked Write 5 Done Done Sent false
8315 3084 Acked Write 6 Done Done Sent false
8316 3085 Acked Read 4 Done Done Sent false
8317 3086 Acked Write 4 Done Done Sent false
8318 3087 Acked Write 9 Done Done Sent false
8319 3088 Acked Read 3 Done Done Sent false
8320 3089 Acked Write 10 Done Done Sent false
8321 3090 Acked Read 1 Done Done Sent false
8322 3091 Acked Read 5 Done Done Sent false
8323 3092 Acked Read 9 Done Done Sent false
8324 3093 Acked Read 2 Done Done Sent false
8325 3094 Acked Write 2 Done Done Sent false
8326 3095 Acked Read 3 Done Done Sent false
8327 3096 Acked Read 4 Done Done Sent false
8328 3097 Acked Write 4 Done Done Sent false
8329 3098 Acked Write 2 Done Done Sent false
8330 3099 Acked Write 10 Done Done Sent false
8331 3100 Acked Read 6 Done Done Sent false
8332 3101 Acked Write 3 Done Done Sent false
8333 3102 Acked Read 2 Done Done Sent false
8334 3103 Acked Read 5 Done Done Sent false
8335 3104 Acked Write 4 Done Done Sent false
8336 3105 Acked Read 8 Done Done Sent false
8337 3106 Acked Read 2 Done Done Sent false
8338 3107 Acked Write 6 Done Done Sent false
8339 3108 Acked Read 6 Done Done Sent false
8340 3109 Acked Write 10 Done Done Sent false
8341 3110 Acked Flush 0 Done Done Sent false
8342 3111 Acked Read 2 Done Done Sent false
8343 3112 Acked Read 9 Done Done Sent false
8344 STATES DS:0 DS:1 DS:2 TOTAL
8345 Sent 0 0 301 301
8346 Done 301 301 0 602
8347 Skipped 0 0 0 0
8348 Error 0 0 0 0
8349 Last Flush: Some(JobId(3110)) Some(JobId(3110)) Some(JobId(2811))
8350 Downstairs last five completed: 2811 2810 2809 2808 2807
8351 Upstairs last five completed: 3112 3111 3110 3109 3108
8352 CLIENT: Up:0 ds:301 act:2
83532025-03-17T22:36:19.472ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83542025-03-17T22:36:20.360ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83552025-03-17T22:36:20.360ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39104) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83562025-03-17T22:36:20.361ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83572025-03-17T22:36:20.362ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83582025-03-17T22:36:20.362ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83592025-03-17T22:36:20.362ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83602025-03-17T22:36:20.362ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83612025-03-17T22:36:20.362ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83622025-03-17T22:36:20.362ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8363 ----------------------------------------------------------------
8364 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8365 Downstairs last five completed: 3113 3112 3111 3110 3109
8366 Upstairs last five completed: 3113 3112 3111 3110 3109
8367 CLIENT: Up:0 ds:0 act:3
8368 Replay: All jobs finished
8369 Main task has work to do, go find it
8370 got dsc Stop(2)
8371 stop 2
8372 Mar 17 22:36:22.604[8830] Got stop action so:false kr:false
8373 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e4e3c7fd-3308-42ac-acb0-a39a656ccd41, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
83742025-03-17T22:36:22.605ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83752025-03-17T22:36:22.605ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
83762025-03-17T22:36:22.605ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8377 Mar 17 22:36:22.605 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/2, method: GET, req_id: e9fb6e05-67cc-4361-bc92-ecefefd24ae6, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
83782025-03-17T22:36:22.605ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
8379 [8830] Exited with: ExitStatus(unix_wait_status(9))
8380 [8830][2] reports Exit
8381 Mar 17 22:36:26.607 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: e0c19b1d-2375-47ec-85d1-c3d78307fbf1, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
8382 001/300 Write block 1612 len 32768 data: 2 2 2 2 3 3 4 3
8383 002/300 Read block 468 len 40960
8384 003/300 Read block 3088 len 32768
8385 004/300 Read block 1645 len 8192
8386 005/300 Write block 221 len 12288 data: 2 2 3
8387 006/300 Write block 3022 len 4096 data: 2
8388 007/300 Write block 764 len 32768 data: 2 2 2 2 2 2 2 3
8389 008/300 Flush
8390 009/300 Read block 1706 len 32768
8391 010/300 Read block 3116 len 40960
8392 011/300 Flush
8393 012/300 Flush
8394 013/300 Flush
8395 014/300 Write block 3301 len 16384 data: 3 3 3 4
8396 015/300 Write block 2733 len 40960 data: 4 4 4 2 2 2 3 3 3 3
8397 016/300 Read block 1463 len 20480
8398 017/300 Write block 2748 len 32768 data: 2 2 2 2 2 2 2 2
8399 018/300 Flush
8400 019/300 Read block 3682 len 28672
8401 020/300 Flush
8402 021/300 Read block 498 len 20480
8403 022/300 Read block 989 len 4096
8404 023/300 Write block 1999 len 24576 data: 3 3 3 3 3 3
8405 024/300 Write block 521 len 20480 data: 3 4 4 4 4
8406 025/300 Read block 2414 len 32768
8407 026/300 Flush
8408 027/300 Write block 779 len 28672 data: 2 2 2 3 3 3 2
8409 028/300 Read block 1355 len 24576
8410 029/300 Write block 965 len 8192 data: 3 3
8411 030/300 Read block 1111 len 20480
8412 031/300 Read block 2705 len 12288
8413 032/300 Write block 699 len 40960 data: 3 3 3 3 3 3 3 3 2 2
8414 033/300 Read block 4244 len 32768
8415 034/300 Read block 3251 len 16384
8416 035/300 Read block 742 len 4096
8417 036/300 Write block 1926 len 28672 data: 4 4 2 2 2 2 2
8418 037/300 Read block 1331 len 24576
8419 038/300 Flush
8420 039/300 Flush
8421 040/300 Read block 2454 len 8192
8422 041/300 Read block 541 len 12288
8423 042/300 Write block 4292 len 28672 data: 3 4 4 4 3 3 3
8424 043/300 Read block 2308 len 16384
8425 044/300 Write block 2758 len 16384 data: 3 3 3 3
8426 045/300 Read block 551 len 20480
8427 046/300 Read block 3799 len 32768
8428 047/300 Read block 3520 len 8192
8429 048/300 Read block 1006 len 28672
8430 049/300 Read block 526 len 36864
8431 050/300 Write block 325 len 8192 data: 2 2
8432 051/300 Read block 198 len 36864
8433 052/300 Read block 4471 len 4096
8434 053/300 Write block 4442 len 36864 data: 2 2 2 2 2 2 2 2 2
8435 054/300 Write block 3702 len 8192 data: 2 2
8436 055/300 Read block 4781 len 28672
8437 056/300 Write block 1991 len 8192 data: 2 2
8438 057/300 Read block 41 len 36864
8439 058/300 Write block 4424 len 12288 data: 4 5 5
8440 059/300 Write block 2942 len 4096 data: 2
8441 060/300 Flush
8442 061/300 Write block 4513 len 36864 data: 2 3 3 3 3 4 4 4 4
8443 062/300 Write block 103 len 40960 data: 4 4 4 4 2 4 4 4 4 3
8444 063/300 Write block 3241 len 20480 data: 3 2 2 2 3
8445 064/300 Write block 1442 len 16384 data: 3 4 3 3
8446 065/300 Read block 2898 len 32768
8447 066/300 Read block 3851 len 40960
8448 067/300 Write block 249 len 12288 data: 2 3 3
8449 068/300 Write block 157 len 8192 data: 2 2
8450 069/300 Write block 4226 len 28672 data: 3 3 3 3 3 4 4
8451 070/300 Read block 2232 len 16384
8452 071/300 Read block 2794 len 40960
8453 072/300 Read block 2692 len 40960
8454 073/300 Read block 1403 len 12288
8455 074/300 Read block 3290 len 20480
8456 075/300 Write block 353 len 8192 data: 2 2
8457 076/300 Read block 3003 len 32768
8458 077/300 Flush
8459 078/300 Write block 140 len 8192 data: 2 2
8460 079/300 Write block 2979 len 12288 data: 3 3 3
8461 080/300 Write block 4822 len 4096 data: 3
8462 081/300 Read block 1627 len 40960
8463 082/300 Read block 1752 len 32768
8464 083/300 Write block 3290 len 36864 data: 3 3 3 3 3 3 2 2 2
8465 084/300 Write block 1394 len 32768 data: 4 4 5 5 4 4 4 4
8466 085/300 Read block 3191 len 36864
8467 086/300 Read block 3866 len 28672
8468 087/300 Write block 229 len 28672 data: 3 4 4 4 3 3 3
8469 088/300 Read block 3440 len 12288
8470 089/300 Write block 4547 len 20480 data: 2 2 3 4 3
8471 090/300 Write block 2974 len 28672 data: 2 2 2 2 3 4 4
8472 091/300 Read block 449 len 24576
8473 092/300 Read block 3444 len 8192
8474 093/300 Read block 1647 len 28672
8475 094/300 Write block 380 len 40960 data: 2 2 2 3 3 3 2 2 2 2
8476 095/300 Write block 2202 len 28672 data: 3 3 3 4 3 2 2
8477 096/300 Flush
8478 097/300 Write block 819 len 40960 data: 4 3 3 2 2 2 2 2 2 2
8479 098/300 Write block 2069 len 4096 data: 3
8480 099/300 Flush
8481 100/300 Flush
8482 101/300 Read block 1424 len 12288
8483 102/300 Write block 4731 len 8192 data: 3 3
8484 103/300 Write block 4575 len 8192 data: 2 2
8485 104/300 Write block 25 len 36864 data: 2 2 2 2 2 2 2 3 3
8486 105/300 Read block 3923 len 20480
8487 106/300 Read block 1566 len 32768
8488 107/300 Write block 903 len 24576 data: 3 3 3 3 3 3
8489 108/300 Read block 3701 len 40960
8490 109/300 Read block 1864 len 28672
8491 110/300 Read block 2398 len 28672
8492 111/300 Write block 3792 len 16384 data: 2 2 2 2
8493 112/300 Flush
8494 113/300 Read block 3301 len 32768
8495 114/300 Write block 2477 len 16384 data: 3 3 3 3
8496 115/300 Flush
8497 116/300 Read block 591 len 40960
8498 117/300 Read block 4408 len 16384
8499 118/300 Write block 4742 len 28672 data: 2 3 3 3 3 3 3
8500 119/300 Write block 3453 len 12288 data: 3 2 2
8501 120/300 Read block 814 len 32768
8502 121/300 Write block 4290 len 40960 data: 2 3 4 5 5 5 4 4 4 3
8503 122/300 Read block 214 len 8192
8504 123/300 Read block 2151 len 16384
8505 124/300 Read block 4873 len 12288
8506 125/300 Read block 4060 len 40960
8507 126/300 Read block 1893 len 40960
8508 127/300 Read block 182 len 36864
8509 128/300 Read block 3310 len 36864
8510 129/300 Flush
8511 130/300 Write block 2879 len 16384 data: 3 2 2 2
8512 131/300 Write block 845 len 8192 data: 2 2
8513 132/300 Flush
8514 133/300 Read block 2166 len 32768
8515 134/300 Read block 4096 len 32768
8516 135/300 Write block 3320 len 16384 data: 4 4 4 4
8517 136/300 Read block 1620 len 12288
8518 137/300 Flush
8519 138/300 Read block 182 len 12288
8520 139/300 Read block 4244 len 40960
8521 140/300 Write block 2312 len 12288 data: 3 3 3
8522 141/300 Write block 723 len 28672 data: 2 2 2 2 2 2 2
8523 142/300 Read block 4364 len 24576
8524 143/300 Write block 2565 len 36864 data: 2 2 2 3 2 2 3 4 4
8525 144/300 Write block 3779 len 20480 data: 4 4 3 3 3
8526 145/300 Write block 4834 len 8192 data: 2 2
8527 146/300 Write block 3246 len 40960 data: 3 3 3 3 3 3 3 2 2 2
8528 147/300 Write block 1347 len 32768 data: 2 2 2 2 2 2 2 2
8529 148/300 Read block 2306 len 24576
8530 149/300 Read block 4829 len 24576
8531 150/300 Write block 1279 len 24576 data: 3 3 2 2 2 2
8532 151/300 Write block 1419 len 36864 data: 2 2 2 2 2 2 3 3 3
8533 152/300 Write block 4735 len 16384 data: 2 2 2 2
8534 153/300 Read block 4517 len 16384
8535 154/300 Write block 3144 len 36864 data: 3 3 3 3 3 4 3 3 4
8536 155/300 Read block 4832 len 8192
8537 156/300 Write block 3017 len 8192 data: 4 3
8538 157/300 Read block 4661 len 20480
8539 158/300 Write block 3256 len 4096 data: 2
8540 159/300 Write block 2342 len 12288 data: 3 3 3
8541 160/300 Write block 2902 len 20480 data: 4 3 3 3 2
8542 161/300 Read block 4382 len 36864
8543 162/300 Write block 4612 len 24576 data: 3 3 3 3 3 3
8544 163/300 Write block 3754 len 36864 data: 3 3 3 3 4 4 4 2 2
8545 164/300 Write block 4302 len 40960 data: 2 3 3 4 4 3 4 4 4 4
8546 165/300 Write block 2115 len 12288 data: 2 2 2
8547 166/300 Flush
8548 167/300 Write block 4916 len 8192 data: 5 5
8549 168/300 Read block 1679 len 28672
8550 169/300 Read block 2462 len 40960
8551 170/300 Read block 3505 len 8192
8552 171/300 Read block 2000 len 12288
8553 172/300 Write block 422 len 16384 data: 4 3 3 3
8554 173/300 Read block 3131 len 28672
8555 174/300 Write block 117 len 36864 data: 3 3 3 4 4 5 5 5 6
8556 175/300 Flush
8557 176/300 Write block 747 len 40960 data: 4 6 4 4 2 2 3 3 2 2
8558 177/300 Write block 851 len 16384 data: 2 2 2 2
8559 178/300 Read block 1726 len 16384
8560 179/300 Read block 3682 len 4096
8561 180/300 Write block 684 len 24576 data: 3 3 2 3 3 3
8562 181/300 Write block 1957 len 16384 data: 3 3 2 2
8563 182/300 Write block 4221 len 32768 data: 3 4 3 3 3 4 4 4
8564 183/300 Read block 3312 len 20480
8565 184/300 Read block 625 len 40960
8566 185/300 Flush
8567 186/300 Write block 2546 len 24576 data: 3 3 3 3 4 4
8568 187/300 Read block 343 len 24576
8569 188/300 Read block 1833 len 12288
8570 189/300 Read block 3041 len 28672
8571 190/300 Write block 4452 len 20480 data: 2 2 2 2 2
8572 191/300 Read block 786 len 24576
8573 192/300 Write block 2980 len 28672 data: 5 4 3 3 3 3 5
8574 193/300 Write block 2345 len 36864 data: 2 2 2 2 2 3 3 2 2
8575 194/300 Read block 2104 len 12288
8576 195/300 Write block 1773 len 4096 data: 3
8577 196/300 Write block 425 len 8192 data: 4 2
8578 197/300 Write block 1523 len 28672 data: 2 2 4 3 3 3 3
8579 198/300 Read block 2848 len 40960
8580 199/300 Read block 4862 len 16384
8581 200/300 Read block 4081 len 20480
8582 201/300 Write block 721 len 40960 data: 2 2 3 3 3 3 3 3 3 2
8583 202/300 Write block 908 len 16384 data: 4 3 3 2
8584 203/300 Read block 231 len 36864
8585 204/300 Read block 3220 len 28672
8586 205/300 Read block 225 len 32768
8587 206/300 Read block 1955 len 8192
8588 207/300 Read block 3053 len 32768
8589 208/300 Flush
8590 209/300 Flush
8591 210/300 Read block 2340 len 40960
8592 211/300 Read block 2417 len 40960
8593 212/300 Read block 2949 len 16384
8594 213/300 Write block 4869 len 16384 data: 3 2 3 3
8595 214/300 Write block 528 len 8192 data: 3 3
8596 215/300 Read block 779 len 24576
8597 216/300 Flush
8598 217/300 Write block 845 len 4096 data: 3
8599 218/300 Flush
8600 219/300 Read block 1997 len 40960
8601 220/300 Write block 4135 len 12288 data: 2 2 2
8602 221/300 Read block 2309 len 40960
8603 222/300 Read block 2957 len 4096
8604 223/300 Write block 443 len 20480 data: 2 2 2 3 3
8605 224/300 Read block 3900 len 40960
8606 225/300 Read block 4526 len 28672
8607 226/300 Read block 1876 len 28672
8608 227/300 Read block 3200 len 28672
8609 228/300 Read block 3992 len 16384
8610 229/300 Write block 2482 len 32768 data: 4 4 3 3 3 3 3 2
8611 230/300 Read block 2139 len 32768
8612 231/300 Read block 1975 len 8192
8613 232/300 Read block 390 len 4096
8614 233/300 Write block 2990 len 16384 data: 5 5 6 5
8615 234/300 Read block 2581 len 12288
8616 235/300 Write block 1524 len 12288 data: 3 5 4
8617 236/300 Write block 3554 len 40960 data: 3 3 4 2 2 2 2 2 2 2
8618 237/300 Read block 969 len 16384
8619 238/300 Read block 512 len 32768
8620 239/300 Write block 4425 len 24576 data: 6 6 5 6 6 6
8621 240/300 Write block 4538 len 8192 data: 2 2
8622 241/300 Flush
8623 242/300 Write block 1478 len 36864 data: 4 6 6 6 6 4 2 2 2
8624 243/300 Read block 4947 len 4096
8625 244/300 Read block 4546 len 4096
8626 245/300 Flush
8627 246/300 Read block 1747 len 28672
8628 247/300 Write block 1736 len 36864 data: 3 3 3 4 4 3 3 3 4
8629 248/300 Write block 4336 len 8192 data: 2 2
8630 249/300 Write block 2630 len 24576 data: 2 2 3 3 2 3
8631 250/300 Read block 3615 len 4096
8632 251/300 Read block 1762 len 12288
8633 252/300 Write block 2938 len 36864 data: 2 2 2 2 3 2 2 3 3
8634 253/300 Read block 2491 len 36864
8635 254/300 Read block 587 len 12288
8636 255/300 Read block 3890 len 32768
8637 256/300 Read block 1246 len 4096
8638 257/300 Read block 3244 len 16384
8639 258/300 Write block 3837 len 36864 data: 3 2 3 3 2 2 2 2 3
8640 259/300 Read block 4570 len 28672
8641 260/300 Read block 1827 len 36864
8642 261/300 Write block 2541 len 32768 data: 4 3 3 2 2 4 4 4
8643 262/300 Write block 4265 len 16384 data: 7 7 6 4
8644 263/300 Read block 3229 len 4096
8645 264/300 Read block 2873 len 36864
8646 265/300 Flush
8647 266/300 Read block 4033 len 36864
8648 267/300 Read block 1465 len 28672
8649 268/300 Read block 1119 len 8192
8650 269/300 Write block 1360 len 40960 data: 4 3 3 3 3 3 2 2 2 2
8651 270/300 Read block 3868 len 28672
8652 271/300 Read block 3057 len 40960
8653 272/300 Write block 3649 len 32768 data: 2 2 2 2 3 3 4 4
8654 273/300 Read block 4317 len 24576
8655 274/300 Write block 2017 len 16384 data: 2 2 2 3
8656 275/300 Read block 849 len 32768
8657 276/300 Read block 576 len 40960
8658 277/300 Write block 815 len 12288 data: 5 5 5
8659 278/300 Write block 705 len 16384 data: 4 4 3 3
8660 279/300 Read block 1283 len 20480
8661 280/300 Flush
8662 281/300 Write block 482 len 4096 data: 3
8663 282/300 Read block 2485 len 28672
8664 283/300 Flush
8665 284/300 Write block 3202 len 16384 data: 2 2 2 2
8666 285/300 Flush
8667 286/300 Write block 1159 len 24576 data: 2 3 3 4 3 3
8668 287/300 Write block 385 len 16384 data: 4 3 3 3
8669 288/300 Read block 1527 len 12288
8670 289/300 Flush
8671 290/300 Read block 4928 len 20480
8672 291/300 Flush
8673 292/300 Write block 4549 len 32768 data: 4 5 4 3 3 3 3 3
8674 293/300 Read block 1947 len 24576
8675 294/300 Read block 2951 len 24576
8676 295/300 Write block 2614 len 32768 data: 3 3 2 2 2 2 2 2
8677 296/300 Read block 1381 len 12288
8678 297/300 Read block 35 len 12288
8679 298/300 Write block 4288 len 32768 data: 2 2 3 4 5 6 6 6
8680 299/300 Read block 331 len 16384
8681 300/300 Read block 3618 len 12288
8682 301/300 Write block 3088 len 4096 data: 2
8683 Main task has work to do, go find it
8684 got dsc Start(2)
8685 start 2
8686 Mar 17 22:36:26.860 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 9f741956-29e5-4490-9613-cc864a4bc89e, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
8687 Replay: started 2, returned:Ok(())
8688 ----------------------------------------------------------------
8689 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8690 States: Active Active Offline
8691 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8692 3114 Acked Write 8 Done Done Sent false
8693 3115 Acked Read 10 Done Done Sent false
8694 3116 Acked Read 8 Done Done Sent false
8695 3117 Acked Read 2 Done Done Sent false
8696 3118 Acked Write 3 Done Done Sent false
8697 3119 Acked Write 1 Done Done Sent false
8698 3120 Acked Write 8 Done Done Sent false
8699 3121 Acked Flush 0 Done Done Sent false
8700 3122 Acked Read 8 Done Done Sent false
8701 3123 Acked Read 10 Done Done Sent false
8702 3124 Acked Flush 0 Done Done Sent false
8703 3125 Acked Flush 0 Done Done Sent false
8704 3126 Acked Flush 0 Done Done Sent false
8705 3127 Acked Write 4 Done Done Sent false
8706 3128 Acked Write 10 Done Done Sent false
8707 3129 Acked Read 5 Done Done Sent false
8708 3130 Acked Write 8 Done Done Sent false
8709 3131 Acked Flush 0 Done Done Sent false
8710 3132 Acked Read 7 Done Done Sent false
8711 3133 Acked Flush 0 Done Done Sent false
8712 3134 Acked Read 5 Done Done Sent false
8713 3135 Acked Read 1 Done Done Sent false
8714 3136 Acked Write 6 Done Done Sent false
8715 3137 Acked Write 5 Done Done Sent false
8716 3138 Acked Read 8 Done Done Sent false
8717 3139 Acked Flush 0 Done Done Sent false
8718 3140 Acked Write 7 Done Done Sent false
8719 3141 Acked Read 6 Done Done Sent false
8720 3142 Acked Write 2 Done Done Sent false
8721 3143 Acked Read 5 Done Done Sent false
8722 3144 Acked Read 3 Done Done Sent false
8723 3145 Acked Write 10 Done Done Sent false
8724 3146 Acked Read 8 Done Done Sent false
8725 3147 Acked Read 4 Done Done Sent false
8726 3148 Acked Read 1 Done Done Sent false
8727 3149 Acked Write 7 Done Done Sent false
8728 3150 Acked Read 6 Done Done Sent false
8729 3151 Acked Flush 0 Done Done Sent false
8730 3152 Acked Flush 0 Done Done Sent false
8731 3153 Acked Read 2 Done Done Sent false
8732 3154 Acked Read 3 Done Done Sent false
8733 3155 Acked Write 7 Done Done Sent false
8734 3156 Acked Read 4 Done Done Sent false
8735 3157 Acked Write 4 Done Done Sent false
8736 3158 Acked Read 5 Done Done Sent false
8737 3159 Acked Read 8 Done Done Sent false
8738 3160 Acked Read 2 Done Done Sent false
8739 3161 Acked Read 7 Done Done Sent false
8740 3162 Acked Read 9 Done Done Sent false
8741 3163 Acked Write 2 Done Done Sent false
8742 3164 Acked Read 9 Done Done Sent false
8743 3165 Acked Read 1 Done Done Sent false
8744 3166 Acked Write 9 Done Done Sent false
8745 3167 Acked Write 2 Done Done Sent false
8746 3168 Acked Read 7 Done Done Sent false
8747 3169 Acked Write 2 Done Done Sent false
8748 3170 Acked Read 9 Done Done Sent false
8749 3171 Acked Write 3 Done Done Sent false
8750 3172 Acked Write 1 Done Done Sent false
8751 3173 Acked Flush 0 Done Done Sent false
8752 3174 Acked Write 9 Done Done Sent false
8753 3175 Acked Write 10 Done Done Sent false
8754 3176 Acked Write 5 Done Done Sent false
8755 3177 Acked Write 4 Done Done Sent false
8756 3178 Acked Read 8 Done Done Sent false
8757 3179 Acked Read 10 Done Done Sent false
8758 3180 Acked Write 3 Done Done Sent false
8759 3181 Acked Write 2 Done Done Sent false
8760 3182 Acked Write 7 Done Done Sent false
8761 3183 Acked Read 4 Done Done Sent false
8762 3184 Acked Read 10 Done Done Sent false
8763 3185 Acked Read 10 Done Done Sent false
8764 3186 Acked Read 3 Done Done Sent false
8765 3187 Acked Read 5 Done Done Sent false
8766 3188 Acked Write 2 Done Done Sent false
8767 3189 Acked Read 8 Done Done Sent false
8768 3190 Acked Flush 0 Done Done Sent false
8769 3191 Acked Write 2 Done Done Sent false
8770 3192 Acked Write 3 Done Done Sent false
8771 3193 Acked Write 1 Done Done Sent false
8772 3194 Acked Read 10 Done Done Sent false
8773 3195 Acked Read 8 Done Done Sent false
8774 3196 Acked Write 9 Done Done Sent false
8775 3197 Acked Write 8 Done Done Sent false
8776 3198 Acked Read 9 Done Done Sent false
8777 3199 Acked Read 7 Done Done Sent false
8778 3200 Acked Write 7 Done Done Sent false
8779 3201 Acked Read 3 Done Done Sent false
8780 3202 Acked Write 5 Done Done Sent false
8781 3203 Acked Write 7 Done Done Sent false
8782 3204 Acked Read 6 Done Done Sent false
8783 3205 Acked Read 2 Done Done Sent false
8784 3206 Acked Read 7 Done Done Sent false
8785 3207 Acked Write 10 Done Done Sent false
8786 3208 Acked Write 7 Done Done Sent false
8787 3209 Acked Flush 0 Done Done Sent false
8788 3210 Acked Write 10 Done Done Sent false
8789 3211 Acked Write 1 Done Done Sent false
8790 3212 Acked Flush 0 Done Done Sent false
8791 3213 Acked Flush 0 Done Done Sent false
8792 3214 Acked Read 3 Done Done Sent false
8793 3215 Acked Write 2 Done Done Sent false
8794 3216 Acked Write 2 Done Done Sent false
8795 3217 Acked Write 9 Done Done Sent false
8796 3218 Acked Read 5 Done Done Sent false
8797 3219 Acked Read 8 Done Done Sent false
8798 3220 Acked Write 6 Done Done Sent false
8799 3221 Acked Read 10 Done Done Sent false
8800 3222 Acked Read 7 Done Done Sent false
8801 3223 Acked Read 7 Done Done Sent false
8802 3224 Acked Write 4 Done Done Sent false
8803 3225 Acked Flush 0 Done Done Sent false
8804 3226 Acked Read 8 Done Done Sent false
8805 3227 Acked Write 4 Done Done Sent false
8806 3228 Acked Flush 0 Done Done Sent false
8807 3229 Acked Read 10 Done Done Sent false
8808 3230 Acked Read 4 Done Done Sent false
8809 3231 Acked Write 7 Done Done Sent false
8810 3232 Acked Write 3 Done Done Sent false
8811 3233 Acked Read 8 Done Done Sent false
8812 3234 Acked Write 10 Done Done Sent false
8813 3235 Acked Read 2 Done Done Sent false
8814 3236 Acked Read 4 Done Done Sent false
8815 3237 Acked Read 3 Done Done Sent false
8816 3238 Acked Read 10 Done Done Sent false
8817 3239 Acked Read 10 Done Done Sent false
8818 3240 Acked Read 9 Done Done Sent false
8819 3241 Acked Read 9 Done Done Sent false
8820 3242 Acked Flush 0 Done Done Sent false
8821 3243 Acked Write 4 Done Done Sent false
8822 3244 Acked Write 2 Done Done Sent false
8823 3245 Acked Flush 0 Done Done Sent false
8824 3246 Acked Read 8 Done Done Sent false
8825 3247 Acked Read 8 Done Done Sent false
8826 3248 Acked Write 4 Done Done Sent false
8827 3249 Acked Read 3 Done Done Sent false
8828 3250 Acked Flush 0 Done Done Sent false
8829 3251 Acked Read 3 Done Done Sent false
8830 3252 Acked Read 10 Done Done Sent false
8831 3253 Acked Write 3 Done Done Sent false
8832 3254 Acked Write 7 Done Done Sent false
8833 3255 Acked Read 6 Done Done Sent false
8834 3256 Acked Write 9 Done Done Sent false
8835 3257 Acked Write 5 Done Done Sent false
8836 3258 Acked Write 2 Done Done Sent false
8837 3259 Acked Write 10 Done Done Sent false
8838 3260 Acked Write 8 Done Done Sent false
8839 3261 Acked Read 6 Done Done Sent false
8840 3262 Acked Read 6 Done Done Sent false
8841 3263 Acked Write 6 Done Done Sent false
8842 3264 Acked Write 9 Done Done Sent false
8843 3265 Acked Write 4 Done Done Sent false
8844 3266 Acked Read 4 Done Done Sent false
8845 3267 Acked Write 9 Done Done Sent false
8846 3268 Acked Read 2 Done Done Sent false
8847 3269 Acked Write 2 Done Done Sent false
8848 3270 Acked Read 5 Done Done Sent false
8849 3271 Acked Write 1 Done Done Sent false
8850 3272 Acked Write 3 Done Done Sent false
8851 3273 Acked Write 5 Done Done Sent false
8852 3274 Acked Read 9 Done Done Sent false
8853 3275 Acked Write 6 Done Done Sent false
8854 3276 Acked Write 9 Done Done Sent false
8855 3277 Acked Write 10 Done Done Sent false
8856 3278 Acked Write 3 Done Done Sent false
8857 3279 Acked Flush 0 Done Done Sent false
8858 3280 Acked Write 2 Done Done Sent false
8859 3281 Acked Read 7 Done Done Sent false
8860 3282 Acked Read 10 Done Done Sent false
8861 3283 Acked Read 2 Done Done Sent false
8862 3284 Acked Read 3 Done Done Sent false
8863 3285 Acked Write 4 Done Done Sent false
8864 3286 Acked Read 7 Done Done Sent false
8865 3287 Acked Write 9 Done Done Sent false
8866 3288 Acked Flush 0 Done Done Sent false
8867 3289 Acked Write 10 Done Done Sent false
8868 3290 Acked Write 4 Done Done Sent false
8869 3291 Acked Read 4 Done Done Sent false
8870 3292 Acked Read 1 Done Done Sent false
8871 3293 Acked Write 6 Done Done Sent false
8872 3294 Acked Write 4 Done Done Sent false
8873 3295 Acked Write 8 Done Done Sent false
8874 3296 Acked Read 5 Done Done Sent false
8875 3297 Acked Read 10 Done Done Sent false
8876 3298 Acked Flush 0 Done Done Sent false
8877 3299 Acked Write 6 Done Done Sent false
8878 3300 Acked Read 6 Done Done Sent false
8879 3301 Acked Read 3 Done Done Sent false
8880 3302 Acked Read 7 Done Done Sent false
8881 3303 Acked Write 5 Done Done Sent false
8882 3304 Acked Read 6 Done Done Sent false
8883 3305 Acked Write 7 Done Done Sent false
8884 3306 Acked Write 9 Done Done Sent false
8885 3307 Acked Read 3 Done Done Sent false
8886 3308 Acked Write 1 Done Done Sent false
8887 3309 Acked Write 2 Done Done Sent false
8888 3310 Acked Write 7 Done Done Sent false
8889 3311 Acked Read 10 Done Done Sent false
8890 3312 Acked Read 4 Done Done Sent false
8891 3313 Acked Read 5 Done Done Sent false
8892 3314 Acked Write 10 Done Done Sent false
8893 3315 Acked Write 4 Done Done Sent false
8894 3316 Acked Read 9 Done Done Sent false
8895 3317 Acked Read 7 Done Done Sent false
8896 3318 Acked Read 8 Done Done Sent false
8897 3319 Acked Read 2 Done Done Sent false
8898 3320 Acked Read 8 Done Done Sent false
8899 3321 Acked Flush 0 Done Done Sent false
8900 3322 Acked Flush 0 Done Done Sent false
8901 3323 Acked Read 10 Done Done Sent false
8902 3324 Acked Read 10 Done Done Sent false
8903 3325 Acked Read 4 Done Done Sent false
8904 3326 Acked Write 4 Done Done Sent false
8905 3327 Acked Write 2 Done Done Sent false
8906 3328 Acked Read 6 Done Done Sent false
8907 3329 Acked Flush 0 Done Done Sent false
8908 3330 Acked Write 1 Done Done Sent false
8909 3331 Acked Flush 0 Done Done Sent false
8910 3332 Acked Read 10 Done Done Sent false
8911 3333 Acked Write 3 Done Done Sent false
8912 3334 Acked Read 10 Done Done Sent false
8913 3335 Acked Read 1 Done Done Sent false
8914 3336 Acked Write 5 Done Done Sent false
8915 3337 Acked Read 10 Done Done Sent false
8916 3338 Acked Read 7 Done Done Sent false
8917 3339 Acked Read 7 Done Done Sent false
8918 3340 Acked Read 7 Done Done Sent false
8919 3341 Acked Read 4 Done Done Sent false
8920 3342 Acked Write 8 Done Done Sent false
8921 3343 Acked Read 8 Done Done Sent false
8922 3344 Acked Read 2 Done Done Sent false
8923 3345 Acked Read 1 Done Done Sent false
8924 3346 Acked Write 4 Done Done Sent false
8925 3347 Acked Read 3 Done Done Sent false
8926 3348 Acked Write 3 Done Done Sent false
8927 3349 Acked Write 10 Done Done Sent false
8928 3350 Acked Read 4 Done Done Sent false
8929 3351 Acked Read 8 Done Done Sent false
8930 3352 Acked Write 6 Done Done Sent false
8931 3353 Acked Write 2 Done Done Sent false
8932 3354 Acked Flush 0 Done Done Sent false
8933 3355 Acked Write 9 Done Done Sent false
8934 3356 Acked Read 1 Done Done Sent false
8935 3357 Acked Read 1 Done Done Sent false
8936 3358 Acked Flush 0 Done Done Sent false
8937 3359 Acked Read 7 Done Done Sent false
8938 3360 Acked Write 9 Done Done Sent false
8939 3361 Acked Write 2 Done Done Sent false
8940 3362 Acked Write 6 Done Done Sent false
8941 3363 Acked Read 1 Done Done Sent false
8942 3364 Acked Read 3 Done Done Sent false
8943 3365 Acked Write 9 Done Done Sent false
8944 3366 Acked Read 9 Done Done Sent false
8945 3367 Acked Read 3 Done Done Sent false
8946 3368 Acked Read 8 Done Done Sent false
8947 3369 Acked Read 1 Done Done Sent false
8948 3370 Acked Read 4 Done Done Sent false
8949 3371 Acked Write 9 Done Done Sent false
8950 3372 Acked Read 7 Done Done Sent false
8951 3373 Acked Read 9 Done Done Sent false
8952 3374 Acked Write 8 Done Done Sent false
8953 3375 Acked Write 4 Done Done Sent false
8954 3376 Acked Read 1 Done Done Sent false
8955 3377 Acked Read 9 Done Done Sent false
8956 3378 Acked Flush 0 Done Done Sent false
8957 3379 Acked Read 9 Done Done Sent false
8958 3380 Acked Read 7 Done Done Sent false
8959 3381 Acked Read 2 Done Done Sent false
8960 3382 Acked Write 10 Done Done Sent false
8961 3383 Acked Read 7 Done Done Sent false
8962 3384 Acked Read 10 Done Done Sent false
8963 3385 Acked Write 8 Done Done Sent false
8964 3386 Acked Read 6 Done Done Sent false
8965 3387 Acked Write 4 Done Done Sent false
8966 3388 Acked Read 8 Done Done Sent false
8967 3389 Acked Read 10 Done Done Sent false
8968 3390 Acked Write 3 Done Done Sent false
8969 3391 Acked Write 4 Done Done Sent false
8970 3392 Acked Read 5 Done Done Sent false
8971 3393 Acked Flush 0 Done Done Sent false
8972 3394 Acked Write 1 Done Done Sent false
8973 3395 Acked Read 7 Done Done Sent false
8974 3396 Acked Flush 0 Done Done Sent false
8975 3397 Acked Write 4 Done Done Sent false
8976 3398 Acked Flush 0 Done Done Sent false
8977 3399 Acked Write 6 Done Done Sent false
8978 3400 Acked Write 4 Done Done Sent false
8979 3401 Acked Read 3 Done Done Sent false
8980 3402 Acked Flush 0 Done Done Sent false
8981 3403 Acked Read 5 Done Done Sent false
8982 3404 Acked Flush 0 Done Done Sent false
8983 3405 Acked Write 8 Done Done Sent false
8984 3406 Acked Read 6 Done Done Sent false
8985 3407 Acked Read 6 Done Done Sent false
8986 3408 Acked Write 8 Done Done Sent false
8987 3409 Acked Read 3 Done Done Sent false
8988 3410 Acked Read 3 Done Done Sent false
8989 3411 Acked Write 8 Done Done Sent false
8990 3412 Acked Read 4 Done Done Sent false
8991 3413 Acked Read 3 Done Done Sent false
8992 3414 Acked Write 1 Done Done Sent false
8993 STATES DS:0 DS:1 DS:2 TOTAL
8994 Sent 0 0 301 301
8995 Done 301 301 0 602
8996 Skipped 0 0 0 0
8997 Error 0 0 0 0
8998 Last Flush: Some(JobId(3404)) Some(JobId(3404)) Some(JobId(3113))
8999 Downstairs last five completed: 3113 3112 3111 3110 3109
9000 Upstairs last five completed: 3414 3413 3412 3411 3410
9001 CLIENT: Up:0 ds:301 act:2
90022025-03-17T22:36:27.485ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
90032025-03-17T22:36:28.611ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
90042025-03-17T22:36:29.489ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
90052025-03-17T22:36:30.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
9006 ----------------------------------------------------------------
9007 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9008 States: Active Active Offline
9009 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9010 3114 Acked Write 8 Done Done Sent false
9011 3115 Acked Read 10 Done Done Sent false
9012 3116 Acked Read 8 Done Done Sent false
9013 3117 Acked Read 2 Done Done Sent false
9014 3118 Acked Write 3 Done Done Sent false
9015 3119 Acked Write 1 Done Done Sent false
9016 3120 Acked Write 8 Done Done Sent false
9017 3121 Acked Flush 0 Done Done Sent false
9018 3122 Acked Read 8 Done Done Sent false
9019 3123 Acked Read 10 Done Done Sent false
9020 3124 Acked Flush 0 Done Done Sent false
9021 3125 Acked Flush 0 Done Done Sent false
9022 3126 Acked Flush 0 Done Done Sent false
9023 3127 Acked Write 4 Done Done Sent false
9024 3128 Acked Write 10 Done Done Sent false
9025 3129 Acked Read 5 Done Done Sent false
9026 3130 Acked Write 8 Done Done Sent false
9027 3131 Acked Flush 0 Done Done Sent false
9028 3132 Acked Read 7 Done Done Sent false
9029 3133 Acked Flush 0 Done Done Sent false
9030 3134 Acked Read 5 Done Done Sent false
9031 3135 Acked Read 1 Done Done Sent false
9032 3136 Acked Write 6 Done Done Sent false
9033 3137 Acked Write 5 Done Done Sent false
9034 3138 Acked Read 8 Done Done Sent false
9035 3139 Acked Flush 0 Done Done Sent false
9036 3140 Acked Write 7 Done Done Sent false
9037 3141 Acked Read 6 Done Done Sent false
9038 3142 Acked Write 2 Done Done Sent false
9039 3143 Acked Read 5 Done Done Sent false
9040 3144 Acked Read 3 Done Done Sent false
9041 3145 Acked Write 10 Done Done Sent false
9042 3146 Acked Read 8 Done Done Sent false
9043 3147 Acked Read 4 Done Done Sent false
9044 3148 Acked Read 1 Done Done Sent false
9045 3149 Acked Write 7 Done Done Sent false
9046 3150 Acked Read 6 Done Done Sent false
9047 3151 Acked Flush 0 Done Done Sent false
9048 3152 Acked Flush 0 Done Done Sent false
9049 3153 Acked Read 2 Done Done Sent false
9050 3154 Acked Read 3 Done Done Sent false
9051 3155 Acked Write 7 Done Done Sent false
9052 3156 Acked Read 4 Done Done Sent false
9053 3157 Acked Write 4 Done Done Sent false
9054 3158 Acked Read 5 Done Done Sent false
9055 3159 Acked Read 8 Done Done Sent false
9056 3160 Acked Read 2 Done Done Sent false
9057 3161 Acked Read 7 Done Done Sent false
9058 3162 Acked Read 9 Done Done Sent false
9059 3163 Acked Write 2 Done Done Sent false
9060 3164 Acked Read 9 Done Done Sent false
9061 3165 Acked Read 1 Done Done Sent false
9062 3166 Acked Write 9 Done Done Sent false
9063 3167 Acked Write 2 Done Done Sent false
9064 3168 Acked Read 7 Done Done Sent false
9065 3169 Acked Write 2 Done Done Sent false
9066 3170 Acked Read 9 Done Done Sent false
9067 3171 Acked Write 3 Done Done Sent false
9068 3172 Acked Write 1 Done Done Sent false
9069 3173 Acked Flush 0 Done Done Sent false
9070 3174 Acked Write 9 Done Done Sent false
9071 3175 Acked Write 10 Done Done Sent false
9072 3176 Acked Write 5 Done Done Sent false
9073 3177 Acked Write 4 Done Done Sent false
9074 3178 Acked Read 8 Done Done Sent false
9075 3179 Acked Read 10 Done Done Sent false
9076 3180 Acked Write 3 Done Done Sent false
9077 3181 Acked Write 2 Done Done Sent false
9078 3182 Acked Write 7 Done Done Sent false
9079 3183 Acked Read 4 Done Done Sent false
9080 3184 Acked Read 10 Done Done Sent false
9081 3185 Acked Read 10 Done Done Sent false
9082 3186 Acked Read 3 Done Done Sent false
9083 3187 Acked Read 5 Done Done Sent false
9084 3188 Acked Write 2 Done Done Sent false
9085 3189 Acked Read 8 Done Done Sent false
9086 3190 Acked Flush 0 Done Done Sent false
9087 3191 Acked Write 2 Done Done Sent false
9088 3192 Acked Write 3 Done Done Sent false
9089 3193 Acked Write 1 Done Done Sent false
9090 3194 Acked Read 10 Done Done Sent false
9091 3195 Acked Read 8 Done Done Sent false
9092 3196 Acked Write 9 Done Done Sent false
9093 3197 Acked Write 8 Done Done Sent false
9094 3198 Acked Read 9 Done Done Sent false
9095 3199 Acked Read 7 Done Done Sent false
9096 3200 Acked Write 7 Done Done Sent false
9097 3201 Acked Read 3 Done Done Sent false
9098 3202 Acked Write 5 Done Done Sent false
9099 3203 Acked Write 7 Done Done Sent false
9100 3204 Acked Read 6 Done Done Sent false
9101 3205 Acked Read 2 Done Done Sent false
9102 3206 Acked Read 7 Done Done Sent false
9103 3207 Acked Write 10 Done Done Sent false
9104 3208 Acked Write 7 Done Done Sent false
9105 3209 Acked Flush 0 Done Done Sent false
9106 3210 Acked Write 10 Done Done Sent false
9107 3211 Acked Write 1 Done Done Sent false
9108 3212 Acked Flush 0 Done Done Sent false
9109 3213 Acked Flush 0 Done Done Sent false
9110 3214 Acked Read 3 Done Done Sent false
9111 3215 Acked Write 2 Done Done Sent false
9112 3216 Acked Write 2 Done Done Sent false
9113 3217 Acked Write 9 Done Done Sent false
9114 3218 Acked Read 5 Done Done Sent false
9115 3219 Acked Read 8 Done Done Sent false
9116 3220 Acked Write 6 Done Done Sent false
9117 3221 Acked Read 10 Done Done Sent false
9118 3222 Acked Read 7 Done Done Sent false
9119 3223 Acked Read 7 Done Done Sent false
9120 3224 Acked Write 4 Done Done Sent false
9121 3225 Acked Flush 0 Done Done Sent false
9122 3226 Acked Read 8 Done Done Sent false
9123 3227 Acked Write 4 Done Done Sent false
9124 3228 Acked Flush 0 Done Done Sent false
9125 3229 Acked Read 10 Done Done Sent false
9126 3230 Acked Read 4 Done Done Sent false
9127 3231 Acked Write 7 Done Done Sent false
9128 3232 Acked Write 3 Done Done Sent false
9129 3233 Acked Read 8 Done Done Sent false
9130 3234 Acked Write 10 Done Done Sent false
9131 3235 Acked Read 2 Done Done Sent false
9132 3236 Acked Read 4 Done Done Sent false
9133 3237 Acked Read 3 Done Done Sent false
9134 3238 Acked Read 10 Done Done Sent false
9135 3239 Acked Read 10 Done Done Sent false
9136 3240 Acked Read 9 Done Done Sent false
9137 3241 Acked Read 9 Done Done Sent false
9138 3242 Acked Flush 0 Done Done Sent false
9139 3243 Acked Write 4 Done Done Sent false
9140 3244 Acked Write 2 Done Done Sent false
9141 3245 Acked Flush 0 Done Done Sent false
9142 3246 Acked Read 8 Done Done Sent false
9143 3247 Acked Read 8 Done Done Sent false
9144 3248 Acked Write 4 Done Done Sent false
9145 3249 Acked Read 3 Done Done Sent false
9146 3250 Acked Flush 0 Done Done Sent false
9147 3251 Acked Read 3 Done Done Sent false
9148 3252 Acked Read 10 Done Done Sent false
9149 3253 Acked Write 3 Done Done Sent false
9150 3254 Acked Write 7 Done Done Sent false
9151 3255 Acked Read 6 Done Done Sent false
9152 3256 Acked Write 9 Done Done Sent false
9153 3257 Acked Write 5 Done Done Sent false
9154 3258 Acked Write 2 Done Done Sent false
9155 3259 Acked Write 10 Done Done Sent false
9156 3260 Acked Write 8 Done Done Sent false
9157 3261 Acked Read 6 Done Done Sent false
9158 3262 Acked Read 6 Done Done Sent false
9159 3263 Acked Write 6 Done Done Sent false
9160 3264 Acked Write 9 Done Done Sent false
9161 3265 Acked Write 4 Done Done Sent false
9162 3266 Acked Read 4 Done Done Sent false
9163 3267 Acked Write 9 Done Done Sent false
9164 3268 Acked Read 2 Done Done Sent false
9165 3269 Acked Write 2 Done Done Sent false
9166 3270 Acked Read 5 Done Done Sent false
9167 3271 Acked Write 1 Done Done Sent false
9168 3272 Acked Write 3 Done Done Sent false
9169 3273 Acked Write 5 Done Done Sent false
9170 3274 Acked Read 9 Done Done Sent false
9171 3275 Acked Write 6 Done Done Sent false
9172 3276 Acked Write 9 Done Done Sent false
9173 3277 Acked Write 10 Done Done Sent false
9174 3278 Acked Write 3 Done Done Sent false
9175 3279 Acked Flush 0 Done Done Sent false
9176 3280 Acked Write 2 Done Done Sent false
9177 3281 Acked Read 7 Done Done Sent false
9178 3282 Acked Read 10 Done Done Sent false
9179 3283 Acked Read 2 Done Done Sent false
9180 3284 Acked Read 3 Done Done Sent false
9181 3285 Acked Write 4 Done Done Sent false
9182 3286 Acked Read 7 Done Done Sent false
9183 3287 Acked Write 9 Done Done Sent false
9184 3288 Acked Flush 0 Done Done Sent false
9185 3289 Acked Write 10 Done Done Sent false
9186 3290 Acked Write 4 Done Done Sent false
9187 3291 Acked Read 4 Done Done Sent false
9188 3292 Acked Read 1 Done Done Sent false
9189 3293 Acked Write 6 Done Done Sent false
9190 3294 Acked Write 4 Done Done Sent false
9191 3295 Acked Write 8 Done Done Sent false
9192 3296 Acked Read 5 Done Done Sent false
9193 3297 Acked Read 10 Done Done Sent false
9194 3298 Acked Flush 0 Done Done Sent false
9195 3299 Acked Write 6 Done Done Sent false
9196 3300 Acked Read 6 Done Done Sent false
9197 3301 Acked Read 3 Done Done Sent false
9198 3302 Acked Read 7 Done Done Sent false
9199 3303 Acked Write 5 Done Done Sent false
9200 3304 Acked Read 6 Done Done Sent false
9201 3305 Acked Write 7 Done Done Sent false
9202 3306 Acked Write 9 Done Done Sent false
9203 3307 Acked Read 3 Done Done Sent false
9204 3308 Acked Write 1 Done Done Sent false
9205 3309 Acked Write 2 Done Done Sent false
9206 3310 Acked Write 7 Done Done Sent false
9207 3311 Acked Read 10 Done Done Sent false
9208 3312 Acked Read 4 Done Done Sent false
9209 3313 Acked Read 5 Done Done Sent false
9210 3314 Acked Write 10 Done Done Sent false
9211 3315 Acked Write 4 Done Done Sent false
9212 3316 Acked Read 9 Done Done Sent false
9213 3317 Acked Read 7 Done Done Sent false
9214 3318 Acked Read 8 Done Done Sent false
9215 3319 Acked Read 2 Done Done Sent false
9216 3320 Acked Read 8 Done Done Sent false
9217 3321 Acked Flush 0 Done Done Sent false
9218 3322 Acked Flush 0 Done Done Sent false
9219 3323 Acked Read 10 Done Done Sent false
9220 3324 Acked Read 10 Done Done Sent false
9221 3325 Acked Read 4 Done Done Sent false
9222 3326 Acked Write 4 Done Done Sent false
9223 3327 Acked Write 2 Done Done Sent false
9224 3328 Acked Read 6 Done Done Sent false
9225 3329 Acked Flush 0 Done Done Sent false
9226 3330 Acked Write 1 Done Done Sent false
9227 3331 Acked Flush 0 Done Done Sent false
9228 3332 Acked Read 10 Done Done Sent false
9229 3333 Acked Write 3 Done Done Sent false
9230 3334 Acked Read 10 Done Done Sent false
9231 3335 Acked Read 1 Done Done Sent false
9232 3336 Acked Write 5 Done Done Sent false
9233 3337 Acked Read 10 Done Done Sent false
9234 3338 Acked Read 7 Done Done Sent false
9235 3339 Acked Read 7 Done Done Sent false
9236 3340 Acked Read 7 Done Done Sent false
9237 3341 Acked Read 4 Done Done Sent false
9238 3342 Acked Write 8 Done Done Sent false
9239 3343 Acked Read 8 Done Done Sent false
9240 3344 Acked Read 2 Done Done Sent false
9241 3345 Acked Read 1 Done Done Sent false
9242 3346 Acked Write 4 Done Done Sent false
9243 3347 Acked Read 3 Done Done Sent false
9244 3348 Acked Write 3 Done Done Sent false
9245 3349 Acked Write 10 Done Done Sent false
9246 3350 Acked Read 4 Done Done Sent false
9247 3351 Acked Read 8 Done Done Sent false
9248 3352 Acked Write 6 Done Done Sent false
9249 3353 Acked Write 2 Done Done Sent false
9250 3354 Acked Flush 0 Done Done Sent false
9251 3355 Acked Write 9 Done Done Sent false
9252 3356 Acked Read 1 Done Done Sent false
9253 3357 Acked Read 1 Done Done Sent false
9254 3358 Acked Flush 0 Done Done Sent false
9255 3359 Acked Read 7 Done Done Sent false
9256 3360 Acked Write 9 Done Done Sent false
9257 3361 Acked Write 2 Done Done Sent false
9258 3362 Acked Write 6 Done Done Sent false
9259 3363 Acked Read 1 Done Done Sent false
9260 3364 Acked Read 3 Done Done Sent false
9261 3365 Acked Write 9 Done Done Sent false
9262 3366 Acked Read 9 Done Done Sent false
9263 3367 Acked Read 3 Done Done Sent false
9264 3368 Acked Read 8 Done Done Sent false
9265 3369 Acked Read 1 Done Done Sent false
9266 3370 Acked Read 4 Done Done Sent false
9267 3371 Acked Write 9 Done Done Sent false
9268 3372 Acked Read 7 Done Done Sent false
9269 3373 Acked Read 9 Done Done Sent false
9270 3374 Acked Write 8 Done Done Sent false
9271 3375 Acked Write 4 Done Done Sent false
9272 3376 Acked Read 1 Done Done Sent false
9273 3377 Acked Read 9 Done Done Sent false
9274 3378 Acked Flush 0 Done Done Sent false
9275 3379 Acked Read 9 Done Done Sent false
9276 3380 Acked Read 7 Done Done Sent false
9277 3381 Acked Read 2 Done Done Sent false
9278 3382 Acked Write 10 Done Done Sent false
9279 3383 Acked Read 7 Done Done Sent false
9280 3384 Acked Read 10 Done Done Sent false
9281 3385 Acked Write 8 Done Done Sent false
9282 3386 Acked Read 6 Done Done Sent false
9283 3387 Acked Write 4 Done Done Sent false
9284 3388 Acked Read 8 Done Done Sent false
9285 3389 Acked Read 10 Done Done Sent false
9286 3390 Acked Write 3 Done Done Sent false
9287 3391 Acked Write 4 Done Done Sent false
9288 3392 Acked Read 5 Done Done Sent false
9289 3393 Acked Flush 0 Done Done Sent false
9290 3394 Acked Write 1 Done Done Sent false
9291 3395 Acked Read 7 Done Done Sent false
9292 3396 Acked Flush 0 Done Done Sent false
9293 3397 Acked Write 4 Done Done Sent false
9294 3398 Acked Flush 0 Done Done Sent false
9295 3399 Acked Write 6 Done Done Sent false
9296 3400 Acked Write 4 Done Done Sent false
9297 3401 Acked Read 3 Done Done Sent false
9298 3402 Acked Flush 0 Done Done Sent false
9299 3403 Acked Read 5 Done Done Sent false
9300 3404 Acked Flush 0 Done Done Sent false
9301 3405 Acked Write 8 Done Done Sent false
9302 3406 Acked Read 6 Done Done Sent false
9303 3407 Acked Read 6 Done Done Sent false
9304 3408 Acked Write 8 Done Done Sent false
9305 3409 Acked Read 3 Done Done Sent false
9306 3410 Acked Read 3 Done Done Sent false
9307 3411 Acked Write 8 Done Done Sent false
9308 3412 Acked Read 4 Done Done Sent false
9309 3413 Acked Read 3 Done Done Sent false
9310 3414 Acked Write 1 Done Done Sent false
9311 STATES DS:0 DS:1 DS:2 TOTAL
9312 Sent 0 0 301 301
9313 Done 301 301 0 602
9314 Skipped 0 0 0 0
9315 Error 0 0 0 0
9316 Last Flush: Some(JobId(3404)) Some(JobId(3404)) Some(JobId(3113))
9317 Downstairs last five completed: 3113 3112 3111 3110 3109
9318 Upstairs last five completed: 3414 3413 3412 3411 3410
9319 CLIENT: Up:0 ds:301 act:2
93202025-03-17T22:36:31.492ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93212025-03-17T22:36:32.494ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93222025-03-17T22:36:32.606ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93232025-03-17T22:36:32.606ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93242025-03-17T22:36:32.606ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93252025-03-17T22:36:32.606ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93262025-03-17T22:36:32.607ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93272025-03-17T22:36:33.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
93282025-03-17T22:36:34.498ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
9329 ----------------------------------------------------------------
9330 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9331 States: Active Active Offline
9332 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9333 3114 Acked Write 8 Done Done Sent false
9334 3115 Acked Read 10 Done Done Sent false
9335 3116 Acked Read 8 Done Done Sent false
9336 3117 Acked Read 2 Done Done Sent false
9337 3118 Acked Write 3 Done Done Sent false
9338 3119 Acked Write 1 Done Done Sent false
9339 3120 Acked Write 8 Done Done Sent false
9340 3121 Acked Flush 0 Done Done Sent false
9341 3122 Acked Read 8 Done Done Sent false
9342 3123 Acked Read 10 Done Done Sent false
9343 3124 Acked Flush 0 Done Done Sent false
9344 3125 Acked Flush 0 Done Done Sent false
9345 3126 Acked Flush 0 Done Done Sent false
9346 3127 Acked Write 4 Done Done Sent false
9347 3128 Acked Write 10 Done Done Sent false
9348 3129 Acked Read 5 Done Done Sent false
9349 3130 Acked Write 8 Done Done Sent false
9350 3131 Acked Flush 0 Done Done Sent false
9351 3132 Acked Read 7 Done Done Sent false
9352 3133 Acked Flush 0 Done Done Sent false
9353 3134 Acked Read 5 Done Done Sent false
9354 3135 Acked Read 1 Done Done Sent false
9355 3136 Acked Write 6 Done Done Sent false
9356 3137 Acked Write 5 Done Done Sent false
9357 3138 Acked Read 8 Done Done Sent false
9358 3139 Acked Flush 0 Done Done Sent false
9359 3140 Acked Write 7 Done Done Sent false
9360 3141 Acked Read 6 Done Done Sent false
9361 3142 Acked Write 2 Done Done Sent false
9362 3143 Acked Read 5 Done Done Sent false
9363 3144 Acked Read 3 Done Done Sent false
9364 3145 Acked Write 10 Done Done Sent false
9365 3146 Acked Read 8 Done Done Sent false
9366 3147 Acked Read 4 Done Done Sent false
9367 3148 Acked Read 1 Done Done Sent false
9368 3149 Acked Write 7 Done Done Sent false
9369 3150 Acked Read 6 Done Done Sent false
9370 3151 Acked Flush 0 Done Done Sent false
9371 3152 Acked Flush 0 Done Done Sent false
9372 3153 Acked Read 2 Done Done Sent false
9373 3154 Acked Read 3 Done Done Sent false
9374 3155 Acked Write 7 Done Done Sent false
9375 3156 Acked Read 4 Done Done Sent false
9376 3157 Acked Write 4 Done Done Sent false
9377 3158 Acked Read 5 Done Done Sent false
9378 3159 Acked Read 8 Done Done Sent false
9379 3160 Acked Read 2 Done Done Sent false
9380 3161 Acked Read 7 Done Done Sent false
9381 3162 Acked Read 9 Done Done Sent false
9382 3163 Acked Write 2 Done Done Sent false
9383 3164 Acked Read 9 Done Done Sent false
9384 3165 Acked Read 1 Done Done Sent false
9385 3166 Acked Write 9 Done Done Sent false
9386 3167 Acked Write 2 Done Done Sent false
9387 3168 Acked Read 7 Done Done Sent false
9388 3169 Acked Write 2 Done Done Sent false
9389 3170 Acked Read 9 Done Done Sent false
9390 3171 Acked Write 3 Done Done Sent false
9391 3172 Acked Write 1 Done Done Sent false
9392 3173 Acked Flush 0 Done Done Sent false
9393 3174 Acked Write 9 Done Done Sent false
9394 3175 Acked Write 10 Done Done Sent false
9395 3176 Acked Write 5 Done Done Sent false
9396 3177 Acked Write 4 Done Done Sent false
9397 3178 Acked Read 8 Done Done Sent false
9398 3179 Acked Read 10 Done Done Sent false
9399 3180 Acked Write 3 Done Done Sent false
9400 3181 Acked Write 2 Done Done Sent false
9401 3182 Acked Write 7 Done Done Sent false
9402 3183 Acked Read 4 Done Done Sent false
9403 3184 Acked Read 10 Done Done Sent false
9404 3185 Acked Read 10 Done Done Sent false
9405 3186 Acked Read 3 Done Done Sent false
9406 3187 Acked Read 5 Done Done Sent false
9407 3188 Acked Write 2 Done Done Sent false
9408 3189 Acked Read 8 Done Done Sent false
9409 3190 Acked Flush 0 Done Done Sent false
9410 3191 Acked Write 2 Done Done Sent false
9411 3192 Acked Write 3 Done Done Sent false
9412 3193 Acked Write 1 Done Done Sent false
9413 3194 Acked Read 10 Done Done Sent false
9414 3195 Acked Read 8 Done Done Sent false
9415 3196 Acked Write 9 Done Done Sent false
9416 3197 Acked Write 8 Done Done Sent false
9417 3198 Acked Read 9 Done Done Sent false
9418 3199 Acked Read 7 Done Done Sent false
9419 3200 Acked Write 7 Done Done Sent false
9420 3201 Acked Read 3 Done Done Sent false
9421 3202 Acked Write 5 Done Done Sent false
9422 3203 Acked Write 7 Done Done Sent false
9423 3204 Acked Read 6 Done Done Sent false
9424 3205 Acked Read 2 Done Done Sent false
9425 3206 Acked Read 7 Done Done Sent false
9426 3207 Acked Write 10 Done Done Sent false
9427 3208 Acked Write 7 Done Done Sent false
9428 3209 Acked Flush 0 Done Done Sent false
9429 3210 Acked Write 10 Done Done Sent false
9430 3211 Acked Write 1 Done Done Sent false
9431 3212 Acked Flush 0 Done Done Sent false
9432 3213 Acked Flush 0 Done Done Sent false
9433 3214 Acked Read 3 Done Done Sent false
9434 3215 Acked Write 2 Done Done Sent false
9435 3216 Acked Write 2 Done Done Sent false
9436 3217 Acked Write 9 Done Done Sent false
9437 3218 Acked Read 5 Done Done Sent false
9438 3219 Acked Read 8 Done Done Sent false
9439 3220 Acked Write 6 Done Done Sent false
9440 3221 Acked Read 10 Done Done Sent false
9441 3222 Acked Read 7 Done Done Sent false
9442 3223 Acked Read 7 Done Done Sent false
9443 3224 Acked Write 4 Done Done Sent false
9444 3225 Acked Flush 0 Done Done Sent false
9445 3226 Acked Read 8 Done Done Sent false
9446 3227 Acked Write 4 Done Done Sent false
9447 3228 Acked Flush 0 Done Done Sent false
9448 3229 Acked Read 10 Done Done Sent false
9449 3230 Acked Read 4 Done Done Sent false
9450 3231 Acked Write 7 Done Done Sent false
9451 3232 Acked Write 3 Done Done Sent false
9452 3233 Acked Read 8 Done Done Sent false
9453 3234 Acked Write 10 Done Done Sent false
9454 3235 Acked Read 2 Done Done Sent false
9455 3236 Acked Read 4 Done Done Sent false
9456 3237 Acked Read 3 Done Done Sent false
9457 3238 Acked Read 10 Done Done Sent false
9458 3239 Acked Read 10 Done Done Sent false
9459 3240 Acked Read 9 Done Done Sent false
9460 3241 Acked Read 9 Done Done Sent false
9461 3242 Acked Flush 0 Done Done Sent false
9462 3243 Acked Write 4 Done Done Sent false
9463 3244 Acked Write 2 Done Done Sent false
9464 3245 Acked Flush 0 Done Done Sent false
9465 3246 Acked Read 8 Done Done Sent false
9466 3247 Acked Read 8 Done Done Sent false
9467 3248 Acked Write 4 Done Done Sent false
9468 3249 Acked Read 3 Done Done Sent false
9469 3250 Acked Flush 0 Done Done Sent false
9470 3251 Acked Read 3 Done Done Sent false
9471 3252 Acked Read 10 Done Done Sent false
9472 3253 Acked Write 3 Done Done Sent false
9473 3254 Acked Write 7 Done Done Sent false
9474 3255 Acked Read 6 Done Done Sent false
9475 3256 Acked Write 9 Done Done Sent false
9476 3257 Acked Write 5 Done Done Sent false
9477 3258 Acked Write 2 Done Done Sent false
9478 3259 Acked Write 10 Done Done Sent false
9479 3260 Acked Write 8 Done Done Sent false
9480 3261 Acked Read 6 Done Done Sent false
9481 3262 Acked Read 6 Done Done Sent false
9482 3263 Acked Write 6 Done Done Sent false
9483 3264 Acked Write 9 Done Done Sent false
9484 3265 Acked Write 4 Done Done Sent false
9485 3266 Acked Read 4 Done Done Sent false
9486 3267 Acked Write 9 Done Done Sent false
9487 3268 Acked Read 2 Done Done Sent false
9488 3269 Acked Write 2 Done Done Sent false
9489 3270 Acked Read 5 Done Done Sent false
9490 3271 Acked Write 1 Done Done Sent false
9491 3272 Acked Write 3 Done Done Sent false
9492 3273 Acked Write 5 Done Done Sent false
9493 3274 Acked Read 9 Done Done Sent false
9494 3275 Acked Write 6 Done Done Sent false
9495 3276 Acked Write 9 Done Done Sent false
9496 3277 Acked Write 10 Done Done Sent false
9497 3278 Acked Write 3 Done Done Sent false
9498 3279 Acked Flush 0 Done Done Sent false
9499 3280 Acked Write 2 Done Done Sent false
9500 3281 Acked Read 7 Done Done Sent false
9501 3282 Acked Read 10 Done Done Sent false
9502 3283 Acked Read 2 Done Done Sent false
9503 3284 Acked Read 3 Done Done Sent false
9504 3285 Acked Write 4 Done Done Sent false
9505 3286 Acked Read 7 Done Done Sent false
9506 3287 Acked Write 9 Done Done Sent false
9507 3288 Acked Flush 0 Done Done Sent false
9508 3289 Acked Write 10 Done Done Sent false
9509 3290 Acked Write 4 Done Done Sent false
9510 3291 Acked Read 4 Done Done Sent false
9511 3292 Acked Read 1 Done Done Sent false
9512 3293 Acked Write 6 Done Done Sent false
9513 3294 Acked Write 4 Done Done Sent false
9514 3295 Acked Write 8 Done Done Sent false
9515 3296 Acked Read 5 Done Done Sent false
9516 3297 Acked Read 10 Done Done Sent false
9517 3298 Acked Flush 0 Done Done Sent false
9518 3299 Acked Write 6 Done Done Sent false
9519 3300 Acked Read 6 Done Done Sent false
9520 3301 Acked Read 3 Done Done Sent false
9521 3302 Acked Read 7 Done Done Sent false
9522 3303 Acked Write 5 Done Done Sent false
9523 3304 Acked Read 6 Done Done Sent false
9524 3305 Acked Write 7 Done Done Sent false
9525 3306 Acked Write 9 Done Done Sent false
9526 3307 Acked Read 3 Done Done Sent false
9527 3308 Acked Write 1 Done Done Sent false
9528 3309 Acked Write 2 Done Done Sent false
9529 3310 Acked Write 7 Done Done Sent false
9530 3311 Acked Read 10 Done Done Sent false
9531 3312 Acked Read 4 Done Done Sent false
9532 3313 Acked Read 5 Done Done Sent false
9533 3314 Acked Write 10 Done Done Sent false
9534 3315 Acked Write 4 Done Done Sent false
9535 3316 Acked Read 9 Done Done Sent false
9536 3317 Acked Read 7 Done Done Sent false
9537 3318 Acked Read 8 Done Done Sent false
9538 3319 Acked Read 2 Done Done Sent false
9539 3320 Acked Read 8 Done Done Sent false
9540 3321 Acked Flush 0 Done Done Sent false
9541 3322 Acked Flush 0 Done Done Sent false
9542 3323 Acked Read 10 Done Done Sent false
9543 3324 Acked Read 10 Done Done Sent false
9544 3325 Acked Read 4 Done Done Sent false
9545 3326 Acked Write 4 Done Done Sent false
9546 3327 Acked Write 2 Done Done Sent false
9547 3328 Acked Read 6 Done Done Sent false
9548 3329 Acked Flush 0 Done Done Sent false
9549 3330 Acked Write 1 Done Done Sent false
9550 3331 Acked Flush 0 Done Done Sent false
9551 3332 Acked Read 10 Done Done Sent false
9552 3333 Acked Write 3 Done Done Sent false
9553 3334 Acked Read 10 Done Done Sent false
9554 3335 Acked Read 1 Done Done Sent false
9555 3336 Acked Write 5 Done Done Sent false
9556 3337 Acked Read 10 Done Done Sent false
9557 3338 Acked Read 7 Done Done Sent false
9558 3339 Acked Read 7 Done Done Sent false
9559 3340 Acked Read 7 Done Done Sent false
9560 3341 Acked Read 4 Done Done Sent false
9561 3342 Acked Write 8 Done Done Sent false
9562 3343 Acked Read 8 Done Done Sent false
9563 3344 Acked Read 2 Done Done Sent false
9564 3345 Acked Read 1 Done Done Sent false
9565 3346 Acked Write 4 Done Done Sent false
9566 3347 Acked Read 3 Done Done Sent false
9567 3348 Acked Write 3 Done Done Sent false
9568 3349 Acked Write 10 Done Done Sent false
9569 3350 Acked Read 4 Done Done Sent false
9570 3351 Acked Read 8 Done Done Sent false
9571 3352 Acked Write 6 Done Done Sent false
9572 3353 Acked Write 2 Done Done Sent false
9573 3354 Acked Flush 0 Done Done Sent false
9574 3355 Acked Write 9 Done Done Sent false
9575 3356 Acked Read 1 Done Done Sent false
9576 3357 Acked Read 1 Done Done Sent false
9577 3358 Acked Flush 0 Done Done Sent false
9578 3359 Acked Read 7 Done Done Sent false
9579 3360 Acked Write 9 Done Done Sent false
9580 3361 Acked Write 2 Done Done Sent false
9581 3362 Acked Write 6 Done Done Sent false
9582 3363 Acked Read 1 Done Done Sent false
9583 3364 Acked Read 3 Done Done Sent false
9584 3365 Acked Write 9 Done Done Sent false
9585 3366 Acked Read 9 Done Done Sent false
9586 3367 Acked Read 3 Done Done Sent false
9587 3368 Acked Read 8 Done Done Sent false
9588 3369 Acked Read 1 Done Done Sent false
9589 3370 Acked Read 4 Done Done Sent false
9590 3371 Acked Write 9 Done Done Sent false
9591 3372 Acked Read 7 Done Done Sent false
9592 3373 Acked Read 9 Done Done Sent false
9593 3374 Acked Write 8 Done Done Sent false
9594 3375 Acked Write 4 Done Done Sent false
9595 3376 Acked Read 1 Done Done Sent false
9596 3377 Acked Read 9 Done Done Sent false
9597 3378 Acked Flush 0 Done Done Sent false
9598 3379 Acked Read 9 Done Done Sent false
9599 3380 Acked Read 7 Done Done Sent false
9600 3381 Acked Read 2 Done Done Sent false
9601 3382 Acked Write 10 Done Done Sent false
9602 3383 Acked Read 7 Done Done Sent false
9603 3384 Acked Read 10 Done Done Sent false
9604 3385 Acked Write 8 Done Done Sent false
9605 3386 Acked Read 6 Done Done Sent false
9606 3387 Acked Write 4 Done Done Sent false
9607 3388 Acked Read 8 Done Done Sent false
9608 3389 Acked Read 10 Done Done Sent false
9609 3390 Acked Write 3 Done Done Sent false
9610 3391 Acked Write 4 Done Done Sent false
9611 3392 Acked Read 5 Done Done Sent false
9612 3393 Acked Flush 0 Done Done Sent false
9613 3394 Acked Write 1 Done Done Sent false
9614 3395 Acked Read 7 Done Done Sent false
9615 3396 Acked Flush 0 Done Done Sent false
9616 3397 Acked Write 4 Done Done Sent false
9617 3398 Acked Flush 0 Done Done Sent false
9618 3399 Acked Write 6 Done Done Sent false
9619 3400 Acked Write 4 Done Done Sent false
9620 3401 Acked Read 3 Done Done Sent false
9621 3402 Acked Flush 0 Done Done Sent false
9622 3403 Acked Read 5 Done Done Sent false
9623 3404 Acked Flush 0 Done Done Sent false
9624 3405 Acked Write 8 Done Done Sent false
9625 3406 Acked Read 6 Done Done Sent false
9626 3407 Acked Read 6 Done Done Sent false
9627 3408 Acked Write 8 Done Done Sent false
9628 3409 Acked Read 3 Done Done Sent false
9629 3410 Acked Read 3 Done Done Sent false
9630 3411 Acked Write 8 Done Done Sent false
9631 3412 Acked Read 4 Done Done Sent false
9632 3413 Acked Read 3 Done Done Sent false
9633 3414 Acked Write 1 Done Done Sent false
9634 STATES DS:0 DS:1 DS:2 TOTAL
9635 Sent 0 0 301 301
9636 Done 301 301 0 602
9637 Skipped 0 0 0 0
9638 Error 0 0 0 0
9639 Last Flush: Some(JobId(3404)) Some(JobId(3404)) Some(JobId(3113))
9640 Downstairs last five completed: 3113 3112 3111 3110 3109
9641 Upstairs last five completed: 3414 3413 3412 3411 3410
9642 CLIENT: Up:0 ds:301 act:2
96432025-03-17T22:36:35.500ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
96442025-03-17T22:36:36.502ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
96452025-03-17T22:36:37.504ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
96462025-03-17T22:36:38.613ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
9647 [8830] got start action
9648 [8830] I am going to restart
9649 Starting downstairs at port 8830
9650 Make output file at "/tmp/dsc/downstairs-8830.txt"
9651 [8830][2] reports Starting
9652 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
9653 [8830][2] reports Running
9654 ----------------------------------------------------------------
9655 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9656 States: Active Active Offline
9657 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9658 3114 Acked Write 8 Done Done Sent false
9659 3115 Acked Read 10 Done Done Sent false
9660 3116 Acked Read 8 Done Done Sent false
9661 3117 Acked Read 2 Done Done Sent false
9662 3118 Acked Write 3 Done Done Sent false
9663 3119 Acked Write 1 Done Done Sent false
9664 3120 Acked Write 8 Done Done Sent false
9665 3121 Acked Flush 0 Done Done Sent false
9666 3122 Acked Read 8 Done Done Sent false
9667 3123 Acked Read 10 Done Done Sent false
9668 3124 Acked Flush 0 Done Done Sent false
9669 3125 Acked Flush 0 Done Done Sent false
9670 3126 Acked Flush 0 Done Done Sent false
9671 3127 Acked Write 4 Done Done Sent false
9672 3128 Acked Write 10 Done Done Sent false
9673 3129 Acked Read 5 Done Done Sent false
9674 3130 Acked Write 8 Done Done Sent false
9675 3131 Acked Flush 0 Done Done Sent false
9676 3132 Acked Read 7 Done Done Sent false
9677 3133 Acked Flush 0 Done Done Sent false
9678 3134 Acked Read 5 Done Done Sent false
9679 3135 Acked Read 1 Done Done Sent false
9680 3136 Acked Write 6 Done Done Sent false
9681 3137 Acked Write 5 Done Done Sent false
9682 3138 Acked Read 8 Done Done Sent false
9683 3139 Acked Flush 0 Done Done Sent false
9684 3140 Acked Write 7 Done Done Sent false
9685 3141 Acked Read 6 Done Done Sent false
9686 3142 Acked Write 2 Done Done Sent false
9687 3143 Acked Read 5 Done Done Sent false
9688 3144 Acked Read 3 Done Done Sent false
9689 3145 Acked Write 10 Done Done Sent false
9690 3146 Acked Read 8 Done Done Sent false
9691 3147 Acked Read 4 Done Done Sent false
9692 3148 Acked Read 1 Done Done Sent false
9693 3149 Acked Write 7 Done Done Sent false
9694 3150 Acked Read 6 Done Done Sent false
9695 3151 Acked Flush 0 Done Done Sent false
9696 3152 Acked Flush 0 Done Done Sent false
9697 3153 Acked Read 2 Done Done Sent false
9698 3154 Acked Read 3 Done Done Sent false
9699 3155 Acked Write 7 Done Done Sent false
9700 3156 Acked Read 4 Done Done Sent false
9701 3157 Acked Write 4 Done Done Sent false
9702 3158 Acked Read 5 Done Done Sent false
9703 3159 Acked Read 8 Done Done Sent false
9704 3160 Acked Read 2 Done Done Sent false
9705 3161 Acked Read 7 Done Done Sent false
9706 3162 Acked Read 9 Done Done Sent false
9707 3163 Acked Write 2 Done Done Sent false
9708 3164 Acked Read 9 Done Done Sent false
9709 3165 Acked Read 1 Done Done Sent false
9710 3166 Acked Write 9 Done Done Sent false
9711 3167 Acked Write 2 Done Done Sent false
9712 3168 Acked Read 7 Done Done Sent false
9713 3169 Acked Write 2 Done Done Sent false
9714 3170 Acked Read 9 Done Done Sent false
9715 3171 Acked Write 3 Done Done Sent false
9716 3172 Acked Write 1 Done Done Sent false
9717 3173 Acked Flush 0 Done Done Sent false
9718 3174 Acked Write 9 Done Done Sent false
9719 3175 Acked Write 10 Done Done Sent false
9720 3176 Acked Write 5 Done Done Sent false
9721 3177 Acked Write 4 Done Done Sent false
9722 3178 Acked Read 8 Done Done Sent false
9723 3179 Acked Read 10 Done Done Sent false
9724 3180 Acked Write 3 Done Done Sent false
9725 3181 Acked Write 2 Done Done Sent false
9726 3182 Acked Write 7 Done Done Sent false
9727 3183 Acked Read 4 Done Done Sent false
9728 3184 Acked Read 10 Done Done Sent false
9729 3185 Acked Read 10 Done Done Sent false
9730 3186 Acked Read 3 Done Done Sent false
9731 3187 Acked Read 5 Done Done Sent false
9732 3188 Acked Write 2 Done Done Sent false
9733 3189 Acked Read 8 Done Done Sent false
9734 3190 Acked Flush 0 Done Done Sent false
9735 3191 Acked Write 2 Done Done Sent false
9736 3192 Acked Write 3 Done Done Sent false
9737 3193 Acked Write 1 Done Done Sent false
9738 3194 Acked Read 10 Done Done Sent false
9739 3195 Acked Read 8 Done Done Sent false
9740 3196 Acked Write 9 Done Done Sent false
9741 3197 Acked Write 8 Done Done Sent false
9742 3198 Acked Read 9 Done Done Sent false
9743 3199 Acked Read 7 Done Done Sent false
9744 3200 Acked Write 7 Done Done Sent false
9745 3201 Acked Read 3 Done Done Sent false
9746 3202 Acked Write 5 Done Done Sent false
9747 3203 Acked Write 7 Done Done Sent false
9748 3204 Acked Read 6 Done Done Sent false
9749 3205 Acked Read 2 Done Done Sent false
9750 3206 Acked Read 7 Done Done Sent false
9751 3207 Acked Write 10 Done Done Sent false
9752 3208 Acked Write 7 Done Done Sent false
9753 3209 Acked Flush 0 Done Done Sent false
9754 3210 Acked Write 10 Done Done Sent false
9755 3211 Acked Write 1 Done Done Sent false
9756 3212 Acked Flush 0 Done Done Sent false
9757 3213 Acked Flush 0 Done Done Sent false
9758 3214 Acked Read 3 Done Done Sent false
9759 3215 Acked Write 2 Done Done Sent false
9760 3216 Acked Write 2 Done Done Sent false
9761 3217 Acked Write 9 Done Done Sent false
9762 3218 Acked Read 5 Done Done Sent false
9763 3219 Acked Read 8 Done Done Sent false
9764 3220 Acked Write 6 Done Done Sent false
9765 3221 Acked Read 10 Done Done Sent false
9766 3222 Acked Read 7 Done Done Sent false
9767 3223 Acked Read 7 Done Done Sent false
9768 3224 Acked Write 4 Done Done Sent false
9769 3225 Acked Flush 0 Done Done Sent false
9770 3226 Acked Read 8 Done Done Sent false
9771 3227 Acked Write 4 Done Done Sent false
9772 3228 Acked Flush 0 Done Done Sent false
9773 3229 Acked Read 10 Done Done Sent false
9774 3230 Acked Read 4 Done Done Sent false
9775 3231 Acked Write 7 Done Done Sent false
9776 3232 Acked Write 3 Done Done Sent false
9777 3233 Acked Read 8 Done Done Sent false
9778 3234 Acked Write 10 Done Done Sent false
9779 3235 Acked Read 2 Done Done Sent false
9780 3236 Acked Read 4 Done Done Sent false
9781 3237 Acked Read 3 Done Done Sent false
9782 3238 Acked Read 10 Done Done Sent false
9783 3239 Acked Read 10 Done Done Sent false
9784 3240 Acked Read 9 Done Done Sent false
9785 3241 Acked Read 9 Done Done Sent false
9786 3242 Acked Flush 0 Done Done Sent false
9787 3243 Acked Write 4 Done Done Sent false
9788 3244 Acked Write 2 Done Done Sent false
9789 3245 Acked Flush 0 Done Done Sent false
9790 3246 Acked Read 8 Done Done Sent false
9791 3247 Acked Read 8 Done Done Sent false
9792 3248 Acked Write 4 Done Done Sent false
9793 3249 Acked Read 3 Done Done Sent false
9794 3250 Acked Flush 0 Done Done Sent false
9795 3251 Acked Read 3 Done Done Sent false
9796 3252 Acked Read 10 Done Done Sent false
9797 3253 Acked Write 3 Done Done Sent false
9798 3254 Acked Write 7 Done Done Sent false
9799 3255 Acked Read 6 Done Done Sent false
9800 3256 Acked Write 9 Done Done Sent false
9801 3257 Acked Write 5 Done Done Sent false
9802 3258 Acked Write 2 Done Done Sent false
9803 3259 Acked Write 10 Done Done Sent false
9804 3260 Acked Write 8 Done Done Sent false
9805 3261 Acked Read 6 Done Done Sent false
9806 3262 Acked Read 6 Done Done Sent false
9807 3263 Acked Write 6 Done Done Sent false
9808 3264 Acked Write 9 Done Done Sent false
9809 3265 Acked Write 4 Done Done Sent false
9810 3266 Acked Read 4 Done Done Sent false
9811 3267 Acked Write 9 Done Done Sent false
9812 3268 Acked Read 2 Done Done Sent false
9813 3269 Acked Write 2 Done Done Sent false
9814 3270 Acked Read 5 Done Done Sent false
9815 3271 Acked Write 1 Done Done Sent false
9816 3272 Acked Write 3 Done Done Sent false
9817 3273 Acked Write 5 Done Done Sent false
9818 3274 Acked Read 9 Done Done Sent false
9819 3275 Acked Write 6 Done Done Sent false
9820 3276 Acked Write 9 Done Done Sent false
9821 3277 Acked Write 10 Done Done Sent false
9822 3278 Acked Write 3 Done Done Sent false
9823 3279 Acked Flush 0 Done Done Sent false
9824 3280 Acked Write 2 Done Done Sent false
9825 3281 Acked Read 7 Done Done Sent false
9826 3282 Acked Read 10 Done Done Sent false
9827 3283 Acked Read 2 Done Done Sent false
9828 3284 Acked Read 3 Done Done Sent false
9829 3285 Acked Write 4 Done Done Sent false
9830 3286 Acked Read 7 Done Done Sent false
9831 3287 Acked Write 9 Done Done Sent false
9832 3288 Acked Flush 0 Done Done Sent false
9833 3289 Acked Write 10 Done Done Sent false
9834 3290 Acked Write 4 Done Done Sent false
9835 3291 Acked Read 4 Done Done Sent false
9836 3292 Acked Read 1 Done Done Sent false
9837 3293 Acked Write 6 Done Done Sent false
9838 3294 Acked Write 4 Done Done Sent false
9839 3295 Acked Write 8 Done Done Sent false
9840 3296 Acked Read 5 Done Done Sent false
9841 3297 Acked Read 10 Done Done Sent false
9842 3298 Acked Flush 0 Done Done Sent false
9843 3299 Acked Write 6 Done Done Sent false
9844 3300 Acked Read 6 Done Done Sent false
9845 3301 Acked Read 3 Done Done Sent false
9846 3302 Acked Read 7 Done Done Sent false
9847 3303 Acked Write 5 Done Done Sent false
9848 3304 Acked Read 6 Done Done Sent false
9849 3305 Acked Write 7 Done Done Sent false
9850 3306 Acked Write 9 Done Done Sent false
9851 3307 Acked Read 3 Done Done Sent false
9852 3308 Acked Write 1 Done Done Sent false
9853 3309 Acked Write 2 Done Done Sent false
9854 3310 Acked Write 7 Done Done Sent false
9855 3311 Acked Read 10 Done Done Sent false
9856 3312 Acked Read 4 Done Done Sent false
9857 3313 Acked Read 5 Done Done Sent false
9858 3314 Acked Write 10 Done Done Sent false
9859 3315 Acked Write 4 Done Done Sent false
9860 3316 Acked Read 9 Done Done Sent false
9861 3317 Acked Read 7 Done Done Sent false
9862 3318 Acked Read 8 Done Done Sent false
9863 3319 Acked Read 2 Done Done Sent false
9864 3320 Acked Read 8 Done Done Sent false
9865 3321 Acked Flush 0 Done Done Sent false
9866 3322 Acked Flush 0 Done Done Sent false
9867 3323 Acked Read 10 Done Done Sent false
9868 3324 Acked Read 10 Done Done Sent false
9869 3325 Acked Read 4 Done Done Sent false
9870 3326 Acked Write 4 Done Done Sent false
9871 3327 Acked Write 2 Done Done Sent false
9872 3328 Acked Read 6 Done Done Sent false
9873 3329 Acked Flush 0 Done Done Sent false
9874 3330 Acked Write 1 Done Done Sent false
9875 3331 Acked Flush 0 Done Done Sent false
9876 3332 Acked Read 10 Done Done Sent false
9877 3333 Acked Write 3 Done Done Sent false
9878 3334 Acked Read 10 Done Done Sent false
9879 3335 Acked Read 1 Done Done Sent false
9880 3336 Acked Write 5 Done Done Sent false
9881 3337 Acked Read 10 Done Done Sent false
9882 3338 Acked Read 7 Done Done Sent false
9883 3339 Acked Read 7 Done Done Sent false
9884 3340 Acked Read 7 Done Done Sent false
9885 3341 Acked Read 4 Done Done Sent false
9886 3342 Acked Write 8 Done Done Sent false
9887 3343 Acked Read 8 Done Done Sent false
9888 3344 Acked Read 2 Done Done Sent false
9889 3345 Acked Read 1 Done Done Sent false
9890 3346 Acked Write 4 Done Done Sent false
9891 3347 Acked Read 3 Done Done Sent false
9892 3348 Acked Write 3 Done Done Sent false
9893 3349 Acked Write 10 Done Done Sent false
9894 3350 Acked Read 4 Done Done Sent false
9895 3351 Acked Read 8 Done Done Sent false
9896 3352 Acked Write 6 Done Done Sent false
9897 3353 Acked Write 2 Done Done Sent false
9898 3354 Acked Flush 0 Done Done Sent false
9899 3355 Acked Write 9 Done Done Sent false
9900 3356 Acked Read 1 Done Done Sent false
9901 3357 Acked Read 1 Done Done Sent false
9902 3358 Acked Flush 0 Done Done Sent false
9903 3359 Acked Read 7 Done Done Sent false
9904 3360 Acked Write 9 Done Done Sent false
9905 3361 Acked Write 2 Done Done Sent false
9906 3362 Acked Write 6 Done Done Sent false
9907 3363 Acked Read 1 Done Done Sent false
9908 3364 Acked Read 3 Done Done Sent false
9909 3365 Acked Write 9 Done Done Sent false
9910 3366 Acked Read 9 Done Done Sent false
9911 3367 Acked Read 3 Done Done Sent false
9912 3368 Acked Read 8 Done Done Sent false
9913 3369 Acked Read 1 Done Done Sent false
9914 3370 Acked Read 4 Done Done Sent false
9915 3371 Acked Write 9 Done Done Sent false
9916 3372 Acked Read 7 Done Done Sent false
9917 3373 Acked Read 9 Done Done Sent false
9918 3374 Acked Write 8 Done Done Sent false
9919 3375 Acked Write 4 Done Done Sent false
9920 3376 Acked Read 1 Done Done Sent false
9921 3377 Acked Read 9 Done Done Sent false
9922 3378 Acked Flush 0 Done Done Sent false
9923 3379 Acked Read 9 Done Done Sent false
9924 3380 Acked Read 7 Done Done Sent false
9925 3381 Acked Read 2 Done Done Sent false
9926 3382 Acked Write 10 Done Done Sent false
9927 3383 Acked Read 7 Done Done Sent false
9928 3384 Acked Read 10 Done Done Sent false
9929 3385 Acked Write 8 Done Done Sent false
9930 3386 Acked Read 6 Done Done Sent false
9931 3387 Acked Write 4 Done Done Sent false
9932 3388 Acked Read 8 Done Done Sent false
9933 3389 Acked Read 10 Done Done Sent false
9934 3390 Acked Write 3 Done Done Sent false
9935 3391 Acked Write 4 Done Done Sent false
9936 3392 Acked Read 5 Done Done Sent false
9937 3393 Acked Flush 0 Done Done Sent false
9938 3394 Acked Write 1 Done Done Sent false
9939 3395 Acked Read 7 Done Done Sent false
9940 3396 Acked Flush 0 Done Done Sent false
9941 3397 Acked Write 4 Done Done Sent false
9942 3398 Acked Flush 0 Done Done Sent false
9943 3399 Acked Write 6 Done Done Sent false
9944 3400 Acked Write 4 Done Done Sent false
9945 3401 Acked Read 3 Done Done Sent false
9946 3402 Acked Flush 0 Done Done Sent false
9947 3403 Acked Read 5 Done Done Sent false
9948 3404 Acked Flush 0 Done Done Sent false
9949 3405 Acked Write 8 Done Done Sent false
9950 3406 Acked Read 6 Done Done Sent false
9951 3407 Acked Read 6 Done Done Sent false
9952 3408 Acked Write 8 Done Done Sent false
9953 3409 Acked Read 3 Done Done Sent false
9954 3410 Acked Read 3 Done Done Sent false
9955 3411 Acked Write 8 Done Done Sent false
9956 3412 Acked Read 4 Done Done Sent false
9957 3413 Acked Read 3 Done Done Sent false
9958 3414 Acked Write 1 Done Done Sent false
9959 STATES DS:0 DS:1 DS:2 TOTAL
9960 Sent 0 0 301 301
9961 Done 301 301 0 602
9962 Skipped 0 0 0 0
9963 Error 0 0 0 0
9964 Last Flush: Some(JobId(3404)) Some(JobId(3404)) Some(JobId(3113))
9965 Downstairs last five completed: 3113 3112 3111 3110 3109
9966 Upstairs last five completed: 3414 3413 3412 3411 3410
9967 CLIENT: Up:0 ds:301 act:2
99682025-03-17T22:36:39.507ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99692025-03-17T22:36:40.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99702025-03-17T22:36:41.511ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99712025-03-17T22:36:42.513ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99722025-03-17T22:36:42.608ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99732025-03-17T22:36:42.608ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38351) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99742025-03-17T22:36:42.609ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99752025-03-17T22:36:42.610ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99762025-03-17T22:36:42.610ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99772025-03-17T22:36:42.610ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99782025-03-17T22:36:42.610ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99792025-03-17T22:36:42.610ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
99802025-03-17T22:36:42.610ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
9981 ----------------------------------------------------------------
9982 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
9983 States: Active Active Active
9984 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9985 3405 Acked Write 8 Done Done Done true
9986 3406 Acked Read 6 Done Done Done true
9987 3407 Acked Read 6 Done Done Done true
9988 3408 Acked Write 8 Done Done Done true
9989 3409 Acked Read 3 Done Done Done true
9990 3410 Acked Read 3 Done Done Done true
9991 3411 Acked Write 8 Done Done Done true
9992 3412 Acked Read 4 Done Done Done true
9993 3413 Acked Read 3 Done Done Done true
9994 3414 Acked Write 1 Done Done Done true
9995 STATES DS:0 DS:1 DS:2 TOTAL
9996 Sent 0 0 0 0
9997 Done 10 10 10 30
9998 Skipped 0 0 0 0
9999 Error 0 0 0 0
10000 Last Flush: Some(JobId(3404)) Some(JobId(3404)) Some(JobId(3404))
10001 Downstairs last five completed: 3404 3403 3402 3401 3400
10002 Upstairs last five completed: 3414 3413 3412 3411 3410
10003 CLIENT: Up:0 ds:10 act:3
10004 ----------------------------------------------------------------
10005 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10006 Downstairs last five completed: 3415 3414 3413 3412 3411
10007 Upstairs last five completed: 3415 3414 3413 3412 3411
10008 CLIENT: Up:0 ds:0 act:3
10009 Replay: All jobs finished
10010 Main task has work to do, go find it
10011 got dsc Stop(2)
10012 stop 2
10013 Mar 17 22:36:46.870 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/2, method: POST, req_id[8830] Got stop action so:false kr:false
10014 : 44beec9b-9ac2-47bf-b7d9-fc11c3983f2e, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
100152025-03-17T22:36:46.870ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
100162025-03-17T22:36:46.870ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
100172025-03-17T22:36:46.870ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
100182025-03-17T22:36:46.871ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
10019 Mar 17 22:36:46.871 INFO request completed, latency_us: 262, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8326f713-a30b-4327-bdab-4b4ab2d02493, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
10020 [8830] Exited with: ExitStatus(unix_wait_status(9))
10021 [8830][2] reports Exit
10022 Mar 17 22:36:50.873 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4f49a3b3-6dfb-43b6-920d-65e7a90fbc94, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
10023 001/300 Read block 209 len 20480
10024 002/300 Read block 932 len 28672
10025 003/300 Read block 167 len 12288
10026 004/300 Read block 1058 len 36864
10027 005/300 Read block 112 len 8192
10028 006/300 Read block 3795 len 32768
10029 007/300 Flush
10030 008/300 Write block 3231 len 36864 data: 2 2 2 2 2 2 2 2 2
10031 009/300 Read block 4315 len 16384
10032 010/300 Read block 3300 len 28672
10033 011/300 Write block 1648 len 40960 data: 3 2 2 2 2 2 2 3 4 4
10034 012/300 Read block 1515 len 16384
10035 013/300 Write block 4784 len 16384 data: 4 3 3 2
10036 014/300 Read block 1893 len 40960
10037 015/300 Read block 2506 len 8192
10038 016/300 Flush
10039 017/300 Read block 3241 len 20480
10040 018/300 Read block 2113 len 40960
10041 019/300 Read block 4877 len 24576
10042 020/300 Read block 2774 len 16384
10043 021/300 Write block 4513 len 8192 data: 3 4
10044 022/300 Flush
10045 023/300 Read block 2329 len 24576
10046 024/300 Read block 4527 len 12288
10047 025/300 Flush
10048 026/300 Read block 2874 len 8192
10049 027/300 Write block 2481 len 24576 data: 4 5 5 4 4 4
10050 028/300 Flush
10051 029/300 Write block 2003 len 20480 data: 4 4 3 3 3
10052 030/300 Flush
10053 031/300 Read block 329 len 40960
10054 032/300 Write block 2549 len 40960 data: 4 5 5 3 3 3 3 4 3 3
10055 033/300 Write block 2847 len 8192 data: 2 2
10056 034/300 Write block 74 len 28672 data: 2 2 2 2 2 2 3
10057 035/300 Flush
10058 036/300 Flush
10059 037/300 Write block 764 len 32768 data: 3 3 3 3 3 3 3 4
10060 038/300 Read block 3060 len 8192
10061 039/300 Write block 1772 len 12288 data: 4 4 3
10062 040/300 Read block 3008 len 20480
10063 041/300 Flush
10064 042/300 Write block 1683 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10065 043/300 Write block 2185 len 28672 data: 3 3 2 2 2 2 2
10066 044/300 Read block 1923 len 12288
10067 045/300 Read block 2701 len 4096
10068 046/300 Read block 2311 len 36864
10069 047/300 Flush
10070 048/300 Write block 4282 len 32768 data: 3 3 3 3 3 3 3 3
10071 049/300 Read block 778 len 32768
10072 050/300 Read block 1119 len 8192
10073 051/300 Read block 823 len 40960
10074 052/300 Write block 2767 len 24576 data: 3 3 3 3 2 3
10075 053/300 Read block 4874 len 20480
10076 054/300 Read block 1897 len 12288
10077 055/300 Read block 3890 len 36864
10078 056/300 Read block 2131 len 24576
10079 057/300 Read block 2286 len 28672
10080 058/300 Read block 4403 len 32768
10081 059/300 Read block 1020 len 4096
10082 060/300 Read block 1754 len 16384
10083 061/300 Flush
10084 062/300 Write block 1454 len 12288 data: 4 4 4
10085 063/300 Write block 1346 len 36864 data: 3 3 3 3 3 3 3 3 3
10086 064/300 Write block 34 len 40960 data: 3 4 3 3 2 3 3 3 3 3
10087 065/300 Write block 2484 len 24576 data: 5 5 5 4 4 3
10088 066/300 Write block 237 len 8192 data: 3 4
10089 067/300 Write block 3249 len 20480 data: 4 4 4 4 3
10090 068/300 Write block 3127 len 32768 data: 3 4 4 4 4 3 4 4
10091 069/300 Write block 517 len 32768 data: 3 3 4 4 4 5 5 5
10092 070/300 Write block 1818 len 36864 data: 3 3 3 3 3 3 2 2 3
10093 071/300 Read block 3958 len 40960
10094 072/300 Read block 379 len 8192
10095 073/300 Read block 4293 len 8192
10096 074/300 Read block 2349 len 28672
10097 075/300 Read block 4103 len 28672
10098 076/300 Read block 3207 len 12288
10099 077/300 Write block 3508 len 12288 data: 2 2 2
10100 078/300 Read block 850 len 32768
10101 079/300 Flush
10102 080/300 Read block 4140 len 4096
10103 081/300 Read block 2578 len 16384
10104 082/300 Read block 3316 len 40960
10105 083/300 Write block 2746 len 32768 data: 2 2 3 3 3 3 3 3
10106 084/300 Write block 149 len 4096 data: 2
10107 085/300 Read block 2818 len 40960
10108 086/300 Read block 4148 len 40960
10109 087/300 Write block 1577 len 12288 data: 3 3 3
10110 088/300 Read block 3288 len 32768
10111 089/300 Flush
10112 090/300 Flush
10113 091/300 Read block 3417 len 4096
10114 092/300 Write block 333 len 40960 data: 3 3 4 4 4 4 3 3 4 4
10115 093/300 Read block 1949 len 24576
10116 094/300 Read block 818 len 36864
10117 095/300 Read block 168 len 16384
10118 096/300 Write block 3783 len 4096 data: 4
10119 097/300 Write block 2620 len 16384 data: 3 3 2 2
10120 098/300 Read block 1259 len 16384
10121 099/300 Read block 667 len 36864
10122 100/300 Read block 2140 len 20480
10123 101/300 Write block 4128 len 32768 data: 2 2 2 3 3 2 2 3
10124 102/300 Write block 1830 len 20480 data: 3 4 4 4 3
10125 103/300 Read block 4012 len 4096
10126 104/300 Write block 931 len 32768 data: 3 2 2 3 4 4 4 5
10127 105/300 Read block 166 len 36864
10128 106/300 Write block 3410 len 4096 data: 4
10129 107/300 Read block 4321 len 36864
10130 108/300 Read block 4315 len 16384
10131 109/300 Read block 2606 len 16384
10132 110/300 Write block 4366 len 40960 data: 4 5 5 3 3 3 3 3 3 3
10133 111/300 Read block 4041 len 20480
10134 112/300 Write block 616 len 4096 data: 3
10135 113/300 Read block 3857 len 12288
10136 114/300 Read block 1767 len 4096
10137 115/300 Write block 1436 len 40960 data: 5 4 2 2 2 2 4 5 4 4
10138 116/300 Flush
10139 117/300 Write block 1838 len 40960 data: 2 2 2 2 2 2 2 3 3 3
10140 118/300 Flush
10141 119/300 Write block 1238 len 28672 data: 2 2 2 2 2 2 2
10142 120/300 Write block 2174 len 36864 data: 2 2 2 3 3 3 3 4 4
10143 121/300 Write block 4970 len 4096 data: 2
10144 122/300 Write block 3719 len 20480 data: 2 2 2 2 2
10145 123/300 Write block 3397 len 16384 data: 3 3 3 4
10146 124/300 Write block 3911 len 36864 data: 5 5 4 4 4 3 3 3 2
10147 125/300 Flush
10148 126/300 Read block 4892 len 20480
10149 127/300 Read block 2232 len 36864
10150 128/300 Write block 2732 len 16384 data: 5 5 5 5
10151 129/300 Write block 4915 len 20480 data: 5 6 6 4 4
10152 130/300 Write block 2110 len 4096 data: 2
10153 131/300 Flush
10154 132/300 Write block 940 len 28672 data: 5 6 6 7 7 6 5
10155 133/300 Read block 1805 len 32768
10156 134/300 Write block 707 len 28672 data: 4 4 3 3 3 3 3
10157 135/300 Read block 3558 len 36864
10158 136/300 Read block 1637 len 20480
10159 137/300 Read block 25 len 20480
10160 138/300 Write block 2539 len 4096 data: 3
10161 139/300 Read block 1661 len 4096
10162 140/300 Read block 4820 len 12288
10163 141/300 Write block 2027 len 32768 data: 6 5 5 4 4 4 4 4
10164 142/300 Write block 4038 len 40960 data: 3 3 3 3 3 2 2 2 2 2
10165 143/300 Write block 3883 len 24576 data: 2 2 2 2 2 2
10166 144/300 Write block 3721 len 20480 data: 3 3 3 2 3
10167 145/300 Read block 1208 len 8192
10168 146/300 Read block 2041 len 28672
10169 147/300 Write block 3388 len 8192 data: 4 3
10170 148/300 Read block 2215 len 36864
10171 149/300 Read block 1678 len 12288
10172 150/300 Flush
10173 151/300 Write block 1109 len 12288 data: 2 2 2
10174 152/300 Read block 1573 len 24576
10175 153/300 Read block 3099 len 32768
10176 154/300 Read block 4710 len 40960
10177 155/300 Write block 2072 len 36864 data: 3 3 4 4 3 3 3 3 3
10178 156/300 Read block 2302 len 20480
10179 157/300 Read block 3727 len 32768
10180 158/300 Write block 4500 len 12288 data: 5 6 6
10181 159/300 Flush
10182 160/300 Read block 3199 len 20480
10183 161/300 Read block 4258 len 4096
10184 162/300 Read block 4273 len 8192
10185 163/300 Write block 988 len 36864 data: 2 2 2 3 3 3 2 3 3
10186 164/300 Write block 4612 len 20480 data: 4 4 4 4 4
10187 165/300 Write block 4562 len 12288 data: 2 2 3
10188 166/300 Flush
10189 167/300 Write block 1454 len 12288 data: 5 5 5
10190 168/300 Flush
10191 169/300 Write block 1414 len 24576 data: 4 5 5 5 5 3
10192 170/300 Write block 2140 len 32768 data: 3 3 3 3 5 5 6 4
10193 171/300 Write block 4773 len 12288 data: 3 3 4
10194 172/300 Read block 3473 len 32768
10195 173/300 Write block 1313 len 16384 data: 3 2 2 2
10196 174/300 Read block 2254 len 40960
10197 175/300 Write block 2170 len 24576 data: 3 2 2 2 3 3
10198 176/300 Read block 4041 len 40960
10199 177/300 Write block 3366 len 32768 data: 3 4 4 4 4 3 3 3
10200 178/300 Read block 1168 len 4096
10201 179/300 Write block 192 len 36864 data: 2 2 2 2 2 2 2 2 2
10202 180/300 Read block 2770 len 28672
10203 181/300 Read block 4756 len 36864
10204 182/300 Flush
10205 183/300 Write block 3433 len 8192 data: 4 4
10206 184/300 Read block 3461 len 12288
10207 185/300 Read block 972 len 28672
10208 186/300 Read block 4892 len 28672
10209 187/300 Read block 2340 len 12288
10210 188/300 Write block 312 len 24576 data: 3 4 4 3 3 3
10211 189/300 Write block 4130 len 40960 data: 3 4 4 3 3 4 3 3 3 3
10212 190/300 Flush
10213 191/300 Read block 1517 len 12288
10214 192/300 Write block 4799 len 8192 data: 4 3
10215 193/300 Flush
10216 194/300 Write block 3873 len 40960 data: 4 6 6 5 6 5 4 3 3 4
10217 195/300 Read block 388 len 40960
10218 196/300 Read block 1141 len 20480
10219 197/300 Read block 474 len 28672
10220 198/300 Write block 2609 len 40960 data: 3 3 3 3 3 4 4 3 3 3
10221 199/300 Write block 2560 len 28672 data: 2 3 3 3 3 3 3
10222 200/300 Read block 4607 len 8192
10223 201/300 Write block 2436 len 36864 data: 3 3 3 3 3 3 2 2 2
10224 202/300 Write block 195 len 28672 data: 3 3 3 3 3 3 2
10225 203/300 Read block 3510 len 36864
10226 204/300 Write block 3958 len 36864 data: 2 2 3 3 2 2 2 2 2
10227 205/300 Write block 1498 len 12288 data: 4 4 3
10228 206/300 Read block 1558 len 16384
10229 207/300 Read block 4459 len 12288
10230 208/300 Write block 2783 len 40960 data: 2 2 3 3 3 4 4 4 4 5
10231 209/300 Flush
10232 210/300 Read block 4081 len 24576
10233 211/300 Read block 4767 len 24576
10234 212/300 Write block 2402 len 16384 data: 4 4 4 3
10235 213/300 Write block 3188 len 36864 data: 3 3 2 2 2 3 4 4 4
10236 214/300 Write block 1336 len 32768 data: 2 2 2 2 3 3 3 3
10237 215/300 Read block 4380 len 8192
10238 216/300 Flush
10239 217/300 Read block 589 len 32768
10240 218/300 Write block 1027 len 32768 data: 2 2 2 2 2 2 2 2
10241 219/300 Flush
10242 220/300 Flush
10243 221/300 Write block 3204 len 40960 data: 3 3 2 2 2 4 5 5 6 6
10244 222/300 Write block 4032 len 24576 data: 2 2 3 3 3 3
10245 223/300 Flush
10246 224/300 Read block 4976 len 8192
10247 225/300 Flush
10248 226/300 Read block 4411 len 32768
10249 227/300 Read block 1938 len 20480
10250 228/300 Flush
10251 229/300 Write block 314 len 28672 data: 5 4 4 4 2 2 2
10252 230/300 Write block 3992 len 20480 data: 3 3 3 3 4
10253 231/300 Write block 3876 len 32768 data: 6 7 6 5 4 4 5 3
10254 232/300 Read block 33 len 16384
10255 233/300 Read block 2766 len 16384
10256 234/300 Write block 3412 len 32768 data: 4 4 4 4 4 4 5 4
10257 235/300 Read block 4934 len 24576
10258 236/300 Write block 2158 len 16384 data: 3 2 2 3
10259 237/300 Read block 2218 len 40960
10260 238/300 Read block 1810 len 28672
10261 239/300 Write block 1532 len 8192 data: 4 6
10262 240/300 Read block 1678 len 24576
10263 241/300 Write block 3382 len 12288 data: 2 2 4
10264 242/300 Read block 4399 len 28672
10265 243/300 Write block 3856 len 8192 data: 3 3
10266 244/300 Write block 3378 len 32768 data: 2 2 3 3 3 3 5 4
10267 245/300 Flush
10268 246/300 Read block 2446 len 40960
10269 247/300 Write block 265 len 12288 data: 4 3 3
10270 248/300 Read block 1760 len 36864
10271 249/300 Read block 311 len 36864
10272 250/300 Write block 4464 len 16384 data: 4 4 3 2
10273 251/300 Read block 2835 len 36864
10274 252/300 Flush
10275 253/300 Read block 2326 len 12288
10276 254/300 Write block 4554 len 4096 data: 4
10277 255/300 Write block 4775 len 4096 data: 5
10278 256/300 Read block 1160 len 16384
10279 257/300 Read block 371 len 12288
10280 258/300 Read block 4401 len 4096
10281 259/300 Write block 808 len 24576 data: 2 2 3 3 4 5
10282 260/300 Read block 1018 len 8192
10283 261/300 Flush
10284 262/300 Read block 599 len 32768
10285 263/300 Flush
10286 264/300 Write block 1592 len 24576 data: 2 4 4 5 5 5
10287 265/300 Read block 4113 len 20480
10288 266/300 Read block 4078 len 4096
10289 267/300 Read block 2377 len 20480
10290 268/300 Read block 3387 len 28672
10291 269/300 Write block 4648 len 36864 data: 3 3 3 3 2 2 2 2 2
10292 270/300 Read block 4605 len 8192
10293 271/300 Read block 1636 len 20480
10294 272/300 Write block 704 len 24576 data: 4 5 5 5 5 4
10295 273/300 Write block 3593 len 8192 data: 2 2
10296 274/300 Read block 2519 len 16384
10297 275/300 Write block 1609 len 36864 data: 3 3 2 3 3 3 3 4 4
10298 276/300 Write block 1771 len 12288 data: 4 5 5
10299 277/300 Read block 3624 len 4096
10300 278/300 Read block 1318 len 20480
10301 279/300 Read block 3362 len 20480
10302 280/300 Write block 3608 len 40960 data: 3 3 3 3 3 2 2 2 2 2
10303 281/300 Read block 1945 len 36864
10304 282/300 Write block 528 len 12288 data: 4 4 3
10305 283/300 Write block 959 len 16384 data: 3 2 2 2
10306 284/300 Write block 3526 len 24576 data: 3 3 3 3 3 2
10307 285/300 Write block 1801 len 16384 data: 2 2 3 4
10308 286/300 Read block 3809 len 40960
10309 287/300 Write block 4271 len 24576 data: 2 2 2 3 3 3
10310 288/300 Write block 4088 len 12288 data: 2 2 2
10311 289/300 Read block 1290 len 12288
10312 290/300 Read block 272 len 24576
10313 291/300 Read block 3559 len 24576
10314 292/300 Write block 1983 len 32768 data: 2 2 2 2 2 2 2 3
10315 293/300 Write block 3835 len 20480 data: 3 3 4 3 4
10316 294/300 Write block 520 len 36864 data: 5 5 6 6 6 5 4 4 5
10317 295/300 Read block 4099 len 4096
10318 296/300 Read block 4204 len 28672
10319 297/300 Read block 274 len 8192
10320 298/300 Read block 456 len 24576
10321 299/300 Write block 4353 len 28672 data: 6 6 5 5 5 4 3
10322 300/300 Read block 426 len 8192
10323 301/300 Read block 899 len 4096
10324 Main task has work to do, go find it
10325 got dsc Start(2)
10326 start 2
10327 Mar 17 22:36:51.116 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2b0ead47-269d-4b9a-bb05-96b8ec53cdf4, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
10328 Replay: started 2, returned:Ok(())
10329 ----------------------------------------------------------------
10330 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10331 States: Active Active Offline
10332 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10333 3416 Acked Read 5 Done Done Sent false
10334 3417 Acked Read 7 Done Done Sent false
10335 3418 Acked Read 3 Done Done Sent false
10336 3419 Acked Read 9 Done Done Sent false
10337 3420 Acked Read 2 Done Done Sent false
10338 3421 Acked Read 8 Done Done Sent false
10339 3422 Acked Flush 0 Done Done Sent false
10340 3423 Acked Write 9 Done Done Sent false
10341 3424 Acked Read 4 Done Done Sent false
10342 3425 Acked Read 7 Done Done Sent false
10343 3426 Acked Write 10 Done Done Sent false
10344 3427 Acked Read 4 Done Done Sent false
10345 3428 Acked Write 4 Done Done Sent false
10346 3429 Acked Read 10 Done Done Sent false
10347 3430 Acked Read 2 Done Done Sent false
10348 3431 Acked Flush 0 Done Done Sent false
10349 3432 Acked Read 5 Done Done Sent false
10350 3433 Acked Read 10 Done Done Sent false
10351 3434 Acked Read 6 Done Done Sent false
10352 3435 Acked Read 4 Done Done Sent false
10353 3436 Acked Write 2 Done Done Sent false
10354 3437 Acked Flush 0 Done Done Sent false
10355 3438 Acked Read 6 Done Done Sent false
10356 3439 Acked Read 3 Done Done Sent false
10357 3440 Acked Flush 0 Done Done Sent false
10358 3441 Acked Read 2 Done Done Sent false
10359 3442 Acked Write 6 Done Done Sent false
10360 3443 Acked Flush 0 Done Done Sent false
10361 3444 Acked Write 5 Done Done Sent false
10362 3445 Acked Flush 0 Done Done Sent false
10363 3446 Acked Read 10 Done Done Sent false
10364 3447 Acked Write 10 Done Done Sent false
10365 3448 Acked Write 2 Done Done Sent false
10366 3449 Acked Write 7 Done Done Sent false
10367 3450 Acked Flush 0 Done Done Sent false
10368 3451 Acked Flush 0 Done Done Sent false
10369 3452 Acked Write 8 Done Done Sent false
10370 3453 Acked Read 2 Done Done Sent false
10371 3454 Acked Write 3 Done Done Sent false
10372 3455 Acked Read 5 Done Done Sent false
10373 3456 Acked Flush 0 Done Done Sent false
10374 3457 Acked Write 10 Done Done Sent false
10375 3458 Acked Write 7 Done Done Sent false
10376 3459 Acked Read 3 Done Done Sent false
10377 3460 Acked Read 1 Done Done Sent false
10378 3461 Acked Read 9 Done Done Sent false
10379 3462 Acked Flush 0 Done Done Sent false
10380 3463 Acked Write 8 Done Done Sent false
10381 3464 Acked Read 8 Done Done Sent false
10382 3465 Acked Read 2 Done Done Sent false
10383 3466 Acked Read 10 Done Done Sent false
10384 3467 Acked Write 6 Done Done Sent false
10385 3468 Acked Read 5 Done Done Sent false
10386 3469 Acked Read 3 Done Done Sent false
10387 3470 Acked Read 9 Done Done Sent false
10388 3471 Acked Read 6 Done Done Sent false
10389 3472 Acked Read 7 Done Done Sent false
10390 3473 Acked Read 8 Done Done Sent false
10391 3474 Acked Read 1 Done Done Sent false
10392 3475 Acked Read 4 Done Done Sent false
10393 3476 Acked Flush 0 Done Done Sent false
10394 3477 Acked Write 3 Done Done Sent false
10395 3478 Acked Write 9 Done Done Sent false
10396 3479 Acked Write 10 Done Done Sent false
10397 3480 Acked Write 6 Done Done Sent false
10398 3481 Acked Write 2 Done Done Sent false
10399 3482 Acked Write 5 Done Done Sent false
10400 3483 Acked Write 8 Done Done Sent false
10401 3484 Acked Write 8 Done Done Sent false
10402 3485 Acked Write 9 Done Done Sent false
10403 3486 Acked Read 10 Done Done Sent false
10404 3487 Acked Read 2 Done Done Sent false
10405 3488 Acked Read 2 Done Done Sent false
10406 3489 Acked Read 7 Done Done Sent false
10407 3490 Acked Read 7 Done Done Sent false
10408 3491 Acked Read 3 Done Done Sent false
10409 3492 Acked Write 3 Done Done Sent false
10410 3493 Acked Read 8 Done Done Sent false
10411 3494 Acked Flush 0 Done Done Sent false
10412 3495 Acked Read 1 Done Done Sent false
10413 3496 Acked Read 4 Done Done Sent false
10414 3497 Acked Read 10 Done Done Sent false
10415 3498 Acked Write 8 Done Done Sent false
10416 3499 Acked Write 1 Done Done Sent false
10417 3500 Acked Read 10 Done Done Sent false
10418 3501 Acked Read 10 Done Done Sent false
10419 3502 Acked Write 3 Done Done Sent false
10420 3503 Acked Read 8 Done Done Sent false
10421 3504 Acked Flush 0 Done Done Sent false
10422 3505 Acked Flush 0 Done Done Sent false
10423 3506 Acked Read 1 Done Done Sent false
10424 3507 Acked Write 10 Done Done Sent false
10425 3508 Acked Read 6 Done Done Sent false
10426 3509 Acked Read 9 Done Done Sent false
10427 3510 Acked Read 4 Done Done Sent false
10428 3511 Acked Write 1 Done Done Sent false
10429 3512 Acked Write 4 Done Done Sent false
10430 3513 Acked Read 4 Done Done Sent false
10431 3514 Acked Read 9 Done Done Sent false
10432 3515 Acked Read 5 Done Done Sent false
10433 3516 Acked Write 8 Done Done Sent false
10434 3517 Acked Write 5 Done Done Sent false
10435 3518 Acked Read 1 Done Done Sent false
10436 3519 Acked Write 8 Done Done Sent false
10437 3520 Acked Read 9 Done Done Sent false
10438 3521 Acked Write 1 Done Done Sent false
10439 3522 Acked Read 9 Done Done Sent false
10440 3523 Acked Read 4 Done Done Sent false
10441 3524 Acked Read 4 Done Done Sent false
10442 3525 Acked Write 10 Done Done Sent false
10443 3526 Acked Read 5 Done Done Sent false
10444 3527 Acked Write 1 Done Done Sent false
10445 3528 Acked Read 3 Done Done Sent false
10446 3529 Acked Read 1 Done Done Sent false
10447 3530 Acked Write 10 Done Done Sent false
10448 3531 Acked Flush 0 Done Done Sent false
10449 3532 Acked Write 10 Done Done Sent false
10450 3533 Acked Flush 0 Done Done Sent false
10451 3534 Acked Write 7 Done Done Sent false
10452 3535 Acked Write 9 Done Done Sent false
10453 3536 Acked Write 1 Done Done Sent false
10454 3537 Acked Write 5 Done Done Sent false
10455 3538 Acked Write 4 Done Done Sent false
10456 3539 Acked Write 9 Done Done Sent false
10457 3540 Acked Flush 0 Done Done Sent false
10458 3541 Acked Read 5 Done Done Sent false
10459 3542 Acked Read 9 Done Done Sent false
10460 3543 Acked Write 4 Done Done Sent false
10461 3544 Acked Write 5 Done Done Sent false
10462 3545 Acked Write 1 Done Done Sent false
10463 3546 Acked Flush 0 Done Done Sent false
10464 3547 Acked Write 7 Done Done Sent false
10465 3548 Acked Read 8 Done Done Sent false
10466 3549 Acked Write 7 Done Done Sent false
10467 3550 Acked Read 9 Done Done Sent false
10468 3551 Acked Read 5 Done Done Sent false
10469 3552 Acked Read 5 Done Done Sent false
10470 3553 Acked Write 1 Done Done Sent false
10471 3554 Acked Read 1 Done Done Sent false
10472 3555 Acked Read 3 Done Done Sent false
10473 3556 Acked Write 8 Done Done Sent false
10474 3557 Acked Write 10 Done Done Sent false
10475 3558 Acked Write 6 Done Done Sent false
10476 3559 Acked Write 5 Done Done Sent false
10477 3560 Acked Read 2 Done Done Sent false
10478 3561 Acked Read 7 Done Done Sent false
10479 3562 Acked Write 2 Done Done Sent false
10480 3563 Acked Read 9 Done Done Sent false
10481 3564 Acked Read 3 Done Done Sent false
10482 3565 Acked Flush 0 Done Done Sent false
10483 3566 Acked Write 3 Done Done Sent false
10484 3567 Acked Read 6 Done Done Sent false
10485 3568 Acked Read 8 Done Done Sent false
10486 3569 Acked Read 10 Done Done Sent false
10487 3570 Acked Write 9 Done Done Sent false
10488 3571 Acked Read 5 Done Done Sent false
10489 3572 Acked Read 8 Done Done Sent false
10490 3573 Acked Write 3 Done Done Sent false
10491 3574 Acked Flush 0 Done Done Sent false
10492 3575 Acked Read 5 Done Done Sent false
10493 3576 Acked Read 1 Done Done Sent false
10494 3577 Acked Read 2 Done Done Sent false
10495 3578 Acked Write 9 Done Done Sent false
10496 3579 Acked Write 5 Done Done Sent false
10497 3580 Acked Write 3 Done Done Sent false
10498 3581 Acked Flush 0 Done Done Sent false
10499 3582 Acked Write 3 Done Done Sent false
10500 3583 Acked Flush 0 Done Done Sent false
10501 3584 Acked Write 6 Done Done Sent false
10502 3585 Acked Write 8 Done Done Sent false
10503 3586 Acked Write 3 Done Done Sent false
10504 3587 Acked Read 8 Done Done Sent false
10505 3588 Acked Write 4 Done Done Sent false
10506 3589 Acked Read 10 Done Done Sent false
10507 3590 Acked Write 6 Done Done Sent false
10508 3591 Acked Read 10 Done Done Sent false
10509 3592 Acked Write 8 Done Done Sent false
10510 3593 Acked Read 1 Done Done Sent false
10511 3594 Acked Write 9 Done Done Sent false
10512 3595 Acked Read 7 Done Done Sent false
10513 3596 Acked Read 9 Done Done Sent false
10514 3597 Acked Flush 0 Done Done Sent false
10515 3598 Acked Write 2 Done Done Sent false
10516 3599 Acked Read 3 Done Done Sent false
10517 3600 Acked Read 7 Done Done Sent false
10518 3601 Acked Read 7 Done Done Sent false
10519 3602 Acked Read 3 Done Done Sent false
10520 3603 Acked Write 6 Done Done Sent false
10521 3604 Acked Write 10 Done Done Sent false
10522 3605 Acked Flush 0 Done Done Sent false
10523 3606 Acked Read 3 Done Done Sent false
10524 3607 Acked Write 2 Done Done Sent false
10525 3608 Acked Flush 0 Done Done Sent false
10526 3609 Acked Write 10 Done Done Sent false
10527 3610 Acked Read 10 Done Done Sent false
10528 3611 Acked Read 5 Done Done Sent false
10529 3612 Acked Read 7 Done Done Sent false
10530 3613 Acked Write 10 Done Done Sent false
10531 3614 Acked Write 7 Done Done Sent false
10532 3615 Acked Read 2 Done Done Sent false
10533 3616 Acked Write 9 Done Done Sent false
10534 3617 Acked Write 7 Done Done Sent false
10535 3618 Acked Read 9 Done Done Sent false
10536 3619 Acked Write 9 Done Done Sent false
10537 3620 Acked Write 3 Done Done Sent false
10538 3621 Acked Read 4 Done Done Sent false
10539 3622 Acked Read 3 Done Done Sent false
10540 3623 Acked Write 10 Done Done Sent false
10541 3624 Acked Flush 0 Done Done Sent false
10542 3625 Acked Read 6 Done Done Sent false
10543 3626 Acked Read 6 Done Done Sent false
10544 3627 Acked Write 4 Done Done Sent false
10545 3628 Acked Write 9 Done Done Sent false
10546 3629 Acked Write 8 Done Done Sent false
10547 3630 Acked Read 2 Done Done Sent false
10548 3631 Acked Flush 0 Done Done Sent false
10549 3632 Acked Read 8 Done Done Sent false
10550 3633 Acked Write 8 Done Done Sent false
10551 3634 Acked Flush 0 Done Done Sent false
10552 3635 Acked Flush 0 Done Done Sent false
10553 3636 Acked Write 10 Done Done Sent false
10554 3637 Acked Write 6 Done Done Sent false
10555 3638 Acked Flush 0 Done Done Sent false
10556 3639 Acked Read 2 Done Done Sent false
10557 3640 Acked Flush 0 Done Done Sent false
10558 3641 Acked Read 8 Done Done Sent false
10559 3642 Acked Read 5 Done Done Sent false
10560 3643 Acked Flush 0 Done Done Sent false
10561 3644 Acked Write 7 Done Done Sent false
10562 3645 Acked Write 5 Done Done Sent false
10563 3646 Acked Write 8 Done Done Sent false
10564 3647 Acked Read 4 Done Done Sent false
10565 3648 Acked Read 4 Done Done Sent false
10566 3649 Acked Write 8 Done Done Sent false
10567 3650 Acked Read 6 Done Done Sent false
10568 3651 Acked Write 4 Done Done Sent false
10569 3652 Acked Read 10 Done Done Sent false
10570 3653 Acked Read 7 Done Done Sent false
10571 3654 Acked Write 2 Done Done Sent false
10572 3655 Acked Read 6 Done Done Sent false
10573 3656 Acked Write 3 Done Done Sent false
10574 3657 Acked Read 7 Done Done Sent false
10575 3658 Acked Write 2 Done Done Sent false
10576 3659 Acked Write 8 Done Done Sent false
10577 3660 Acked Flush 0 Done Done Sent false
10578 3661 Acked Read 10 Done Done Sent false
10579 3662 Acked Write 3 Done Done Sent false
10580 3663 Acked Read 9 Done Done Sent false
10581 3664 Acked Read 9 Done Done Sent false
10582 3665 Acked Write 4 Done Done Sent false
10583 3666 Acked Read 9 Done Done Sent false
10584 3667 Acked Flush 0 Done Done Sent false
10585 3668 Acked Read 3 Done Done Sent false
10586 3669 Acked Write 1 Done Done Sent false
10587 3670 Acked Write 1 Done Done Sent false
10588 3671 Acked Read 4 Done Done Sent false
10589 3672 Acked Read 3 Done Done Sent false
10590 3673 Acked Read 1 Done Done Sent false
10591 3674 Acked Write 6 Done Done Sent false
10592 3675 Acked Read 2 Done Done Sent false
10593 3676 Acked Flush 0 Done Done Sent false
10594 3677 Acked Read 8 Done Done Sent false
10595 3678 Acked Flush 0 Done Done Sent false
10596 3679 Acked Write 6 Done Done Sent false
10597 3680 Acked Read 5 Done Done Sent false
10598 3681 Acked Read 1 Done Done Sent false
10599 3682 Acked Read 5 Done Done Sent false
10600 3683 Acked Read 7 Done Done Sent false
10601 3684 Acked Write 9 Done Done Sent false
10602 3685 Acked Read 2 Done Done Sent false
10603 3686 Acked Read 5 Done Done Sent false
10604 3687 Acked Write 6 Done Done Sent false
10605 3688 Acked Write 2 Done Done Sent false
10606 3689 Acked Read 4 Done Done Sent false
10607 3690 Acked Write 9 Done Done Sent false
10608 3691 Acked Write 3 Done Done Sent false
10609 3692 Acked Read 1 Done Done Sent false
10610 3693 Acked Read 5 Done Done Sent false
10611 3694 Acked Read 5 Done Done Sent false
10612 3695 Acked Write 10 Done Done Sent false
10613 3696 Acked Read 9 Done Done Sent false
10614 3697 Acked Write 3 Done Done Sent false
10615 3698 Acked Write 4 Done Done Sent false
10616 3699 Acked Write 6 Done Done Sent false
10617 3700 Acked Write 4 Done Done Sent false
10618 3701 Acked Read 10 Done Done Sent false
10619 3702 Acked Write 6 Done Done Sent false
10620 3703 Acked Write 3 Done Done Sent false
10621 3704 Acked Read 3 Done Done Sent false
10622 3705 Acked Read 6 Done Done Sent false
10623 3706 Acked Read 6 Done Done Sent false
10624 3707 Acked Write 8 Done Done Sent false
10625 3708 Acked Write 5 Done Done Sent false
10626 3709 Acked Write 9 Done Done Sent false
10627 3710 Acked Read 1 Done Done Sent false
10628 3711 Acked Read 7 Done Done Sent false
10629 3712 Acked Read 2 Done Done Sent false
10630 3713 Acked Read 6 Done Done Sent false
10631 3714 Acked Write 7 Done Done Sent false
10632 3715 Acked Read 2 Done Done Sent false
10633 3716 Acked Read 1 Done Done Sent false
10634 STATES DS:0 DS:1 DS:2 TOTAL
10635 Sent 0 0 301 301
10636 Done 301 301 0 602
10637 Skipped 0 0 0 0
10638 Error 0 0 0 0
10639 Last Flush: Some(JobId(3678)) Some(JobId(3678)) Some(JobId(3415))
10640 Downstairs last five completed: 3415 3414 3413 3412 3411
10641 Upstairs last five completed: 3716 3715 3714 3713 3712
10642 CLIENT: Up:0 ds:301 act:2
106432025-03-17T22:36:51.618ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
106442025-03-17T22:36:52.529ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
10645 [8830] got start action
10646 [8830] I am going to restart
10647 Starting downstairs at port 8830
10648 Make output file at "/tmp/dsc/downstairs-8830.txt"
10649 [8830][2] reports Starting
10650 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217)
10651 [8830][2] reports Running
106522025-03-17T22:36:53.616ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
106532025-03-17T22:36:54.532ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
10654 ----------------------------------------------------------------
10655 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10656 States: Active Active Offline
10657 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10658 3416 Acked Read 5 Done Done Sent false
10659 3417 Acked Read 7 Done Done Sent false
10660 3418 Acked Read 3 Done Done Sent false
10661 3419 Acked Read 9 Done Done Sent false
10662 3420 Acked Read 2 Done Done Sent false
10663 3421 Acked Read 8 Done Done Sent false
10664 3422 Acked Flush 0 Done Done Sent false
10665 3423 Acked Write 9 Done Done Sent false
10666 3424 Acked Read 4 Done Done Sent false
10667 3425 Acked Read 7 Done Done Sent false
10668 3426 Acked Write 10 Done Done Sent false
10669 3427 Acked Read 4 Done Done Sent false
10670 3428 Acked Write 4 Done Done Sent false
10671 3429 Acked Read 10 Done Done Sent false
10672 3430 Acked Read 2 Done Done Sent false
10673 3431 Acked Flush 0 Done Done Sent false
10674 3432 Acked Read 5 Done Done Sent false
10675 3433 Acked Read 10 Done Done Sent false
10676 3434 Acked Read 6 Done Done Sent false
10677 3435 Acked Read 4 Done Done Sent false
10678 3436 Acked Write 2 Done Done Sent false
10679 3437 Acked Flush 0 Done Done Sent false
10680 3438 Acked Read 6 Done Done Sent false
10681 3439 Acked Read 3 Done Done Sent false
10682 3440 Acked Flush 0 Done Done Sent false
10683 3441 Acked Read 2 Done Done Sent false
10684 3442 Acked Write 6 Done Done Sent false
10685 3443 Acked Flush 0 Done Done Sent false
10686 3444 Acked Write 5 Done Done Sent false
10687 3445 Acked Flush 0 Done Done Sent false
10688 3446 Acked Read 10 Done Done Sent false
10689 3447 Acked Write 10 Done Done Sent false
10690 3448 Acked Write 2 Done Done Sent false
10691 3449 Acked Write 7 Done Done Sent false
10692 3450 Acked Flush 0 Done Done Sent false
10693 3451 Acked Flush 0 Done Done Sent false
10694 3452 Acked Write 8 Done Done Sent false
10695 3453 Acked Read 2 Done Done Sent false
10696 3454 Acked Write 3 Done Done Sent false
10697 3455 Acked Read 5 Done Done Sent false
10698 3456 Acked Flush 0 Done Done Sent false
10699 3457 Acked Write 10 Done Done Sent false
10700 3458 Acked Write 7 Done Done Sent false
10701 3459 Acked Read 3 Done Done Sent false
10702 3460 Acked Read 1 Done Done Sent false
10703 3461 Acked Read 9 Done Done Sent false
10704 3462 Acked Flush 0 Done Done Sent false
10705 3463 Acked Write 8 Done Done Sent false
10706 3464 Acked Read 8 Done Done Sent false
10707 3465 Acked Read 2 Done Done Sent false
10708 3466 Acked Read 10 Done Done Sent false
10709 3467 Acked Write 6 Done Done Sent false
10710 3468 Acked Read 5 Done Done Sent false
10711 3469 Acked Read 3 Done Done Sent false
10712 3470 Acked Read 9 Done Done Sent false
10713 3471 Acked Read 6 Done Done Sent false
10714 3472 Acked Read 7 Done Done Sent false
10715 3473 Acked Read 8 Done Done Sent false
10716 3474 Acked Read 1 Done Done Sent false
10717 3475 Acked Read 4 Done Done Sent false
10718 3476 Acked Flush 0 Done Done Sent false
10719 3477 Acked Write 3 Done Done Sent false
10720 3478 Acked Write 9 Done Done Sent false
10721 3479 Acked Write 10 Done Done Sent false
10722 3480 Acked Write 6 Done Done Sent false
10723 3481 Acked Write 2 Done Done Sent false
10724 3482 Acked Write 5 Done Done Sent false
10725 3483 Acked Write 8 Done Done Sent false
10726 3484 Acked Write 8 Done Done Sent false
10727 3485 Acked Write 9 Done Done Sent false
10728 3486 Acked Read 10 Done Done Sent false
10729 3487 Acked Read 2 Done Done Sent false
10730 3488 Acked Read 2 Done Done Sent false
10731 3489 Acked Read 7 Done Done Sent false
10732 3490 Acked Read 7 Done Done Sent false
10733 3491 Acked Read 3 Done Done Sent false
10734 3492 Acked Write 3 Done Done Sent false
10735 3493 Acked Read 8 Done Done Sent false
10736 3494 Acked Flush 0 Done Done Sent false
10737 3495 Acked Read 1 Done Done Sent false
10738 3496 Acked Read 4 Done Done Sent false
10739 3497 Acked Read 10 Done Done Sent false
10740 3498 Acked Write 8 Done Done Sent false
10741 3499 Acked Write 1 Done Done Sent false
10742 3500 Acked Read 10 Done Done Sent false
10743 3501 Acked Read 10 Done Done Sent false
10744 3502 Acked Write 3 Done Done Sent false
10745 3503 Acked Read 8 Done Done Sent false
10746 3504 Acked Flush 0 Done Done Sent false
10747 3505 Acked Flush 0 Done Done Sent false
10748 3506 Acked Read 1 Done Done Sent false
10749 3507 Acked Write 10 Done Done Sent false
10750 3508 Acked Read 6 Done Done Sent false
10751 3509 Acked Read 9 Done Done Sent false
10752 3510 Acked Read 4 Done Done Sent false
10753 3511 Acked Write 1 Done Done Sent false
10754 3512 Acked Write 4 Done Done Sent false
10755 3513 Acked Read 4 Done Done Sent false
10756 3514 Acked Read 9 Done Done Sent false
10757 3515 Acked Read 5 Done Done Sent false
10758 3516 Acked Write 8 Done Done Sent false
10759 3517 Acked Write 5 Done Done Sent false
10760 3518 Acked Read 1 Done Done Sent false
10761 3519 Acked Write 8 Done Done Sent false
10762 3520 Acked Read 9 Done Done Sent false
10763 3521 Acked Write 1 Done Done Sent false
10764 3522 Acked Read 9 Done Done Sent false
10765 3523 Acked Read 4 Done Done Sent false
10766 3524 Acked Read 4 Done Done Sent false
10767 3525 Acked Write 10 Done Done Sent false
10768 3526 Acked Read 5 Done Done Sent false
10769 3527 Acked Write 1 Done Done Sent false
10770 3528 Acked Read 3 Done Done Sent false
10771 3529 Acked Read 1 Done Done Sent false
10772 3530 Acked Write 10 Done Done Sent false
10773 3531 Acked Flush 0 Done Done Sent false
10774 3532 Acked Write 10 Done Done Sent false
10775 3533 Acked Flush 0 Done Done Sent false
10776 3534 Acked Write 7 Done Done Sent false
10777 3535 Acked Write 9 Done Done Sent false
10778 3536 Acked Write 1 Done Done Sent false
10779 3537 Acked Write 5 Done Done Sent false
10780 3538 Acked Write 4 Done Done Sent false
10781 3539 Acked Write 9 Done Done Sent false
10782 3540 Acked Flush 0 Done Done Sent false
10783 3541 Acked Read 5 Done Done Sent false
10784 3542 Acked Read 9 Done Done Sent false
10785 3543 Acked Write 4 Done Done Sent false
10786 3544 Acked Write 5 Done Done Sent false
10787 3545 Acked Write 1 Done Done Sent false
10788 3546 Acked Flush 0 Done Done Sent false
10789 3547 Acked Write 7 Done Done Sent false
10790 3548 Acked Read 8 Done Done Sent false
10791 3549 Acked Write 7 Done Done Sent false
10792 3550 Acked Read 9 Done Done Sent false
10793 3551 Acked Read 5 Done Done Sent false
10794 3552 Acked Read 5 Done Done Sent false
10795 3553 Acked Write 1 Done Done Sent false
10796 3554 Acked Read 1 Done Done Sent false
10797 3555 Acked Read 3 Done Done Sent false
10798 3556 Acked Write 8 Done Done Sent false
10799 3557 Acked Write 10 Done Done Sent false
10800 3558 Acked Write 6 Done Done Sent false
10801 3559 Acked Write 5 Done Done Sent false
10802 3560 Acked Read 2 Done Done Sent false
10803 3561 Acked Read 7 Done Done Sent false
10804 3562 Acked Write 2 Done Done Sent false
10805 3563 Acked Read 9 Done Done Sent false
10806 3564 Acked Read 3 Done Done Sent false
10807 3565 Acked Flush 0 Done Done Sent false
10808 3566 Acked Write 3 Done Done Sent false
10809 3567 Acked Read 6 Done Done Sent false
10810 3568 Acked Read 8 Done Done Sent false
10811 3569 Acked Read 10 Done Done Sent false
10812 3570 Acked Write 9 Done Done Sent false
10813 3571 Acked Read 5 Done Done Sent false
10814 3572 Acked Read 8 Done Done Sent false
10815 3573 Acked Write 3 Done Done Sent false
10816 3574 Acked Flush 0 Done Done Sent false
10817 3575 Acked Read 5 Done Done Sent false
10818 3576 Acked Read 1 Done Done Sent false
10819 3577 Acked Read 2 Done Done Sent false
10820 3578 Acked Write 9 Done Done Sent false
10821 3579 Acked Write 5 Done Done Sent false
10822 3580 Acked Write 3 Done Done Sent false
10823 3581 Acked Flush 0 Done Done Sent false
10824 3582 Acked Write 3 Done Done Sent false
10825 3583 Acked Flush 0 Done Done Sent false
10826 3584 Acked Write 6 Done Done Sent false
10827 3585 Acked Write 8 Done Done Sent false
10828 3586 Acked Write 3 Done Done Sent false
10829 3587 Acked Read 8 Done Done Sent false
10830 3588 Acked Write 4 Done Done Sent false
10831 3589 Acked Read 10 Done Done Sent false
10832 3590 Acked Write 6 Done Done Sent false
10833 3591 Acked Read 10 Done Done Sent false
10834 3592 Acked Write 8 Done Done Sent false
10835 3593 Acked Read 1 Done Done Sent false
10836 3594 Acked Write 9 Done Done Sent false
10837 3595 Acked Read 7 Done Done Sent false
10838 3596 Acked Read 9 Done Done Sent false
10839 3597 Acked Flush 0 Done Done Sent false
10840 3598 Acked Write 2 Done Done Sent false
10841 3599 Acked Read 3 Done Done Sent false
10842 3600 Acked Read 7 Done Done Sent false
10843 3601 Acked Read 7 Done Done Sent false
10844 3602 Acked Read 3 Done Done Sent false
10845 3603 Acked Write 6 Done Done Sent false
10846 3604 Acked Write 10 Done Done Sent false
10847 3605 Acked Flush 0 Done Done Sent false
10848 3606 Acked Read 3 Done Done Sent false
10849 3607 Acked Write 2 Done Done Sent false
10850 3608 Acked Flush 0 Done Done Sent false
10851 3609 Acked Write 10 Done Done Sent false
10852 3610 Acked Read 10 Done Done Sent false
10853 3611 Acked Read 5 Done Done Sent false
10854 3612 Acked Read 7 Done Done Sent false
10855 3613 Acked Write 10 Done Done Sent false
10856 3614 Acked Write 7 Done Done Sent false
10857 3615 Acked Read 2 Done Done Sent false
10858 3616 Acked Write 9 Done Done Sent false
10859 3617 Acked Write 7 Done Done Sent false
10860 3618 Acked Read 9 Done Done Sent false
10861 3619 Acked Write 9 Done Done Sent false
10862 3620 Acked Write 3 Done Done Sent false
10863 3621 Acked Read 4 Done Done Sent false
10864 3622 Acked Read 3 Done Done Sent false
10865 3623 Acked Write 10 Done Done Sent false
10866 3624 Acked Flush 0 Done Done Sent false
10867 3625 Acked Read 6 Done Done Sent false
10868 3626 Acked Read 6 Done Done Sent false
10869 3627 Acked Write 4 Done Done Sent false
10870 3628 Acked Write 9 Done Done Sent false
10871 3629 Acked Write 8 Done Done Sent false
10872 3630 Acked Read 2 Done Done Sent false
10873 3631 Acked Flush 0 Done Done Sent false
10874 3632 Acked Read 8 Done Done Sent false
10875 3633 Acked Write 8 Done Done Sent false
10876 3634 Acked Flush 0 Done Done Sent false
10877 3635 Acked Flush 0 Done Done Sent false
10878 3636 Acked Write 10 Done Done Sent false
10879 3637 Acked Write 6 Done Done Sent false
10880 3638 Acked Flush 0 Done Done Sent false
10881 3639 Acked Read 2 Done Done Sent false
10882 3640 Acked Flush 0 Done Done Sent false
10883 3641 Acked Read 8 Done Done Sent false
10884 3642 Acked Read 5 Done Done Sent false
10885 3643 Acked Flush 0 Done Done Sent false
10886 3644 Acked Write 7 Done Done Sent false
10887 3645 Acked Write 5 Done Done Sent false
10888 3646 Acked Write 8 Done Done Sent false
10889 3647 Acked Read 4 Done Done Sent false
10890 3648 Acked Read 4 Done Done Sent false
10891 3649 Acked Write 8 Done Done Sent false
10892 3650 Acked Read 6 Done Done Sent false
10893 3651 Acked Write 4 Done Done Sent false
10894 3652 Acked Read 10 Done Done Sent false
10895 3653 Acked Read 7 Done Done Sent false
10896 3654 Acked Write 2 Done Done Sent false
10897 3655 Acked Read 6 Done Done Sent false
10898 3656 Acked Write 3 Done Done Sent false
10899 3657 Acked Read 7 Done Done Sent false
10900 3658 Acked Write 2 Done Done Sent false
10901 3659 Acked Write 8 Done Done Sent false
10902 3660 Acked Flush 0 Done Done Sent false
10903 3661 Acked Read 10 Done Done Sent false
10904 3662 Acked Write 3 Done Done Sent false
10905 3663 Acked Read 9 Done Done Sent false
10906 3664 Acked Read 9 Done Done Sent false
10907 3665 Acked Write 4 Done Done Sent false
10908 3666 Acked Read 9 Done Done Sent false
10909 3667 Acked Flush 0 Done Done Sent false
10910 3668 Acked Read 3 Done Done Sent false
10911 3669 Acked Write 1 Done Done Sent false
10912 3670 Acked Write 1 Done Done Sent false
10913 3671 Acked Read 4 Done Done Sent false
10914 3672 Acked Read 3 Done Done Sent false
10915 3673 Acked Read 1 Done Done Sent false
10916 3674 Acked Write 6 Done Done Sent false
10917 3675 Acked Read 2 Done Done Sent false
10918 3676 Acked Flush 0 Done Done Sent false
10919 3677 Acked Read 8 Done Done Sent false
10920 3678 Acked Flush 0 Done Done Sent false
10921 3679 Acked Write 6 Done Done Sent false
10922 3680 Acked Read 5 Done Done Sent false
10923 3681 Acked Read 1 Done Done Sent false
10924 3682 Acked Read 5 Done Done Sent false
10925 3683 Acked Read 7 Done Done Sent false
10926 3684 Acked Write 9 Done Done Sent false
10927 3685 Acked Read 2 Done Done Sent false
10928 3686 Acked Read 5 Done Done Sent false
10929 3687 Acked Write 6 Done Done Sent false
10930 3688 Acked Write 2 Done Done Sent false
10931 3689 Acked Read 4 Done Done Sent false
10932 3690 Acked Write 9 Done Done Sent false
10933 3691 Acked Write 3 Done Done Sent false
10934 3692 Acked Read 1 Done Done Sent false
10935 3693 Acked Read 5 Done Done Sent false
10936 3694 Acked Read 5 Done Done Sent false
10937 3695 Acked Write 10 Done Done Sent false
10938 3696 Acked Read 9 Done Done Sent false
10939 3697 Acked Write 3 Done Done Sent false
10940 3698 Acked Write 4 Done Done Sent false
10941 3699 Acked Write 6 Done Done Sent false
10942 3700 Acked Write 4 Done Done Sent false
10943 3701 Acked Read 10 Done Done Sent false
10944 3702 Acked Write 6 Done Done Sent false
10945 3703 Acked Write 3 Done Done Sent false
10946 3704 Acked Read 3 Done Done Sent false
10947 3705 Acked Read 6 Done Done Sent false
10948 3706 Acked Read 6 Done Done Sent false
10949 3707 Acked Write 8 Done Done Sent false
10950 3708 Acked Write 5 Done Done Sent false
10951 3709 Acked Write 9 Done Done Sent false
10952 3710 Acked Read 1 Done Done Sent false
10953 3711 Acked Read 7 Done Done Sent false
10954 3712 Acked Read 2 Done Done Sent false
10955 3713 Acked Read 6 Done Done Sent false
10956 3714 Acked Write 7 Done Done Sent false
10957 3715 Acked Read 2 Done Done Sent false
10958 3716 Acked Read 1 Done Done Sent false
10959 STATES DS:0 DS:1 DS:2 TOTAL
10960 Sent 0 0 301 301
10961 Done 301 301 0 602
10962 Skipped 0 0 0 0
10963 Error 0 0 0 0
10964 Last Flush: Some(JobId(3678)) Some(JobId(3678)) Some(JobId(3415))
10965 Downstairs last five completed: 3415 3414 3413 3412 3411
10966 Upstairs last five completed: 3716 3715 3714 3713 3712
10967 CLIENT: Up:0 ds:301 act:2
109682025-03-17T22:36:55.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109692025-03-17T22:36:56.536ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109702025-03-17T22:36:56.872ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109712025-03-17T22:36:56.873ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51876) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109722025-03-17T22:36:56.874ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109732025-03-17T22:36:56.874ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109742025-03-17T22:36:56.874ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109752025-03-17T22:36:56.874ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109762025-03-17T22:36:56.874ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109772025-03-17T22:36:56.874ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109782025-03-17T22:36:56.874ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
10979 ----------------------------------------------------------------
10980 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10981 Downstairs last five completed: 3717 3716 3715 3714 3713
10982 Upstairs last five completed: 3717 3716 3715 3714 3713
10983 CLIENT: Up:0 ds:0 act:3
10984 Replay: All jobs finished
10985 Main task has work to do, go find it
10986 got dsc Stop(2)
10987 stop 2
10988 Mar 17 22:36:59.121 INFO request completed[8830] Got stop action so:false kr:false
10989 , latency_us: 367, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2e9651f7-6379-4295-a3d0-6b35ceed3946, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
10990 Mar 17 22:36:59.122 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/2, method: GET, req_id: e179c199-02bd-4d16-be4f-c6e762cec2ff, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
109912025-03-17T22:36:59.122ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109922025-03-17T22:36:59.122ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109932025-03-17T22:36:59.122ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
109942025-03-17T22:36:59.122ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
10995 [8830] Exited with: ExitStatus(unix_wait_status(9))
10996 [8830][2] reports Exit
10997 Mar 17 22:37:03.124 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8451b58f-3eb4-4eb6-a438-1536592aeb28, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
10998 001/300 Read block 3128 len 40960
10999 002/300 Read block 4829 len 40960
11000 003/300 Write block 412 len 40960 data: 2 2 4 6 5 5 5 5 5 4
11001 004/300 Read block 4221 len 8192
11002 005/300 Write block 2575 len 28672 data: 4 5 5 5 5 5 4
11003 006/300 Write block 726 len 4096 data: 4
11004 007/300 Write block 555 len 8192 data: 3 3
11005 008/300 Read block 2780 len 32768
11006 009/300 Write block 3177 len 20480 data: 5 6 6 6 6
11007 010/300 Read block 1785 len 12288
11008 011/300 Flush
11009 012/300 Write block 639 len 36864 data: 3 2 2 2 3 3 4 4 2
11010 013/300 Read block 4842 len 8192
11011 014/300 Write block 3643 len 4096 data: 3
11012 015/300 Read block 4583 len 32768
11013 016/300 Read block 2496 len 36864
11014 017/300 Flush
11015 018/300 Write block 404 len 20480 data: 4 5 4 3 5
11016 019/300 Write block 4773 len 12288 data: 4 4 6
11017 020/300 Read block 3603 len 20480
11018 021/300 Read block 57 len 36864
11019 022/300 Write block 439 len 24576 data: 2 2 2 2 3 3
11020 023/300 Write block 3090 len 32768 data: 2 3 3 3 3 3 3 3
11021 024/300 Read block 219 len 24576
11022 025/300 Read block 39 len 40960
11023 026/300 Write block 579 len 40960 data: 2 2 2 2 2 2 2 2 2 2
11024 027/300 Read block 1170 len 4096
11025 028/300 Flush
11026 029/300 Read block 3323 len 32768
11027 030/300 Write block 3735 len 20480 data: 4 3 3 3 2
11028 031/300 Write block 3921 len 28672 data: 2 2 2 2 2 3 3
11029 032/300 Flush
11030 033/300 Write block 3962 len 20480 data: 3 3 3 3 3
11031 034/300 Write block 3870 len 28672 data: 2 3 3 5 7 7 7
11032 035/300 Write block 2407 len 24576 data: 3 2 2 2 2 2
11033 036/300 Write block 2193 len 16384 data: 3 2 2 2
11034 037/300 Read block 1209 len 20480
11035 038/300 Write block 693 len 8192 data: 2 2
11036 039/300 Write block 2293 len 32768 data: 3 4 5 4 3 3 3 3
11037 040/300 Write block 157 len 12288 data: 3 3 3
11038 041/300 Read block 2521 len 36864
11039 042/300 Read block 1429 len 12288
11040 043/300 Write block 3209 len 28672 data: 5 6 6 7 7 5 5
11041 044/300 Write block 1060 len 8192 data: 2 2
11042 045/300 Write block 1368 len 4096 data: 3
11043 046/300 Read block 2539 len 20480
11044 047/300 Read block 2789 len 8192
11045 048/300 Read block 3272 len 32768
11046 049/300 Write block 2749 len 8192 data: 4 4
11047 050/300 Read block 3904 len 32768
11048 051/300 Read block 4789 len 4096
11049 052/300 Write block 2845 len 4096 data: 2
11050 053/300 Read block 3921 len 40960
11051 054/300 Write block 2881 len 20480 data: 3 3 2 2 2
11052 055/300 Write block 3999 len 32768 data: 3 3 3 3 3 3 2 2
11053 056/300 Read block 4082 len 36864
11054 057/300 Read block 679 len 20480
11055 058/300 Write block 4012 len 40960 data: 3 3 3 3 3 4 3 3 3 3
11056 059/300 Read block 3317 len 36864
11057 060/300 Read block 4053 len 32768
11058 061/300 Read block 4015 len 16384
11059 062/300 Write block 3985 len 28672 data: 2 3 4 4 3 3 3
11060 063/300 Write block 2714 len 4096 data: 2
11061 064/300 Read block 2502 len 24576
11062 065/300 Write block 1725 len 40960 data: 3 3 3 3 3 3 3 3 3 3
11063 066/300 Flush
11064 067/300 Flush
11065 068/300 Flush
11066 069/300 Write block 3881 len 36864 data: 5 6 4 3 3 3 3 3 2
11067 070/300 Read block 1722 len 24576
11068 071/300 Write block 728 len 36864 data: 4 4 3 2 2 3 3 3 3
11069 072/300 Write block 4005 len 8192 data: 3 3
11070 073/300 Write block 4952 len 16384 data: 3 4 4 3
11071 074/300 Write block 1299 len 4096 data: 2
11072 075/300 Read block 1566 len 20480
11073 076/300 Write block 4876 len 4096 data: 2
11074 077/300 Write block 378 len 16384 data: 2 2 3 3
11075 078/300 Write block 3551 len 20480 data: 3 3 3 4 4
11076 079/300 Flush
11077 080/300 Write block 1259 len 36864 data: 3 3 4 5 3 3 3 3 3
11078 081/300 Write block 2363 len 4096 data: 2
11079 082/300 Read block 2007 len 12288
11080 083/300 Write block 3987 len 24576 data: 5 5 4 4 4 4
11081 084/300 Flush
11082 085/300 Write block 468 len 8192 data: 2 2
11083 086/300 Write block 4705 len 8192 data: 3 2
11084 087/300 Read block 1942 len 40960
11085 088/300 Read block 1020 len 20480
11086 089/300 Read block 4449 len 32768
11087 090/300 Read block 3941 len 4096
11088 091/300 Write block 2961 len 12288 data: 2 2 2
11089 092/300 Write block 982 len 32768 data: 5 5 5 3 2 2 3 3
11090 093/300 Write block 1663 len 20480 data: 3 3 2 2 2
11091 094/300 Read block 2607 len 32768
11092 095/300 Write block 3504 len 24576 data: 2 2 3 3 3 3
11093 096/300 Write block 291 len 40960 data: 4 4 5 5 5 5 6 4 4 4
11094 097/300 Flush
11095 098/300 Write block 4881 len 36864 data: 2 2 2 2 2 2 2 2 2
11096 099/300 Read block 3487 len 4096
11097 100/300 Read block 1368 len 36864
11098 101/300 Read block 886 len 40960
11099 102/300 Write block 1764 len 8192 data: 2 3
11100 103/300 Read block 2025 len 24576
11101 104/300 Write block 2030 len 20480 data: 5 5 5 5 5
11102 105/300 Read block 4570 len 12288
11103 106/300 Read block 138 len 36864
11104 107/300 Write block 1773 len 12288 data: 6 4 3
11105 108/300 Flush
11106 109/300 Read block 1138 len 40960
11107 110/300 Read block 3314 len 20480
11108 111/300 Read block 183 len 40960
11109 112/300 Write block 615 len 16384 data: 3 4 3 3
11110 113/300 Read block 261 len 8192
11111 114/300 Write block 1134 len 36864 data: 2 2 2 2 2 2 2 2 2
11112 115/300 Write block 174 len 28672 data: 4 5 5 5 6 4 3
11113 116/300 Write block 698 len 36864 data: 3 4 4 4 4 4 5 6 6
11114 117/300 Read block 1358 len 16384
11115 118/300 Read block 2993 len 12288
11116 119/300 Write block 2878 len 24576 data: 3 4 3 4 4 3
11117 120/300 Read block 529 len 8192
11118 121/300 Write block 142 len 24576 data: 2 2 2 2 2 2
11119 122/300 Write block 2741 len 8192 data: 4 4
11120 123/300 Read block 3293 len 12288
11121 124/300 Read block 4720 len 40960
11122 125/300 Read block 4738 len 24576
11123 126/300 Write block 2114 len 28672 data: 2 3 3 3 2 3 3
11124 127/300 Write block 4920 len 20480 data: 3 4 4 5 5
11125 128/300 Write block 3329 len 28672 data: 3 3 3 2 2 2 2
11126 129/300 Flush
11127 130/300 Write block 1313 len 16384 data: 4 3 3 3
11128 131/300 Write block 22 len 28672 data: 2 3 3 3 3 3 3
11129 132/300 Write block 4780 len 40960 data: 3 3 4 5 5 4 4 3 2 2
11130 133/300 Write block 265 len 32768 data: 5 4 4 3 3 3 3 2
11131 134/300 Read block 102 len 16384
11132 135/300 Read block 649 len 8192
11133 136/300 Flush
11134 137/300 Write block 293 len 20480 data: 6 6 6 6 7
11135 138/300 Read block 2207 len 32768
11136 139/300 Write block 2908 len 8192 data: 2 3
11137 140/300 Write block 2387 len 4096 data: 2
11138 141/300 Read block 1666 len 28672
11139 142/300 Read block 834 len 16384
11140 143/300 Read block 4720 len 16384
11141 144/300 Write block 4353 len 4096 data: 7
11142 145/300 Read block 4948 len 28672
11143 146/300 Read block 630 len 12288
11144 147/300 Read block 4647 len 24576
11145 148/300 Read block 3684 len 28672
11146 149/300 Read block 4490 len 8192
11147 150/300 Read block 2195 len 32768
11148 151/300 Read block 2317 len 36864
11149 152/300 Write block 4917 len 20480 data: 7 5 5 4 5
11150 153/300 Read block 3352 len 20480
11151 154/300 Read block 512 len 40960
11152 155/300 Read block 1598 len 8192
11153 156/300 Write block 1720 len 28672 data: 3 5 3 3 3 4 4
11154 157/300 Write block 4924 len 32768 data: 6 4 4 4 5 4 4 4
11155 158/300 Write block 1052 len 32768 data: 2 2 2 2 2 2 2 2
11156 159/300 Write block 1660 len 20480 data: 4 3 3 4 4
11157 160/300 Write block 2909 len 36864 data: 4 3 4 4 4 3 3 3 4
11158 161/300 Write block 3658 len 4096 data: 3
11159 162/300 Read block 3678 len 8192
11160 163/300 Write block 362 len 36864 data: 2 2 2 2 2 2 2 3 3
11161 164/300 Read block 4263 len 8192
11162 165/300 Read block 1955 len 32768
11163 166/300 Write block 4533 len 20480 data: 4 2 3 2 2
11164 167/300 Flush
11165 168/300 Write block 315 len 8192 data: 5 5
11166 169/300 Read block 132 len 28672
11167 170/300 Read block 2864 len 12288
11168 171/300 Write block 2655 len 32768 data: 2 2 2 2 2 2 2 3
11169 172/300 Write block 3693 len 4096 data: 2
11170 173/300 Read block 4107 len 24576
11171 174/300 Read block 3097 len 12288
11172 175/300 Write block 4744 len 28672 data: 4 4 4 4 4 3 3
11173 176/300 Flush
11174 177/300 Read block 4103 len 28672
11175 178/300 Read block 2072 len 12288
11176 179/300 Read block 869 len 16384
11177 180/300 Read block 966 len 12288
11178 181/300 Read block 4110 len 16384
11179 182/300 Write block 4086 len 28672 data: 2 2 3 3 3 3 3
11180 183/300 Write block 714 len 4096 data: 3
11181 184/300 Write block 2517 len 36864 data: 4 4 4 3 3 3 3 3 3
11182 185/300 Read block 3416 len 8192
11183 186/300 Read block 4379 len 36864
11184 187/300 Write block 788 len 40960 data: 2 3 3 3 3 4 2 2 2 2
11185 188/300 Read block 2521 len 24576
11186 189/300 Write block 4836 len 12288 data: 2 2 2
11187 190/300 Write block 767 len 12288 data: 4 4 4
11188 191/300 Read block 1511 len 32768
11189 192/300 Write block 484 len 28672 data: 4 3 4 4 4 3 3
11190 193/300 Write block 1512 len 24576 data: 4 4 3 3 3 2
11191 194/300 Flush
11192 195/300 Read block 3914 len 4096
11193 196/300 Write block 2176 len 40960 data: 3 4 4 4 4 5 5 4 4 4
11194 197/300 Read block 4365 len 20480
11195 198/300 Read block 4747 len 40960
11196 199/300 Write block 2825 len 16384 data: 2 2 2 2
11197 200/300 Read block 4937 len 16384
11198 201/300 Flush
11199 202/300 Read block 2236 len 28672
11200 203/300 Flush
11201 204/300 Flush
11202 205/300 Read block 413 len 20480
11203 206/300 Read block 2530 len 28672
11204 207/300 Write block 2656 len 8192 data: 3 3
11205 208/300 Write block 1479 len 32768 data: 7 7 7 7 5 3 3 3
11206 209/300 Read block 3719 len 4096
11207 210/300 Read block 1422 len 4096
11208 211/300 Write block 709 len 32768 data: 5 4 4 4 4 4 3 3
11209 212/300 Read block 787 len 16384
11210 213/300 Write block 4333 len 40960 data: 3 3 2 3 3 2 3 3 2 2
11211 214/300 Write block 3714 len 24576 data: 3 3 3 2 2 3
11212 215/300 Read block 4603 len 8192
11213 216/300 Read block 1309 len 24576
11214 217/300 Write block 1361 len 16384 data: 4 4 4 4
11215 218/300 Read block 833 len 24576
11216 219/300 Write block 4638 len 40960 data: 3 4 4 4 4 3 3 3 3 3
11217 220/300 Read block 4354 len 32768
11218 221/300 Write block 2228 len 32768 data: 2 2 2 2 3 4 4 4
11219 222/300 Read block 867 len 8192
11220 223/300 Read block 2342 len 28672
11221 224/300 Read block 1750 len 40960
11222 225/300 Write block 2976 len 16384 data: 3 3 4 5
11223 226/300 Read block 4820 len 12288
11224 227/300 Read block 2879 len 16384
11225 228/300 Write block 793 len 40960 data: 5 3 3 3 3 2 2 2 2 2
11226 229/300 Write block 4006 len 32768 data: 4 2 2 3 3 3 4 4
11227 230/300 Write block 4305 len 16384 data: 5 5 4 5
11228 231/300 Read block 1202 len 40960
11229 232/300 Read block 2640 len 4096
11230 233/300 Read block 184 len 12288
11231 234/300 Read block 739 len 28672
11232 235/300 Read block 985 len 40960
11233 236/300 Write block 290 len 28672 data: 4 5 5 7 7 7 7
11234 237/300 Read block 3049 len 20480
11235 238/300 Read block 1342 len 40960
11236 239/300 Read block 1487 len 20480
11237 240/300 Flush
11238 241/300 Write block 4573 len 20480 data: 2 2 3 3 2
11239 242/300 Write block 3731 len 40960 data: 4 4 5 5 5 4 4 4 3 2
11240 243/300 Read block 4217 len 32768
11241 244/300 Read block 2185 len 28672
11242 245/300 Write block 4395 len 28672 data: 3 3 3 3 3 3 2
11243 246/300 Read block 1840 len 32768
11244 247/300 Read block 1537 len 40960
11245 248/300 Write block 4004 len 4096 data: 4
11246 249/300 Read block 2590 len 20480
11247 250/300 Read block 3557 len 40960
11248 251/300 Write block 4840 len 4096 data: 2
11249 252/300 Write block 4435 len 36864 data: 4 4 4 3 2 2 2 3 3
11250 253/300 Flush
11251 254/300 Write block 778 len 32768 data: 2 3 3 3 4 4 4 3
11252 255/300 Read block 550 len 32768
11253 256/300 Flush
11254 257/300 Read block 848 len 12288
11255 258/300 Write block 4630 len 36864 data: 4 4 4 5 5 5 5 4 4
11256 259/300 Read block 2744 len 40960
11257 260/300 Write block 1791 len 28672 data: 3 2 2 3 3 3 2
11258 261/300 Write block 1790 len 4096 data: 3
11259 262/300 Write block 4536 len 24576 data: 3 3 3 3 2 2
11260 263/300 Write block 4601 len 24576 data: 3 3 3 3 3 2
11261 264/300 Read block 3750 len 36864
11262 265/300 Read block 1043 len 36864
11263 266/300 Write block 1589 len 32768 data: 2 2 2 3 5 5 6 6
11264 267/300 Flush
11265 268/300 Read block 3638 len 36864
11266 269/300 Read block 2755 len 24576
11267 270/300 Read block 418 len 24576
11268 271/300 Write block 3804 len 32768 data: 2 3 3 3 3 3 4 4
11269 272/300 Read block 3026 len 36864
11270 273/300 Read block 4290 len 40960
11271 274/300 Read block 4348 len 24576
11272 275/300 Read block 4580 len 8192
11273 276/300 Write block 4142 len 20480 data: 4 4 4 4 4
11274 277/300 Flush
11275 278/300 Write block 4243 len 32768 data: 2 2 2 2 3 3 3 3
11276 279/300 Flush
11277 280/300 Write block 508 len 16384 data: 2 2 2 2
11278 281/300 Write block 772 len 28672 data: 3 3 3 3 3 2 3
11279 282/300 Write block 2077 len 16384 data: 4 4 4 4
11280 283/300 Read block 189 len 32768
11281 284/300 Read block 1470 len 32768
11282 285/300 Read block 1887 len 28672
11283 286/300 Read block 4807 len 36864
11284 287/300 Write block 3434 len 8192 data: 5 4
11285 288/300 Read block 3416 len 16384
11286 289/300 Write block 2319 len 28672 data: 2 2 2 2 2 2 2
11287 290/300 Read block 1314 len 36864
11288 291/300 Write block 4337 len 40960 data: 4 3 4 4 3 3 2 3 3 4
11289 292/300 Write block 3280 len 40960 data: 2 2 2 2 2 2 2 2 3 3
11290 293/300 Read block 1103 len 24576
11291 294/300 Write block 369 len 4096 data: 4
11292 295/300 Write block 433 len 16384 data: 3 3 2 2
11293 296/300 Write block 4493 len 8192 data: 3 3
11294 297/300 Flush
11295 298/300 Flush
11296 299/300 Flush
11297 300/300 Read block 263 len 20480
11298 301/300 Read block 2855 len 12288
11299 Main task has work to do, go find it
11300 got dsc Start(2)
11301 start 2
11302 Mar 17 22:37:03.370 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 596ac0b3-4328-4707-b1b4-7dabe6ccebf1, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
11303 Replay: started 2, returned:Ok(())
11304 ----------------------------------------------------------------
11305 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11306 States: Active Active Offline
11307 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11308 3718 Acked Read 10 Done Done Sent false
11309 3719 Acked Read 10 Done Done Sent false
11310 3720 Acked Write 10 Done Done Sent false
11311 3721 Acked Read 2 Done Done Sent false
11312 3722 Acked Write 7 Done Done Sent false
11313 3723 Acked Write 1 Done Done Sent false
11314 3724 Acked Write 2 Done Done Sent false
11315 3725 Acked Read 8 Done Done Sent false
11316 3726 Acked Write 5 Done Done Sent false
11317 3727 Acked Read 3 Done Done Sent false
11318 3728 Acked Flush 0 Done Done Sent false
11319 3729 Acked Write 9 Done Done Sent false
11320 3730 Acked Read 2 Done Done Sent false
11321 3731 Acked Write 1 Done Done Sent false
11322 3732 Acked Read 8 Done Done Sent false
11323 3733 Acked Read 9 Done Done Sent false
11324 3734 Acked Flush 0 Done Done Sent false
11325 3735 Acked Write 5 Done Done Sent false
11326 3736 Acked Write 3 Done Done Sent false
11327 3737 Acked Read 5 Done Done Sent false
11328 3738 Acked Read 9 Done Done Sent false
11329 3739 Acked Write 6 Done Done Sent false
11330 3740 Acked Write 8 Done Done Sent false
11331 3741 Acked Read 6 Done Done Sent false
11332 3742 Acked Read 10 Done Done Sent false
11333 3743 Acked Write 10 Done Done Sent false
11334 3744 Acked Read 1 Done Done Sent false
11335 3745 Acked Flush 0 Done Done Sent false
11336 3746 Acked Read 8 Done Done Sent false
11337 3747 Acked Write 5 Done Done Sent false
11338 3748 Acked Write 7 Done Done Sent false
11339 3749 Acked Flush 0 Done Done Sent false
11340 3750 Acked Write 5 Done Done Sent false
11341 3751 Acked Write 7 Done Done Sent false
11342 3752 Acked Write 6 Done Done Sent false
11343 3753 Acked Write 4 Done Done Sent false
11344 3754 Acked Read 5 Done Done Sent false
11345 3755 Acked Write 2 Done Done Sent false
11346 3756 Acked Write 8 Done Done Sent false
11347 3757 Acked Write 3 Done Done Sent false
11348 3758 Acked Read 9 Done Done Sent false
11349 3759 Acked Read 3 Done Done Sent false
11350 3760 Acked Write 7 Done Done Sent false
11351 3761 Acked Write 2 Done Done Sent false
11352 3762 Acked Write 1 Done Done Sent false
11353 3763 Acked Read 5 Done Done Sent false
11354 3764 Acked Read 2 Done Done Sent false
11355 3765 Acked Read 8 Done Done Sent false
11356 3766 Acked Write 2 Done Done Sent false
11357 3767 Acked Read 8 Done Done Sent false
11358 3768 Acked Read 1 Done Done Sent false
11359 3769 Acked Write 1 Done Done Sent false
11360 3770 Acked Read 10 Done Done Sent false
11361 3771 Acked Write 5 Done Done Sent false
11362 3772 Acked Write 8 Done Done Sent false
11363 3773 Acked Read 9 Done Done Sent false
11364 3774 Acked Read 5 Done Done Sent false
11365 3775 Acked Write 10 Done Done Sent false
11366 3776 Acked Read 9 Done Done Sent false
11367 3777 Acked Read 8 Done Done Sent false
11368 3778 Acked Read 4 Done Done Sent false
11369 3779 Acked Write 7 Done Done Sent false
11370 3780 Acked Write 1 Done Done Sent false
11371 3781 Acked Read 6 Done Done Sent false
11372 3782 Acked Write 10 Done Done Sent false
11373 3783 Acked Flush 0 Done Done Sent false
11374 3784 Acked Flush 0 Done Done Sent false
11375 3785 Acked Flush 0 Done Done Sent false
11376 3786 Acked Write 9 Done Done Sent false
11377 3787 Acked Read 6 Done Done Sent false
11378 3788 Acked Write 9 Done Done Sent false
11379 3789 Acked Write 2 Done Done Sent false
11380 3790 Acked Write 4 Done Done Sent false
11381 3791 Acked Write 1 Done Done Sent false
11382 3792 Acked Read 5 Done Done Sent false
11383 3793 Acked Write 1 Done Done Sent false
11384 3794 Acked Write 4 Done Done Sent false
11385 3795 Acked Write 5 Done Done Sent false
11386 3796 Acked Flush 0 Done Done Sent false
11387 3797 Acked Write 9 Done Done Sent false
11388 3798 Acked Write 1 Done Done Sent false
11389 3799 Acked Read 3 Done Done Sent false
11390 3800 Acked Write 6 Done Done Sent false
11391 3801 Acked Flush 0 Done Done Sent false
11392 3802 Acked Write 2 Done Done Sent false
11393 3803 Acked Write 2 Done Done Sent false
11394 3804 Acked Read 10 Done Done Sent false
11395 3805 Acked Read 5 Done Done Sent false
11396 3806 Acked Read 8 Done Done Sent false
11397 3807 Acked Read 1 Done Done Sent false
11398 3808 Acked Write 3 Done Done Sent false
11399 3809 Acked Write 8 Done Done Sent false
11400 3810 Acked Write 5 Done Done Sent false
11401 3811 Acked Read 8 Done Done Sent false
11402 3812 Acked Write 6 Done Done Sent false
11403 3813 Acked Write 10 Done Done Sent false
11404 3814 Acked Flush 0 Done Done Sent false
11405 3815 Acked Write 9 Done Done Sent false
11406 3816 Acked Read 1 Done Done Sent false
11407 3817 Acked Read 9 Done Done Sent false
11408 3818 Acked Read 10 Done Done Sent false
11409 3819 Acked Write 2 Done Done Sent false
11410 3820 Acked Read 6 Done Done Sent false
11411 3821 Acked Write 5 Done Done Sent false
11412 3822 Acked Read 3 Done Done Sent false
11413 3823 Acked Read 9 Done Done Sent false
11414 3824 Acked Write 3 Done Done Sent false
11415 3825 Acked Flush 0 Done Done Sent false
11416 3826 Acked Read 10 Done Done Sent false
11417 3827 Acked Read 5 Done Done Sent false
11418 3828 Acked Read 10 Done Done Sent false
11419 3829 Acked Write 4 Done Done Sent false
11420 3830 Acked Read 2 Done Done Sent false
11421 3831 Acked Write 9 Done Done Sent false
11422 3832 Acked Write 7 Done Done Sent false
11423 3833 Acked Write 9 Done Done Sent false
11424 3834 Acked Read 4 Done Done Sent false
11425 3835 Acked Read 3 Done Done Sent false
11426 3836 Acked Write 6 Done Done Sent false
11427 3837 Acked Read 2 Done Done Sent false
11428 3838 Acked Write 6 Done Done Sent false
11429 3839 Acked Write 2 Done Done Sent false
11430 3840 Acked Read 3 Done Done Sent false
11431 3841 Acked Read 10 Done Done Sent false
11432 3842 Acked Read 6 Done Done Sent false
11433 3843 Acked Write 7 Done Done Sent false
11434 3844 Acked Write 5 Done Done Sent false
11435 3845 Acked Write 7 Done Done Sent false
11436 3846 Acked Flush 0 Done Done Sent false
11437 3847 Acked Write 4 Done Done Sent false
11438 3848 Acked Write 7 Done Done Sent false
11439 3849 Acked Write 10 Done Done Sent false
11440 3850 Acked Write 8 Done Done Sent false
11441 3851 Acked Read 4 Done Done Sent false
11442 3852 Acked Read 2 Done Done Sent false
11443 3853 Acked Flush 0 Done Done Sent false
11444 3854 Acked Write 5 Done Done Sent false
11445 3855 Acked Read 8 Done Done Sent false
11446 3856 Acked Write 2 Done Done Sent false
11447 3857 Acked Write 1 Done Done Sent false
11448 3858 Acked Read 7 Done Done Sent false
11449 3859 Acked Read 4 Done Done Sent false
11450 3860 Acked Read 4 Done Done Sent false
11451 3861 Acked Write 1 Done Done Sent false
11452 3862 Acked Read 7 Done Done Sent false
11453 3863 Acked Read 3 Done Done Sent false
11454 3864 Acked Read 6 Done Done Sent false
11455 3865 Acked Read 7 Done Done Sent false
11456 3866 Acked Read 2 Done Done Sent false
11457 3867 Acked Read 8 Done Done Sent false
11458 3868 Acked Read 9 Done Done Sent false
11459 3869 Acked Write 5 Done Done Sent false
11460 3870 Acked Read 5 Done Done Sent false
11461 3871 Acked Read 10 Done Done Sent false
11462 3872 Acked Read 2 Done Done Sent false
11463 3873 Acked Write 7 Done Done Sent false
11464 3874 Acked Write 8 Done Done Sent false
11465 3875 Acked Write 8 Done Done Sent false
11466 3876 Acked Write 5 Done Done Sent false
11467 3877 Acked Write 9 Done Done Sent false
11468 3878 Acked Write 1 Done Done Sent false
11469 3879 Acked Read 2 Done Done Sent false
11470 3880 Acked Write 9 Done Done Sent false
11471 3881 Acked Read 2 Done Done Sent false
11472 3882 Acked Read 8 Done Done Sent false
11473 3883 Acked Write 5 Done Done Sent false
11474 3884 Acked Flush 0 Done Done Sent false
11475 3885 Acked Write 2 Done Done Sent false
11476 3886 Acked Read 7 Done Done Sent false
11477 3887 Acked Read 3 Done Done Sent false
11478 3888 Acked Write 8 Done Done Sent false
11479 3889 Acked Write 1 Done Done Sent false
11480 3890 Acked Read 6 Done Done Sent false
11481 3891 Acked Read 3 Done Done Sent false
11482 3892 Acked Write 7 Done Done Sent false
11483 3893 Acked Flush 0 Done Done Sent false
11484 3894 Acked Read 7 Done Done Sent false
11485 3895 Acked Read 3 Done Done Sent false
11486 3896 Acked Read 4 Done Done Sent false
11487 3897 Acked Read 3 Done Done Sent false
11488 3898 Acked Read 4 Done Done Sent false
11489 3899 Acked Write 7 Done Done Sent false
11490 3900 Acked Write 1 Done Done Sent false
11491 3901 Acked Write 9 Done Done Sent false
11492 3902 Acked Read 2 Done Done Sent false
11493 3903 Acked Read 9 Done Done Sent false
11494 3904 Acked Write 10 Done Done Sent false
11495 3905 Acked Read 6 Done Done Sent false
11496 3906 Acked Write 3 Done Done Sent false
11497 3907 Acked Write 3 Done Done Sent false
11498 3908 Acked Read 8 Done Done Sent false
11499 3909 Acked Write 7 Done Done Sent false
11500 3910 Acked Write 6 Done Done Sent false
11501 3911 Acked Flush 0 Done Done Sent false
11502 3912 Acked Read 1 Done Done Sent false
11503 3913 Acked Write 10 Done Done Sent false
11504 3914 Acked Read 5 Done Done Sent false
11505 3915 Acked Read 10 Done Done Sent false
11506 3916 Acked Write 4 Done Done Sent false
11507 3917 Acked Read 4 Done Done Sent false
11508 3918 Acked Flush 0 Done Done Sent false
11509 3919 Acked Read 7 Done Done Sent false
11510 3920 Acked Flush 0 Done Done Sent false
11511 3921 Acked Flush 0 Done Done Sent false
11512 3922 Acked Read 5 Done Done Sent false
11513 3923 Acked Read 7 Done Done Sent false
11514 3924 Acked Write 2 Done Done Sent false
11515 3925 Acked Write 8 Done Done Sent false
11516 3926 Acked Read 1 Done Done Sent false
11517 3927 Acked Read 1 Done Done Sent false
11518 3928 Acked Write 8 Done Done Sent false
11519 3929 Acked Read 4 Done Done Sent false
11520 3930 Acked Write 10 Done Done Sent false
11521 3931 Acked Write 6 Done Done Sent false
11522 3932 Acked Read 2 Done Done Sent false
11523 3933 Acked Read 6 Done Done Sent false
11524 3934 Acked Write 4 Done Done Sent false
11525 3935 Acked Read 6 Done Done Sent false
11526 3936 Acked Write 10 Done Done Sent false
11527 3937 Acked Read 8 Done Done Sent false
11528 3938 Acked Write 8 Done Done Sent false
11529 3939 Acked Read 2 Done Done Sent false
11530 3940 Acked Read 7 Done Done Sent false
11531 3941 Acked Read 10 Done Done Sent false
11532 3942 Acked Write 4 Done Done Sent false
11533 3943 Acked Read 3 Done Done Sent false
11534 3944 Acked Read 4 Done Done Sent false
11535 3945 Acked Write 10 Done Done Sent false
11536 3946 Acked Write 8 Done Done Sent false
11537 3947 Acked Write 4 Done Done Sent false
11538 3948 Acked Read 10 Done Done Sent false
11539 3949 Acked Read 1 Done Done Sent false
11540 3950 Acked Read 3 Done Done Sent false
11541 3951 Acked Read 7 Done Done Sent false
11542 3952 Acked Read 10 Done Done Sent false
11543 3953 Acked Write 7 Done Done Sent false
11544 3954 Acked Read 5 Done Done Sent false
11545 3955 Acked Read 10 Done Done Sent false
11546 3956 Acked Read 5 Done Done Sent false
11547 3957 Acked Flush 0 Done Done Sent false
11548 3958 Acked Write 5 Done Done Sent false
11549 3959 Acked Write 10 Done Done Sent false
11550 3960 Acked Read 8 Done Done Sent false
11551 3961 Acked Read 7 Done Done Sent false
11552 3962 Acked Write 7 Done Done Sent false
11553 3963 Acked Read 8 Done Done Sent false
11554 3964 Acked Read 10 Done Done Sent false
11555 3965 Acked Write 1 Done Done Sent false
11556 3966 Acked Read 5 Done Done Sent false
11557 3967 Acked Read 10 Done Done Sent false
11558 3968 Acked Write 1 Done Done Sent false
11559 3969 Acked Write 9 Done Done Sent false
11560 3970 Acked Flush 0 Done Done Sent false
11561 3971 Acked Write 8 Done Done Sent false
11562 3972 Acked Read 8 Done Done Sent false
11563 3973 Acked Flush 0 Done Done Sent false
11564 3974 Acked Read 3 Done Done Sent false
11565 3975 Acked Write 9 Done Done Sent false
11566 3976 Acked Read 10 Done Done Sent false
11567 3977 Acked Write 7 Done Done Sent false
11568 3978 Acked Write 1 Done Done Sent false
11569 3979 Acked Write 6 Done Done Sent false
11570 3980 Acked Write 6 Done Done Sent false
11571 3981 Acked Read 9 Done Done Sent false
11572 3982 Acked Read 9 Done Done Sent false
11573 3983 Acked Write 8 Done Done Sent false
11574 3984 Acked Flush 0 Done Done Sent false
11575 3985 Acked Read 9 Done Done Sent false
11576 3986 Acked Read 6 Done Done Sent false
11577 3987 Acked Read 6 Done Done Sent false
11578 3988 Acked Write 8 Done Done Sent false
11579 3989 Acked Read 9 Done Done Sent false
11580 3990 Acked Read 10 Done Done Sent false
11581 3991 Acked Read 6 Done Done Sent false
11582 3992 Acked Read 2 Done Done Sent false
11583 3993 Acked Write 5 Done Done Sent false
11584 3994 Acked Flush 0 Done Done Sent false
11585 3995 Acked Write 8 Done Done Sent false
11586 3996 Acked Flush 0 Done Done Sent false
11587 3997 Acked Write 4 Done Done Sent false
11588 3998 Acked Write 7 Done Done Sent false
11589 3999 Acked Write 4 Done Done Sent false
11590 4000 Acked Read 8 Done Done Sent false
11591 4001 Acked Read 8 Done Done Sent false
11592 4002 Acked Read 7 Done Done Sent false
11593 4003 Acked Read 9 Done Done Sent false
11594 4004 Acked Write 2 Done Done Sent false
11595 4005 Acked Read 4 Done Done Sent false
11596 4006 Acked Write 7 Done Done Sent false
11597 4007 Acked Read 9 Done Done Sent false
11598 4008 Acked Write 10 Done Done Sent false
11599 4009 Acked Write 10 Done Done Sent false
11600 4010 Acked Read 6 Done Done Sent false
11601 4011 Acked Write 1 Done Done Sent false
11602 4012 Acked Write 4 Done Done Sent false
11603 4013 Acked Write 2 Done Done Sent false
11604 4014 Acked Flush 0 Done Done Sent false
11605 4015 Acked Flush 0 Done Done Sent false
11606 4016 Acked Flush 0 Done Done Sent false
11607 4017 Acked Read 5 Done Done Sent false
11608 4018 Acked Read 3 Done Done Sent false
11609 STATES DS:0 DS:1 DS:2 TOTAL
11610 Sent 0 0 301 301
11611 Done 301 301 0 602
11612 Skipped 0 0 0 0
11613 Error 0 0 0 0
11614 Last Flush: Some(JobId(4016)) Some(JobId(4016)) Some(JobId(3717))
11615 Downstairs last five completed: 3717 3716 3715 3714 3713
11616 Upstairs last five completed: 4018 4017 4016 4015 4014
11617 CLIENT: Up:0 ds:301 act:2
116182025-03-17T22:37:03.873ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
116192025-03-17T22:37:04.549ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
116202025-03-17T22:37:05.049ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
116212025-03-17T22:37:05.551ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
116222025-03-17T22:37:06.552ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11623 ----------------------------------------------------------------
11624 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11625 States: Active Active Offline
11626 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11627 3718 Acked Read 10 Done Done Sent false
11628 3719 Acked Read 10 Done Done Sent false
11629 3720 Acked Write 10 Done Done Sent false
11630 3721 Acked Read 2 Done Done Sent false
11631 3722 Acked Write 7 Done Done Sent false
11632 3723 Acked Write 1 Done Done Sent false
11633 3724 Acked Write 2 Done Done Sent false
11634 3725 Acked Read 8 Done Done Sent false
11635 3726 Acked Write 5 Done Done Sent false
11636 3727 Acked Read 3 Done Done Sent false
11637 3728 Acked Flush 0 Done Done Sent false
11638 3729 Acked Write 9 Done Done Sent false
11639 3730 Acked Read 2 Done Done Sent false
11640 3731 Acked Write 1 Done Done Sent false
11641 3732 Acked Read 8 Done Done Sent false
11642 3733 Acked Read 9 Done Done Sent false
11643 3734 Acked Flush 0 Done Done Sent false
11644 3735 Acked Write 5 Done Done Sent false
11645 3736 Acked Write 3 Done Done Sent false
11646 3737 Acked Read 5 Done Done Sent false
11647 3738 Acked Read 9 Done Done Sent false
11648 3739 Acked Write 6 Done Done Sent false
11649 3740 Acked Write 8 Done Done Sent false
11650 3741 Acked Read 6 Done Done Sent false
11651 3742 Acked Read 10 Done Done Sent false
11652 3743 Acked Write 10 Done Done Sent false
11653 3744 Acked Read 1 Done Done Sent false
11654 3745 Acked Flush 0 Done Done Sent false
11655 3746 Acked Read 8 Done Done Sent false
11656 3747 Acked Write 5 Done Done Sent false
11657 3748 Acked Write 7 Done Done Sent false
11658 3749 Acked Flush 0 Done Done Sent false
11659 3750 Acked Write 5 Done Done Sent false
11660 3751 Acked Write 7 Done Done Sent false
11661 3752 Acked Write 6 Done Done Sent false
11662 3753 Acked Write 4 Done Done Sent false
11663 3754 Acked Read 5 Done Done Sent false
11664 3755 Acked Write 2 Done Done Sent false
11665 3756 Acked Write 8 Done Done Sent false
11666 3757 Acked Write 3 Done Done Sent false
11667 3758 Acked Read 9 Done Done Sent false
11668 3759 Acked Read 3 Done Done Sent false
11669 3760 Acked Write 7 Done Done Sent false
11670 3761 Acked Write 2 Done Done Sent false
11671 3762 Acked Write 1 Done Done Sent false
11672 3763 Acked Read 5 Done Done Sent false
11673 3764 Acked Read 2 Done Done Sent false
11674 3765 Acked Read 8 Done Done Sent false
11675 3766 Acked Write 2 Done Done Sent false
11676 3767 Acked Read 8 Done Done Sent false
11677 3768 Acked Read 1 Done Done Sent false
11678 3769 Acked Write 1 Done Done Sent false
11679 3770 Acked Read 10 Done Done Sent false
11680 3771 Acked Write 5 Done Done Sent false
11681 3772 Acked Write 8 Done Done Sent false
11682 3773 Acked Read 9 Done Done Sent false
11683 3774 Acked Read 5 Done Done Sent false
11684 3775 Acked Write 10 Done Done Sent false
11685 3776 Acked Read 9 Done Done Sent false
11686 3777 Acked Read 8 Done Done Sent false
11687 3778 Acked Read 4 Done Done Sent false
11688 3779 Acked Write 7 Done Done Sent false
11689 3780 Acked Write 1 Done Done Sent false
11690 3781 Acked Read 6 Done Done Sent false
11691 3782 Acked Write 10 Done Done Sent false
11692 3783 Acked Flush 0 Done Done Sent false
11693 3784 Acked Flush 0 Done Done Sent false
11694 3785 Acked Flush 0 Done Done Sent false
11695 3786 Acked Write 9 Done Done Sent false
11696 3787 Acked Read 6 Done Done Sent false
11697 3788 Acked Write 9 Done Done Sent false
11698 3789 Acked Write 2 Done Done Sent false
11699 3790 Acked Write 4 Done Done Sent false
11700 3791 Acked Write 1 Done Done Sent false
11701 3792 Acked Read 5 Done Done Sent false
11702 3793 Acked Write 1 Done Done Sent false
11703 3794 Acked Write 4 Done Done Sent false
11704 3795 Acked Write 5 Done Done Sent false
11705 3796 Acked Flush 0 Done Done Sent false
11706 3797 Acked Write 9 Done Done Sent false
11707 3798 Acked Write 1 Done Done Sent false
11708 3799 Acked Read 3 Done Done Sent false
11709 3800 Acked Write 6 Done Done Sent false
11710 3801 Acked Flush 0 Done Done Sent false
11711 3802 Acked Write 2 Done Done Sent false
11712 3803 Acked Write 2 Done Done Sent false
11713 3804 Acked Read 10 Done Done Sent false
11714 3805 Acked Read 5 Done Done Sent false
11715 3806 Acked Read 8 Done Done Sent false
11716 3807 Acked Read 1 Done Done Sent false
11717 3808 Acked Write 3 Done Done Sent false
11718 3809 Acked Write 8 Done Done Sent false
11719 3810 Acked Write 5 Done Done Sent false
11720 3811 Acked Read 8 Done Done Sent false
11721 3812 Acked Write 6 Done Done Sent false
11722 3813 Acked Write 10 Done Done Sent false
11723 3814 Acked Flush 0 Done Done Sent false
11724 3815 Acked Write 9 Done Done Sent false
11725 3816 Acked Read 1 Done Done Sent false
11726 3817 Acked Read 9 Done Done Sent false
11727 3818 Acked Read 10 Done Done Sent false
11728 3819 Acked Write 2 Done Done Sent false
11729 3820 Acked Read 6 Done Done Sent false
11730 3821 Acked Write 5 Done Done Sent false
11731 3822 Acked Read 3 Done Done Sent false
11732 3823 Acked Read 9 Done Done Sent false
11733 3824 Acked Write 3 Done Done Sent false
11734 3825 Acked Flush 0 Done Done Sent false
11735 3826 Acked Read 10 Done Done Sent false
11736 3827 Acked Read 5 Done Done Sent false
11737 3828 Acked Read 10 Done Done Sent false
11738 3829 Acked Write 4 Done Done Sent false
11739 3830 Acked Read 2 Done Done Sent false
11740 3831 Acked Write 9 Done Done Sent false
11741 3832 Acked Write 7 Done Done Sent false
11742 3833 Acked Write 9 Done Done Sent false
11743 3834 Acked Read 4 Done Done Sent false
11744 3835 Acked Read 3 Done Done Sent false
11745 3836 Acked Write 6 Done Done Sent false
11746 3837 Acked Read 2 Done Done Sent false
11747 3838 Acked Write 6 Done Done Sent false
11748 3839 Acked Write 2 Done Done Sent false
11749 3840 Acked Read 3 Done Done Sent false
11750 3841 Acked Read 10 Done Done Sent false
11751 3842 Acked Read 6 Done Done Sent false
11752 3843 Acked Write 7 Done Done Sent false
11753 3844 Acked Write 5 Done Done Sent false
11754 3845 Acked Write 7 Done Done Sent false
11755 3846 Acked Flush 0 Done Done Sent false
11756 3847 Acked Write 4 Done Done Sent false
11757 3848 Acked Write 7 Done Done Sent false
11758 3849 Acked Write 10 Done Done Sent false
11759 3850 Acked Write 8 Done Done Sent false
11760 3851 Acked Read 4 Done Done Sent false
11761 3852 Acked Read 2 Done Done Sent false
11762 3853 Acked Flush 0 Done Done Sent false
11763 3854 Acked Write 5 Done Done Sent false
11764 3855 Acked Read 8 Done Done Sent false
11765 3856 Acked Write 2 Done Done Sent false
11766 3857 Acked Write 1 Done Done Sent false
11767 3858 Acked Read 7 Done Done Sent false
11768 3859 Acked Read 4 Done Done Sent false
11769 3860 Acked Read 4 Done Done Sent false
11770 3861 Acked Write 1 Done Done Sent false
11771 3862 Acked Read 7 Done Done Sent false
11772 3863 Acked Read 3 Done Done Sent false
11773 3864 Acked Read 6 Done Done Sent false
11774 3865 Acked Read 7 Done Done Sent false
11775 3866 Acked Read 2 Done Done Sent false
11776 3867 Acked Read 8 Done Done Sent false
11777 3868 Acked Read 9 Done Done Sent false
11778 3869 Acked Write 5 Done Done Sent false
11779 3870 Acked Read 5 Done Done Sent false
11780 3871 Acked Read 10 Done Done Sent false
11781 3872 Acked Read 2 Done Done Sent false
11782 3873 Acked Write 7 Done Done Sent false
11783 3874 Acked Write 8 Done Done Sent false
11784 3875 Acked Write 8 Done Done Sent false
11785 3876 Acked Write 5 Done Done Sent false
11786 3877 Acked Write 9 Done Done Sent false
11787 3878 Acked Write 1 Done Done Sent false
11788 3879 Acked Read 2 Done Done Sent false
11789 3880 Acked Write 9 Done Done Sent false
11790 3881 Acked Read 2 Done Done Sent false
11791 3882 Acked Read 8 Done Done Sent false
11792 3883 Acked Write 5 Done Done Sent false
11793 3884 Acked Flush 0 Done Done Sent false
11794 3885 Acked Write 2 Done Done Sent false
11795 3886 Acked Read 7 Done Done Sent false
11796 3887 Acked Read 3 Done Done Sent false
11797 3888 Acked Write 8 Done Done Sent false
11798 3889 Acked Write 1 Done Done Sent false
11799 3890 Acked Read 6 Done Done Sent false
11800 3891 Acked Read 3 Done Done Sent false
11801 3892 Acked Write 7 Done Done Sent false
11802 3893 Acked Flush 0 Done Done Sent false
11803 3894 Acked Read 7 Done Done Sent false
11804 3895 Acked Read 3 Done Done Sent false
11805 3896 Acked Read 4 Done Done Sent false
11806 3897 Acked Read 3 Done Done Sent false
11807 3898 Acked Read 4 Done Done Sent false
11808 3899 Acked Write 7 Done Done Sent false
11809 3900 Acked Write 1 Done Done Sent false
11810 3901 Acked Write 9 Done Done Sent false
11811 3902 Acked Read 2 Done Done Sent false
11812 3903 Acked Read 9 Done Done Sent false
11813 3904 Acked Write 10 Done Done Sent false
11814 3905 Acked Read 6 Done Done Sent false
11815 3906 Acked Write 3 Done Done Sent false
11816 3907 Acked Write 3 Done Done Sent false
11817 3908 Acked Read 8 Done Done Sent false
11818 3909 Acked Write 7 Done Done Sent false
11819 3910 Acked Write 6 Done Done Sent false
11820 3911 Acked Flush 0 Done Done Sent false
11821 3912 Acked Read 1 Done Done Sent false
11822 3913 Acked Write 10 Done Done Sent false
11823 3914 Acked Read 5 Done Done Sent false
11824 3915 Acked Read 10 Done Done Sent false
11825 3916 Acked Write 4 Done Done Sent false
11826 3917 Acked Read 4 Done Done Sent false
11827 3918 Acked Flush 0 Done Done Sent false
11828 3919 Acked Read 7 Done Done Sent false
11829 3920 Acked Flush 0 Done Done Sent false
11830 3921 Acked Flush 0 Done Done Sent false
11831 3922 Acked Read 5 Done Done Sent false
11832 3923 Acked Read 7 Done Done Sent false
11833 3924 Acked Write 2 Done Done Sent false
11834 3925 Acked Write 8 Done Done Sent false
11835 3926 Acked Read 1 Done Done Sent false
11836 3927 Acked Read 1 Done Done Sent false
11837 3928 Acked Write 8 Done Done Sent false
11838 3929 Acked Read 4 Done Done Sent false
11839 3930 Acked Write 10 Done Done Sent false
11840 3931 Acked Write 6 Done Done Sent false
11841 3932 Acked Read 2 Done Done Sent false
11842 3933 Acked Read 6 Done Done Sent false
11843 3934 Acked Write 4 Done Done Sent false
11844 3935 Acked Read 6 Done Done Sent false
11845 3936 Acked Write 10 Done Done Sent false
11846 3937 Acked Read 8 Done Done Sent false
11847 3938 Acked Write 8 Done Done Sent false
11848 3939 Acked Read 2 Done Done Sent false
11849 3940 Acked Read 7 Done Done Sent false
11850 3941 Acked Read 10 Done Done Sent false
11851 3942 Acked Write 4 Done Done Sent false
11852 3943 Acked Read 3 Done Done Sent false
11853 3944 Acked Read 4 Done Done Sent false
11854 3945 Acked Write 10 Done Done Sent false
11855 3946 Acked Write 8 Done Done Sent false
11856 3947 Acked Write 4 Done Done Sent false
11857 3948 Acked Read 10 Done Done Sent false
11858 3949 Acked Read 1 Done Done Sent false
11859 3950 Acked Read 3 Done Done Sent false
11860 3951 Acked Read 7 Done Done Sent false
11861 3952 Acked Read 10 Done Done Sent false
11862 3953 Acked Write 7 Done Done Sent false
11863 3954 Acked Read 5 Done Done Sent false
11864 3955 Acked Read 10 Done Done Sent false
11865 3956 Acked Read 5 Done Done Sent false
11866 3957 Acked Flush 0 Done Done Sent false
11867 3958 Acked Write 5 Done Done Sent false
11868 3959 Acked Write 10 Done Done Sent false
11869 3960 Acked Read 8 Done Done Sent false
11870 3961 Acked Read 7 Done Done Sent false
11871 3962 Acked Write 7 Done Done Sent false
11872 3963 Acked Read 8 Done Done Sent false
11873 3964 Acked Read 10 Done Done Sent false
11874 3965 Acked Write 1 Done Done Sent false
11875 3966 Acked Read 5 Done Done Sent false
11876 3967 Acked Read 10 Done Done Sent false
11877 3968 Acked Write 1 Done Done Sent false
11878 3969 Acked Write 9 Done Done Sent false
11879 3970 Acked Flush 0 Done Done Sent false
11880 3971 Acked Write 8 Done Done Sent false
11881 3972 Acked Read 8 Done Done Sent false
11882 3973 Acked Flush 0 Done Done Sent false
11883 3974 Acked Read 3 Done Done Sent false
11884 3975 Acked Write 9 Done Done Sent false
11885 3976 Acked Read 10 Done Done Sent false
11886 3977 Acked Write 7 Done Done Sent false
11887 3978 Acked Write 1 Done Done Sent false
11888 3979 Acked Write 6 Done Done Sent false
11889 3980 Acked Write 6 Done Done Sent false
11890 3981 Acked Read 9 Done Done Sent false
11891 3982 Acked Read 9 Done Done Sent false
11892 3983 Acked Write 8 Done Done Sent false
11893 3984 Acked Flush 0 Done Done Sent false
11894 3985 Acked Read 9 Done Done Sent false
11895 3986 Acked Read 6 Done Done Sent false
11896 3987 Acked Read 6 Done Done Sent false
11897 3988 Acked Write 8 Done Done Sent false
11898 3989 Acked Read 9 Done Done Sent false
11899 3990 Acked Read 10 Done Done Sent false
11900 3991 Acked Read 6 Done Done Sent false
11901 3992 Acked Read 2 Done Done Sent false
11902 3993 Acked Write 5 Done Done Sent false
11903 3994 Acked Flush 0 Done Done Sent false
11904 3995 Acked Write 8 Done Done Sent false
11905 3996 Acked Flush 0 Done Done Sent false
11906 3997 Acked Write 4 Done Done Sent false
11907 3998 Acked Write 7 Done Done Sent false
11908 3999 Acked Write 4 Done Done Sent false
11909 4000 Acked Read 8 Done Done Sent false
11910 4001 Acked Read 8 Done Done Sent false
11911 4002 Acked Read 7 Done Done Sent false
11912 4003 Acked Read 9 Done Done Sent false
11913 4004 Acked Write 2 Done Done Sent false
11914 4005 Acked Read 4 Done Done Sent false
11915 4006 Acked Write 7 Done Done Sent false
11916 4007 Acked Read 9 Done Done Sent false
11917 4008 Acked Write 10 Done Done Sent false
11918 4009 Acked Write 10 Done Done Sent false
11919 4010 Acked Read 6 Done Done Sent false
11920 4011 Acked Write 1 Done Done Sent false
11921 4012 Acked Write 4 Done Done Sent false
11922 4013 Acked Write 2 Done Done Sent false
11923 4014 Acked Flush 0 Done Done Sent false
11924 4015 Acked Flush 0 Done Done Sent false
11925 4016 Acked Flush 0 Done Done Sent false
11926 4017 Acked Read 5 Done Done Sent false
11927 4018 Acked Read 3 Done Done Sent false
11928 STATES DS:0 DS:1 DS:2 TOTAL
11929 Sent 0 0 301 301
11930 Done 301 301 0 602
11931 Skipped 0 0 0 0
11932 Error 0 0 0 0
11933 Last Flush: Some(JobId(4016)) Some(JobId(4016)) Some(JobId(3717))
11934 Downstairs last five completed: 3717 3716 3715 3714 3713
11935 Upstairs last five completed: 4018 4017 4016 4015 4014
11936 CLIENT: Up:0 ds:301 act:2
119372025-03-17T22:37:08.619ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119382025-03-17T22:37:09.123ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119392025-03-17T22:37:09.123ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119402025-03-17T22:37:09.123ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119412025-03-17T22:37:09.123ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119422025-03-17T22:37:09.123ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11943 [8830] got start action
11944 [8830] I am going to restart
11945 Starting downstairs at port 8830
11946 Make output file at "/tmp/dsc/downstairs-8830.txt"
11947 [8830][2] reports Starting
11948 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1218)
11949 [8830][2] reports Running
119502025-03-17T22:37:09.624ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
119512025-03-17T22:37:10.557ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
11952 ----------------------------------------------------------------
11953 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11954 States: Active Active Offline
11955 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11956 3718 Acked Read 10 Done Done Sent false
11957 3719 Acked Read 10 Done Done Sent false
11958 3720 Acked Write 10 Done Done Sent false
11959 3721 Acked Read 2 Done Done Sent false
11960 3722 Acked Write 7 Done Done Sent false
11961 3723 Acked Write 1 Done Done Sent false
11962 3724 Acked Write 2 Done Done Sent false
11963 3725 Acked Read 8 Done Done Sent false
11964 3726 Acked Write 5 Done Done Sent false
11965 3727 Acked Read 3 Done Done Sent false
11966 3728 Acked Flush 0 Done Done Sent false
11967 3729 Acked Write 9 Done Done Sent false
11968 3730 Acked Read 2 Done Done Sent false
11969 3731 Acked Write 1 Done Done Sent false
11970 3732 Acked Read 8 Done Done Sent false
11971 3733 Acked Read 9 Done Done Sent false
11972 3734 Acked Flush 0 Done Done Sent false
11973 3735 Acked Write 5 Done Done Sent false
11974 3736 Acked Write 3 Done Done Sent false
11975 3737 Acked Read 5 Done Done Sent false
11976 3738 Acked Read 9 Done Done Sent false
11977 3739 Acked Write 6 Done Done Sent false
11978 3740 Acked Write 8 Done Done Sent false
11979 3741 Acked Read 6 Done Done Sent false
11980 3742 Acked Read 10 Done Done Sent false
11981 3743 Acked Write 10 Done Done Sent false
11982 3744 Acked Read 1 Done Done Sent false
11983 3745 Acked Flush 0 Done Done Sent false
11984 3746 Acked Read 8 Done Done Sent false
11985 3747 Acked Write 5 Done Done Sent false
11986 3748 Acked Write 7 Done Done Sent false
11987 3749 Acked Flush 0 Done Done Sent false
11988 3750 Acked Write 5 Done Done Sent false
11989 3751 Acked Write 7 Done Done Sent false
11990 3752 Acked Write 6 Done Done Sent false
11991 3753 Acked Write 4 Done Done Sent false
11992 3754 Acked Read 5 Done Done Sent false
11993 3755 Acked Write 2 Done Done Sent false
11994 3756 Acked Write 8 Done Done Sent false
11995 3757 Acked Write 3 Done Done Sent false
11996 3758 Acked Read 9 Done Done Sent false
11997 3759 Acked Read 3 Done Done Sent false
11998 3760 Acked Write 7 Done Done Sent false
11999 3761 Acked Write 2 Done Done Sent false
12000 3762 Acked Write 1 Done Done Sent false
12001 3763 Acked Read 5 Done Done Sent false
12002 3764 Acked Read 2 Done Done Sent false
12003 3765 Acked Read 8 Done Done Sent false
12004 3766 Acked Write 2 Done Done Sent false
12005 3767 Acked Read 8 Done Done Sent false
12006 3768 Acked Read 1 Done Done Sent false
12007 3769 Acked Write 1 Done Done Sent false
12008 3770 Acked Read 10 Done Done Sent false
12009 3771 Acked Write 5 Done Done Sent false
12010 3772 Acked Write 8 Done Done Sent false
12011 3773 Acked Read 9 Done Done Sent false
12012 3774 Acked Read 5 Done Done Sent false
12013 3775 Acked Write 10 Done Done Sent false
12014 3776 Acked Read 9 Done Done Sent false
12015 3777 Acked Read 8 Done Done Sent false
12016 3778 Acked Read 4 Done Done Sent false
12017 3779 Acked Write 7 Done Done Sent false
12018 3780 Acked Write 1 Done Done Sent false
12019 3781 Acked Read 6 Done Done Sent false
12020 3782 Acked Write 10 Done Done Sent false
12021 3783 Acked Flush 0 Done Done Sent false
12022 3784 Acked Flush 0 Done Done Sent false
12023 3785 Acked Flush 0 Done Done Sent false
12024 3786 Acked Write 9 Done Done Sent false
12025 3787 Acked Read 6 Done Done Sent false
12026 3788 Acked Write 9 Done Done Sent false
12027 3789 Acked Write 2 Done Done Sent false
12028 3790 Acked Write 4 Done Done Sent false
12029 3791 Acked Write 1 Done Done Sent false
12030 3792 Acked Read 5 Done Done Sent false
12031 3793 Acked Write 1 Done Done Sent false
12032 3794 Acked Write 4 Done Done Sent false
12033 3795 Acked Write 5 Done Done Sent false
12034 3796 Acked Flush 0 Done Done Sent false
12035 3797 Acked Write 9 Done Done Sent false
12036 3798 Acked Write 1 Done Done Sent false
12037 3799 Acked Read 3 Done Done Sent false
12038 3800 Acked Write 6 Done Done Sent false
12039 3801 Acked Flush 0 Done Done Sent false
12040 3802 Acked Write 2 Done Done Sent false
12041 3803 Acked Write 2 Done Done Sent false
12042 3804 Acked Read 10 Done Done Sent false
12043 3805 Acked Read 5 Done Done Sent false
12044 3806 Acked Read 8 Done Done Sent false
12045 3807 Acked Read 1 Done Done Sent false
12046 3808 Acked Write 3 Done Done Sent false
12047 3809 Acked Write 8 Done Done Sent false
12048 3810 Acked Write 5 Done Done Sent false
12049 3811 Acked Read 8 Done Done Sent false
12050 3812 Acked Write 6 Done Done Sent false
12051 3813 Acked Write 10 Done Done Sent false
12052 3814 Acked Flush 0 Done Done Sent false
12053 3815 Acked Write 9 Done Done Sent false
12054 3816 Acked Read 1 Done Done Sent false
12055 3817 Acked Read 9 Done Done Sent false
12056 3818 Acked Read 10 Done Done Sent false
12057 3819 Acked Write 2 Done Done Sent false
12058 3820 Acked Read 6 Done Done Sent false
12059 3821 Acked Write 5 Done Done Sent false
12060 3822 Acked Read 3 Done Done Sent false
12061 3823 Acked Read 9 Done Done Sent false
12062 3824 Acked Write 3 Done Done Sent false
12063 3825 Acked Flush 0 Done Done Sent false
12064 3826 Acked Read 10 Done Done Sent false
12065 3827 Acked Read 5 Done Done Sent false
12066 3828 Acked Read 10 Done Done Sent false
12067 3829 Acked Write 4 Done Done Sent false
12068 3830 Acked Read 2 Done Done Sent false
12069 3831 Acked Write 9 Done Done Sent false
12070 3832 Acked Write 7 Done Done Sent false
12071 3833 Acked Write 9 Done Done Sent false
12072 3834 Acked Read 4 Done Done Sent false
12073 3835 Acked Read 3 Done Done Sent false
12074 3836 Acked Write 6 Done Done Sent false
12075 3837 Acked Read 2 Done Done Sent false
12076 3838 Acked Write 6 Done Done Sent false
12077 3839 Acked Write 2 Done Done Sent false
12078 3840 Acked Read 3 Done Done Sent false
12079 3841 Acked Read 10 Done Done Sent false
12080 3842 Acked Read 6 Done Done Sent false
12081 3843 Acked Write 7 Done Done Sent false
12082 3844 Acked Write 5 Done Done Sent false
12083 3845 Acked Write 7 Done Done Sent false
12084 3846 Acked Flush 0 Done Done Sent false
12085 3847 Acked Write 4 Done Done Sent false
12086 3848 Acked Write 7 Done Done Sent false
12087 3849 Acked Write 10 Done Done Sent false
12088 3850 Acked Write 8 Done Done Sent false
12089 3851 Acked Read 4 Done Done Sent false
12090 3852 Acked Read 2 Done Done Sent false
12091 3853 Acked Flush 0 Done Done Sent false
12092 3854 Acked Write 5 Done Done Sent false
12093 3855 Acked Read 8 Done Done Sent false
12094 3856 Acked Write 2 Done Done Sent false
12095 3857 Acked Write 1 Done Done Sent false
12096 3858 Acked Read 7 Done Done Sent false
12097 3859 Acked Read 4 Done Done Sent false
12098 3860 Acked Read 4 Done Done Sent false
12099 3861 Acked Write 1 Done Done Sent false
12100 3862 Acked Read 7 Done Done Sent false
12101 3863 Acked Read 3 Done Done Sent false
12102 3864 Acked Read 6 Done Done Sent false
12103 3865 Acked Read 7 Done Done Sent false
12104 3866 Acked Read 2 Done Done Sent false
12105 3867 Acked Read 8 Done Done Sent false
12106 3868 Acked Read 9 Done Done Sent false
12107 3869 Acked Write 5 Done Done Sent false
12108 3870 Acked Read 5 Done Done Sent false
12109 3871 Acked Read 10 Done Done Sent false
12110 3872 Acked Read 2 Done Done Sent false
12111 3873 Acked Write 7 Done Done Sent false
12112 3874 Acked Write 8 Done Done Sent false
12113 3875 Acked Write 8 Done Done Sent false
12114 3876 Acked Write 5 Done Done Sent false
12115 3877 Acked Write 9 Done Done Sent false
12116 3878 Acked Write 1 Done Done Sent false
12117 3879 Acked Read 2 Done Done Sent false
12118 3880 Acked Write 9 Done Done Sent false
12119 3881 Acked Read 2 Done Done Sent false
12120 3882 Acked Read 8 Done Done Sent false
12121 3883 Acked Write 5 Done Done Sent false
12122 3884 Acked Flush 0 Done Done Sent false
12123 3885 Acked Write 2 Done Done Sent false
12124 3886 Acked Read 7 Done Done Sent false
12125 3887 Acked Read 3 Done Done Sent false
12126 3888 Acked Write 8 Done Done Sent false
12127 3889 Acked Write 1 Done Done Sent false
12128 3890 Acked Read 6 Done Done Sent false
12129 3891 Acked Read 3 Done Done Sent false
12130 3892 Acked Write 7 Done Done Sent false
12131 3893 Acked Flush 0 Done Done Sent false
12132 3894 Acked Read 7 Done Done Sent false
12133 3895 Acked Read 3 Done Done Sent false
12134 3896 Acked Read 4 Done Done Sent false
12135 3897 Acked Read 3 Done Done Sent false
12136 3898 Acked Read 4 Done Done Sent false
12137 3899 Acked Write 7 Done Done Sent false
12138 3900 Acked Write 1 Done Done Sent false
12139 3901 Acked Write 9 Done Done Sent false
12140 3902 Acked Read 2 Done Done Sent false
12141 3903 Acked Read 9 Done Done Sent false
12142 3904 Acked Write 10 Done Done Sent false
12143 3905 Acked Read 6 Done Done Sent false
12144 3906 Acked Write 3 Done Done Sent false
12145 3907 Acked Write 3 Done Done Sent false
12146 3908 Acked Read 8 Done Done Sent false
12147 3909 Acked Write 7 Done Done Sent false
12148 3910 Acked Write 6 Done Done Sent false
12149 3911 Acked Flush 0 Done Done Sent false
12150 3912 Acked Read 1 Done Done Sent false
12151 3913 Acked Write 10 Done Done Sent false
12152 3914 Acked Read 5 Done Done Sent false
12153 3915 Acked Read 10 Done Done Sent false
12154 3916 Acked Write 4 Done Done Sent false
12155 3917 Acked Read 4 Done Done Sent false
12156 3918 Acked Flush 0 Done Done Sent false
12157 3919 Acked Read 7 Done Done Sent false
12158 3920 Acked Flush 0 Done Done Sent false
12159 3921 Acked Flush 0 Done Done Sent false
12160 3922 Acked Read 5 Done Done Sent false
12161 3923 Acked Read 7 Done Done Sent false
12162 3924 Acked Write 2 Done Done Sent false
12163 3925 Acked Write 8 Done Done Sent false
12164 3926 Acked Read 1 Done Done Sent false
12165 3927 Acked Read 1 Done Done Sent false
12166 3928 Acked Write 8 Done Done Sent false
12167 3929 Acked Read 4 Done Done Sent false
12168 3930 Acked Write 10 Done Done Sent false
12169 3931 Acked Write 6 Done Done Sent false
12170 3932 Acked Read 2 Done Done Sent false
12171 3933 Acked Read 6 Done Done Sent false
12172 3934 Acked Write 4 Done Done Sent false
12173 3935 Acked Read 6 Done Done Sent false
12174 3936 Acked Write 10 Done Done Sent false
12175 3937 Acked Read 8 Done Done Sent false
12176 3938 Acked Write 8 Done Done Sent false
12177 3939 Acked Read 2 Done Done Sent false
12178 3940 Acked Read 7 Done Done Sent false
12179 3941 Acked Read 10 Done Done Sent false
12180 3942 Acked Write 4 Done Done Sent false
12181 3943 Acked Read 3 Done Done Sent false
12182 3944 Acked Read 4 Done Done Sent false
12183 3945 Acked Write 10 Done Done Sent false
12184 3946 Acked Write 8 Done Done Sent false
12185 3947 Acked Write 4 Done Done Sent false
12186 3948 Acked Read 10 Done Done Sent false
12187 3949 Acked Read 1 Done Done Sent false
12188 3950 Acked Read 3 Done Done Sent false
12189 3951 Acked Read 7 Done Done Sent false
12190 3952 Acked Read 10 Done Done Sent false
12191 3953 Acked Write 7 Done Done Sent false
12192 3954 Acked Read 5 Done Done Sent false
12193 3955 Acked Read 10 Done Done Sent false
12194 3956 Acked Read 5 Done Done Sent false
12195 3957 Acked Flush 0 Done Done Sent false
12196 3958 Acked Write 5 Done Done Sent false
12197 3959 Acked Write 10 Done Done Sent false
12198 3960 Acked Read 8 Done Done Sent false
12199 3961 Acked Read 7 Done Done Sent false
12200 3962 Acked Write 7 Done Done Sent false
12201 3963 Acked Read 8 Done Done Sent false
12202 3964 Acked Read 10 Done Done Sent false
12203 3965 Acked Write 1 Done Done Sent false
12204 3966 Acked Read 5 Done Done Sent false
12205 3967 Acked Read 10 Done Done Sent false
12206 3968 Acked Write 1 Done Done Sent false
12207 3969 Acked Write 9 Done Done Sent false
12208 3970 Acked Flush 0 Done Done Sent false
12209 3971 Acked Write 8 Done Done Sent false
12210 3972 Acked Read 8 Done Done Sent false
12211 3973 Acked Flush 0 Done Done Sent false
12212 3974 Acked Read 3 Done Done Sent false
12213 3975 Acked Write 9 Done Done Sent false
12214 3976 Acked Read 10 Done Done Sent false
12215 3977 Acked Write 7 Done Done Sent false
12216 3978 Acked Write 1 Done Done Sent false
12217 3979 Acked Write 6 Done Done Sent false
12218 3980 Acked Write 6 Done Done Sent false
12219 3981 Acked Read 9 Done Done Sent false
12220 3982 Acked Read 9 Done Done Sent false
12221 3983 Acked Write 8 Done Done Sent false
12222 3984 Acked Flush 0 Done Done Sent false
12223 3985 Acked Read 9 Done Done Sent false
12224 3986 Acked Read 6 Done Done Sent false
12225 3987 Acked Read 6 Done Done Sent false
12226 3988 Acked Write 8 Done Done Sent false
12227 3989 Acked Read 9 Done Done Sent false
12228 3990 Acked Read 10 Done Done Sent false
12229 3991 Acked Read 6 Done Done Sent false
12230 3992 Acked Read 2 Done Done Sent false
12231 3993 Acked Write 5 Done Done Sent false
12232 3994 Acked Flush 0 Done Done Sent false
12233 3995 Acked Write 8 Done Done Sent false
12234 3996 Acked Flush 0 Done Done Sent false
12235 3997 Acked Write 4 Done Done Sent false
12236 3998 Acked Write 7 Done Done Sent false
12237 3999 Acked Write 4 Done Done Sent false
12238 4000 Acked Read 8 Done Done Sent false
12239 4001 Acked Read 8 Done Done Sent false
12240 4002 Acked Read 7 Done Done Sent false
12241 4003 Acked Read 9 Done Done Sent false
12242 4004 Acked Write 2 Done Done Sent false
12243 4005 Acked Read 4 Done Done Sent false
12244 4006 Acked Write 7 Done Done Sent false
12245 4007 Acked Read 9 Done Done Sent false
12246 4008 Acked Write 10 Done Done Sent false
12247 4009 Acked Write 10 Done Done Sent false
12248 4010 Acked Read 6 Done Done Sent false
12249 4011 Acked Write 1 Done Done Sent false
12250 4012 Acked Write 4 Done Done Sent false
12251 4013 Acked Write 2 Done Done Sent false
12252 4014 Acked Flush 0 Done Done Sent false
12253 4015 Acked Flush 0 Done Done Sent false
12254 4016 Acked Flush 0 Done Done Sent false
12255 4017 Acked Read 5 Done Done Sent false
12256 4018 Acked Read 3 Done Done Sent false
12257 STATES DS:0 DS:1 DS:2 TOTAL
12258 Sent 0 0 301 301
12259 Done 301 301 0 602
12260 Skipped 0 0 0 0
12261 Error 0 0 0 0
12262 Last Flush: Some(JobId(4016)) Some(JobId(4016)) Some(JobId(3717))
12263 Downstairs last five completed: 3717 3716 3715 3714 3713
12264 Upstairs last five completed: 4018 4017 4016 4015 4014
12265 CLIENT: Up:0 ds:301 act:2
122662025-03-17T22:37:11.877ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
122672025-03-17T22:37:12.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
122682025-03-17T22:37:13.620ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
122692025-03-17T22:37:14.563ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
12270 ----------------------------------------------------------------
12271 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12272 States: Active Active Offline
12273 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12274 3718 Acked Read 10 Done Done Sent false
12275 3719 Acked Read 10 Done Done Sent false
12276 3720 Acked Write 10 Done Done Sent false
12277 3721 Acked Read 2 Done Done Sent false
12278 3722 Acked Write 7 Done Done Sent false
12279 3723 Acked Write 1 Done Done Sent false
12280 3724 Acked Write 2 Done Done Sent false
12281 3725 Acked Read 8 Done Done Sent false
12282 3726 Acked Write 5 Done Done Sent false
12283 3727 Acked Read 3 Done Done Sent false
12284 3728 Acked Flush 0 Done Done Sent false
12285 3729 Acked Write 9 Done Done Sent false
12286 3730 Acked Read 2 Done Done Sent false
12287 3731 Acked Write 1 Done Done Sent false
12288 3732 Acked Read 8 Done Done Sent false
12289 3733 Acked Read 9 Done Done Sent false
12290 3734 Acked Flush 0 Done Done Sent false
12291 3735 Acked Write 5 Done Done Sent false
12292 3736 Acked Write 3 Done Done Sent false
12293 3737 Acked Read 5 Done Done Sent false
12294 3738 Acked Read 9 Done Done Sent false
12295 3739 Acked Write 6 Done Done Sent false
12296 3740 Acked Write 8 Done Done Sent false
12297 3741 Acked Read 6 Done Done Sent false
12298 3742 Acked Read 10 Done Done Sent false
12299 3743 Acked Write 10 Done Done Sent false
12300 3744 Acked Read 1 Done Done Sent false
12301 3745 Acked Flush 0 Done Done Sent false
12302 3746 Acked Read 8 Done Done Sent false
12303 3747 Acked Write 5 Done Done Sent false
12304 3748 Acked Write 7 Done Done Sent false
12305 3749 Acked Flush 0 Done Done Sent false
12306 3750 Acked Write 5 Done Done Sent false
12307 3751 Acked Write 7 Done Done Sent false
12308 3752 Acked Write 6 Done Done Sent false
12309 3753 Acked Write 4 Done Done Sent false
12310 3754 Acked Read 5 Done Done Sent false
12311 3755 Acked Write 2 Done Done Sent false
12312 3756 Acked Write 8 Done Done Sent false
12313 3757 Acked Write 3 Done Done Sent false
12314 3758 Acked Read 9 Done Done Sent false
12315 3759 Acked Read 3 Done Done Sent false
12316 3760 Acked Write 7 Done Done Sent false
12317 3761 Acked Write 2 Done Done Sent false
12318 3762 Acked Write 1 Done Done Sent false
12319 3763 Acked Read 5 Done Done Sent false
12320 3764 Acked Read 2 Done Done Sent false
12321 3765 Acked Read 8 Done Done Sent false
12322 3766 Acked Write 2 Done Done Sent false
12323 3767 Acked Read 8 Done Done Sent false
12324 3768 Acked Read 1 Done Done Sent false
12325 3769 Acked Write 1 Done Done Sent false
12326 3770 Acked Read 10 Done Done Sent false
12327 3771 Acked Write 5 Done Done Sent false
12328 3772 Acked Write 8 Done Done Sent false
12329 3773 Acked Read 9 Done Done Sent false
12330 3774 Acked Read 5 Done Done Sent false
12331 3775 Acked Write 10 Done Done Sent false
12332 3776 Acked Read 9 Done Done Sent false
12333 3777 Acked Read 8 Done Done Sent false
12334 3778 Acked Read 4 Done Done Sent false
12335 3779 Acked Write 7 Done Done Sent false
12336 3780 Acked Write 1 Done Done Sent false
12337 3781 Acked Read 6 Done Done Sent false
12338 3782 Acked Write 10 Done Done Sent false
12339 3783 Acked Flush 0 Done Done Sent false
12340 3784 Acked Flush 0 Done Done Sent false
12341 3785 Acked Flush 0 Done Done Sent false
12342 3786 Acked Write 9 Done Done Sent false
12343 3787 Acked Read 6 Done Done Sent false
12344 3788 Acked Write 9 Done Done Sent false
12345 3789 Acked Write 2 Done Done Sent false
12346 3790 Acked Write 4 Done Done Sent false
12347 3791 Acked Write 1 Done Done Sent false
12348 3792 Acked Read 5 Done Done Sent false
12349 3793 Acked Write 1 Done Done Sent false
12350 3794 Acked Write 4 Done Done Sent false
12351 3795 Acked Write 5 Done Done Sent false
12352 3796 Acked Flush 0 Done Done Sent false
12353 3797 Acked Write 9 Done Done Sent false
12354 3798 Acked Write 1 Done Done Sent false
12355 3799 Acked Read 3 Done Done Sent false
12356 3800 Acked Write 6 Done Done Sent false
12357 3801 Acked Flush 0 Done Done Sent false
12358 3802 Acked Write 2 Done Done Sent false
12359 3803 Acked Write 2 Done Done Sent false
12360 3804 Acked Read 10 Done Done Sent false
12361 3805 Acked Read 5 Done Done Sent false
12362 3806 Acked Read 8 Done Done Sent false
12363 3807 Acked Read 1 Done Done Sent false
12364 3808 Acked Write 3 Done Done Sent false
12365 3809 Acked Write 8 Done Done Sent false
12366 3810 Acked Write 5 Done Done Sent false
12367 3811 Acked Read 8 Done Done Sent false
12368 3812 Acked Write 6 Done Done Sent false
12369 3813 Acked Write 10 Done Done Sent false
12370 3814 Acked Flush 0 Done Done Sent false
12371 3815 Acked Write 9 Done Done Sent false
12372 3816 Acked Read 1 Done Done Sent false
12373 3817 Acked Read 9 Done Done Sent false
12374 3818 Acked Read 10 Done Done Sent false
12375 3819 Acked Write 2 Done Done Sent false
12376 3820 Acked Read 6 Done Done Sent false
12377 3821 Acked Write 5 Done Done Sent false
12378 3822 Acked Read 3 Done Done Sent false
12379 3823 Acked Read 9 Done Done Sent false
12380 3824 Acked Write 3 Done Done Sent false
12381 3825 Acked Flush 0 Done Done Sent false
12382 3826 Acked Read 10 Done Done Sent false
12383 3827 Acked Read 5 Done Done Sent false
12384 3828 Acked Read 10 Done Done Sent false
12385 3829 Acked Write 4 Done Done Sent false
12386 3830 Acked Read 2 Done Done Sent false
12387 3831 Acked Write 9 Done Done Sent false
12388 3832 Acked Write 7 Done Done Sent false
12389 3833 Acked Write 9 Done Done Sent false
12390 3834 Acked Read 4 Done Done Sent false
12391 3835 Acked Read 3 Done Done Sent false
12392 3836 Acked Write 6 Done Done Sent false
12393 3837 Acked Read 2 Done Done Sent false
12394 3838 Acked Write 6 Done Done Sent false
12395 3839 Acked Write 2 Done Done Sent false
12396 3840 Acked Read 3 Done Done Sent false
12397 3841 Acked Read 10 Done Done Sent false
12398 3842 Acked Read 6 Done Done Sent false
12399 3843 Acked Write 7 Done Done Sent false
12400 3844 Acked Write 5 Done Done Sent false
12401 3845 Acked Write 7 Done Done Sent false
12402 3846 Acked Flush 0 Done Done Sent false
12403 3847 Acked Write 4 Done Done Sent false
12404 3848 Acked Write 7 Done Done Sent false
12405 3849 Acked Write 10 Done Done Sent false
12406 3850 Acked Write 8 Done Done Sent false
12407 3851 Acked Read 4 Done Done Sent false
12408 3852 Acked Read 2 Done Done Sent false
12409 3853 Acked Flush 0 Done Done Sent false
12410 3854 Acked Write 5 Done Done Sent false
12411 3855 Acked Read 8 Done Done Sent false
12412 3856 Acked Write 2 Done Done Sent false
12413 3857 Acked Write 1 Done Done Sent false
12414 3858 Acked Read 7 Done Done Sent false
12415 3859 Acked Read 4 Done Done Sent false
12416 3860 Acked Read 4 Done Done Sent false
12417 3861 Acked Write 1 Done Done Sent false
12418 3862 Acked Read 7 Done Done Sent false
12419 3863 Acked Read 3 Done Done Sent false
12420 3864 Acked Read 6 Done Done Sent false
12421 3865 Acked Read 7 Done Done Sent false
12422 3866 Acked Read 2 Done Done Sent false
12423 3867 Acked Read 8 Done Done Sent false
12424 3868 Acked Read 9 Done Done Sent false
12425 3869 Acked Write 5 Done Done Sent false
12426 3870 Acked Read 5 Done Done Sent false
12427 3871 Acked Read 10 Done Done Sent false
12428 3872 Acked Read 2 Done Done Sent false
12429 3873 Acked Write 7 Done Done Sent false
12430 3874 Acked Write 8 Done Done Sent false
12431 3875 Acked Write 8 Done Done Sent false
12432 3876 Acked Write 5 Done Done Sent false
12433 3877 Acked Write 9 Done Done Sent false
12434 3878 Acked Write 1 Done Done Sent false
12435 3879 Acked Read 2 Done Done Sent false
12436 3880 Acked Write 9 Done Done Sent false
12437 3881 Acked Read 2 Done Done Sent false
12438 3882 Acked Read 8 Done Done Sent false
12439 3883 Acked Write 5 Done Done Sent false
12440 3884 Acked Flush 0 Done Done Sent false
12441 3885 Acked Write 2 Done Done Sent false
12442 3886 Acked Read 7 Done Done Sent false
12443 3887 Acked Read 3 Done Done Sent false
12444 3888 Acked Write 8 Done Done Sent false
12445 3889 Acked Write 1 Done Done Sent false
12446 3890 Acked Read 6 Done Done Sent false
12447 3891 Acked Read 3 Done Done Sent false
12448 3892 Acked Write 7 Done Done Sent false
12449 3893 Acked Flush 0 Done Done Sent false
12450 3894 Acked Read 7 Done Done Sent false
12451 3895 Acked Read 3 Done Done Sent false
12452 3896 Acked Read 4 Done Done Sent false
12453 3897 Acked Read 3 Done Done Sent false
12454 3898 Acked Read 4 Done Done Sent false
12455 3899 Acked Write 7 Done Done Sent false
12456 3900 Acked Write 1 Done Done Sent false
12457 3901 Acked Write 9 Done Done Sent false
12458 3902 Acked Read 2 Done Done Sent false
12459 3903 Acked Read 9 Done Done Sent false
12460 3904 Acked Write 10 Done Done Sent false
12461 3905 Acked Read 6 Done Done Sent false
12462 3906 Acked Write 3 Done Done Sent false
12463 3907 Acked Write 3 Done Done Sent false
12464 3908 Acked Read 8 Done Done Sent false
12465 3909 Acked Write 7 Done Done Sent false
12466 3910 Acked Write 6 Done Done Sent false
12467 3911 Acked Flush 0 Done Done Sent false
12468 3912 Acked Read 1 Done Done Sent false
12469 3913 Acked Write 10 Done Done Sent false
12470 3914 Acked Read 5 Done Done Sent false
12471 3915 Acked Read 10 Done Done Sent false
12472 3916 Acked Write 4 Done Done Sent false
12473 3917 Acked Read 4 Done Done Sent false
12474 3918 Acked Flush 0 Done Done Sent false
12475 3919 Acked Read 7 Done Done Sent false
12476 3920 Acked Flush 0 Done Done Sent false
12477 3921 Acked Flush 0 Done Done Sent false
12478 3922 Acked Read 5 Done Done Sent false
12479 3923 Acked Read 7 Done Done Sent false
12480 3924 Acked Write 2 Done Done Sent false
12481 3925 Acked Write 8 Done Done Sent false
12482 3926 Acked Read 1 Done Done Sent false
12483 3927 Acked Read 1 Done Done Sent false
12484 3928 Acked Write 8 Done Done Sent false
12485 3929 Acked Read 4 Done Done Sent false
12486 3930 Acked Write 10 Done Done Sent false
12487 3931 Acked Write 6 Done Done Sent false
12488 3932 Acked Read 2 Done Done Sent false
12489 3933 Acked Read 6 Done Done Sent false
12490 3934 Acked Write 4 Done Done Sent false
12491 3935 Acked Read 6 Done Done Sent false
12492 3936 Acked Write 10 Done Done Sent false
12493 3937 Acked Read 8 Done Done Sent false
12494 3938 Acked Write 8 Done Done Sent false
12495 3939 Acked Read 2 Done Done Sent false
12496 3940 Acked Read 7 Done Done Sent false
12497 3941 Acked Read 10 Done Done Sent false
12498 3942 Acked Write 4 Done Done Sent false
12499 3943 Acked Read 3 Done Done Sent false
12500 3944 Acked Read 4 Done Done Sent false
12501 3945 Acked Write 10 Done Done Sent false
12502 3946 Acked Write 8 Done Done Sent false
12503 3947 Acked Write 4 Done Done Sent false
12504 3948 Acked Read 10 Done Done Sent false
12505 3949 Acked Read 1 Done Done Sent false
12506 3950 Acked Read 3 Done Done Sent false
12507 3951 Acked Read 7 Done Done Sent false
12508 3952 Acked Read 10 Done Done Sent false
12509 3953 Acked Write 7 Done Done Sent false
12510 3954 Acked Read 5 Done Done Sent false
12511 3955 Acked Read 10 Done Done Sent false
12512 3956 Acked Read 5 Done Done Sent false
12513 3957 Acked Flush 0 Done Done Sent false
12514 3958 Acked Write 5 Done Done Sent false
12515 3959 Acked Write 10 Done Done Sent false
12516 3960 Acked Read 8 Done Done Sent false
12517 3961 Acked Read 7 Done Done Sent false
12518 3962 Acked Write 7 Done Done Sent false
12519 3963 Acked Read 8 Done Done Sent false
12520 3964 Acked Read 10 Done Done Sent false
12521 3965 Acked Write 1 Done Done Sent false
12522 3966 Acked Read 5 Done Done Sent false
12523 3967 Acked Read 10 Done Done Sent false
12524 3968 Acked Write 1 Done Done Sent false
12525 3969 Acked Write 9 Done Done Sent false
12526 3970 Acked Flush 0 Done Done Sent false
12527 3971 Acked Write 8 Done Done Sent false
12528 3972 Acked Read 8 Done Done Sent false
12529 3973 Acked Flush 0 Done Done Sent false
12530 3974 Acked Read 3 Done Done Sent false
12531 3975 Acked Write 9 Done Done Sent false
12532 3976 Acked Read 10 Done Done Sent false
12533 3977 Acked Write 7 Done Done Sent false
12534 3978 Acked Write 1 Done Done Sent false
12535 3979 Acked Write 6 Done Done Sent false
12536 3980 Acked Write 6 Done Done Sent false
12537 3981 Acked Read 9 Done Done Sent false
12538 3982 Acked Read 9 Done Done Sent false
12539 3983 Acked Write 8 Done Done Sent false
12540 3984 Acked Flush 0 Done Done Sent false
12541 3985 Acked Read 9 Done Done Sent false
12542 3986 Acked Read 6 Done Done Sent false
12543 3987 Acked Read 6 Done Done Sent false
12544 3988 Acked Write 8 Done Done Sent false
12545 3989 Acked Read 9 Done Done Sent false
12546 3990 Acked Read 10 Done Done Sent false
12547 3991 Acked Read 6 Done Done Sent false
12548 3992 Acked Read 2 Done Done Sent false
12549 3993 Acked Write 5 Done Done Sent false
12550 3994 Acked Flush 0 Done Done Sent false
12551 3995 Acked Write 8 Done Done Sent false
12552 3996 Acked Flush 0 Done Done Sent false
12553 3997 Acked Write 4 Done Done Sent false
12554 3998 Acked Write 7 Done Done Sent false
12555 3999 Acked Write 4 Done Done Sent false
12556 4000 Acked Read 8 Done Done Sent false
12557 4001 Acked Read 8 Done Done Sent false
12558 4002 Acked Read 7 Done Done Sent false
12559 4003 Acked Read 9 Done Done Sent false
12560 4004 Acked Write 2 Done Done Sent false
12561 4005 Acked Read 4 Done Done Sent false
12562 4006 Acked Write 7 Done Done Sent false
12563 4007 Acked Read 9 Done Done Sent false
12564 4008 Acked Write 10 Done Done Sent false
12565 4009 Acked Write 10 Done Done Sent false
12566 4010 Acked Read 6 Done Done Sent false
12567 4011 Acked Write 1 Done Done Sent false
12568 4012 Acked Write 4 Done Done Sent false
12569 4013 Acked Write 2 Done Done Sent false
12570 4014 Acked Flush 0 Done Done Sent false
12571 4015 Acked Flush 0 Done Done Sent false
12572 4016 Acked Flush 0 Done Done Sent false
12573 4017 Acked Read 5 Done Done Sent false
12574 4018 Acked Read 3 Done Done Sent false
12575 STATES DS:0 DS:1 DS:2 TOTAL
12576 Sent 0 0 301 301
12577 Done 301 301 0 602
12578 Skipped 0 0 0 0
12579 Error 0 0 0 0
12580 Last Flush: Some(JobId(4016)) Some(JobId(4016)) Some(JobId(3717))
12581 Downstairs last five completed: 3717 3716 3715 3714 3713
12582 Upstairs last five completed: 4018 4017 4016 4015 4014
12583 CLIENT: Up:0 ds:301 act:2
125842025-03-17T22:37:15.879ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125852025-03-17T22:37:16.567ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125862025-03-17T22:37:17.569ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125872025-03-17T22:37:18.622ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125882025-03-17T22:37:19.124ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125892025-03-17T22:37:19.124ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48244) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125902025-03-17T22:37:19.125ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125912025-03-17T22:37:19.126ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125922025-03-17T22:37:19.126ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125932025-03-17T22:37:19.126ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125942025-03-17T22:37:19.126ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125952025-03-17T22:37:19.126ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
125962025-03-17T22:37:19.126ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
12597 ----------------------------------------------------------------
12598 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
12599 States: Active Active Active
12600 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12601 4017 Acked Read 5 Done Done Done true
12602 4018 Acked Read 3 Done Done Done true
12603 STATES DS:0 DS:1 DS:2 TOTAL
12604 Sent 0 0 0 0
12605 Done 2 2 2 6
12606 Skipped 0 0 0 0
12607 Error 0 0 0 0
12608 Last Flush: Some(JobId(4016)) Some(JobId(4016)) Some(JobId(4016))
12609 Downstairs last five completed: 4016 4015 4014 4013 4012
12610 Upstairs last five completed: 4018 4017 4016 4015 4014
12611 CLIENT: Up:0 ds:2 act:3
12612 ----------------------------------------------------------------
12613 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12614 Downstairs last five completed: 4019 4018 4017 4016 4015
12615 Upstairs last five completed: 4019 4018 4017 4016 4015
12616 CLIENT: Up:0 ds:0 act:3
12617 Replay: All jobs finished
12618 Main task has work to do, go find it
12619 got dsc Stop(2)
12620 stop 2
12621 Mar 17 22:37:23.382 INFO request completed, latency_us: 353, response_code[8830] Got stop action so:false kr:false
12622 : 204, uri: /stop/cid/2, method: POST, req_id: dab2c82b-f92f-41a8-9ce9-cac0b9ff88e4, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
12623 Mar 17 22:37:23.383 INFO request completed, latency_us: 146, response_code{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:37:23.383269332Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
12624 : 200, uri: /state/cid/2, method: GET, req_id: 4a743842-bc50-43ea-ba25-cfb12a978d3b, remote_addr:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:37:23.383343718Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
12625 127.0.0.1:35180, local_addr: 127.0.0.1:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:37:23.38337102Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
12626 9998
126272025-03-17T22:37:23.383ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
12628 [8830] Exited with: ExitStatus(unix_wait_status(9))
12629 [8830][2] reports Exit
12630 Mar 17 22:37:27.385 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3a4d0bed-a683-4278-93de-ae81d026219b, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
12631 001/300 Read block 4441 len 20480
12632 002/300 Write block 1336 len 24576 data: 3 3 3 3 4 4
12633 003/300 Flush
12634 004/300 Write block 2329 len 16384 data: 3 3 3 3
12635 005/300 Read block 2726 len 12288
12636 006/300 Write block 138 len 24576 data: 4 3 3 3 3 3
12637 007/300 Write block 1439 len 16384 data: 3 3 3 5
12638 008/300 Write block 4023 len 40960 data: 4 4 4 4 4 4 3 2 2 3
12639 009/300 Read block 4000 len 20480
12640 010/300 Write block 1893 len 40960 data: 2 2 3 3 3 3 2 3 3 3
12641 011/300 Write block 3890 len 8192 data: 2 3
12642 012/300 Write block 1982 len 28672 data: 2 3 3 3 3 3 3
12643 013/300 Read block 1620 len 8192
12644 014/300 Write block 1374 len 28672 data: 2 3 3 3 3 2 3
12645 015/300 Read block 191 len 28672
12646 016/300 Write block 284 len 4096 data: 2
12647 017/300 Read block 3168 len 24576
12648 018/300 Read block 195 len 20480
12649 019/300 Read block 4989 len 36864
12650 020/300 Write block 263 len 24576 data: 4 5 6 5 5 4
12651 021/300 Write block 1745 len 20480 data: 4 4 3 3 2
12652 022/300 Read block 1536 len 4096
12653 023/300 Read block 1050 len 16384
12654 024/300 Read block 790 len 20480
12655 025/300 Write block 533 len 12288 data: 4 5 5
12656 026/300 Read block 3556 len 16384
12657 027/300 Write block 4073 len 36864 data: 3 3 3 2 2 2 2 3 3
12658 028/300 Write block 4232 len 24576 data: 5 4 3 3 3 3
12659 029/300 Write block 2436 len 28672 data: 4 4 4 4 4 4 3
12660 030/300 Read block 1589 len 28672
12661 031/300 Read block 3291 len 8192
12662 032/300 Read block 2241 len 4096
12663 033/300 Read block 4117 len 24576
12664 034/300 Write block 322 len 12288 data: 2 2 2
12665 035/300 Read block 483 len 4096
12666 036/300 Write block 1844 len 20480 data: 3 4 4 4 3
12667 037/300 Flush
12668 038/300 Read block 2838 len 40960
12669 039/300 Write block 3207 len 8192 data: 3 3
12670 040/300 Write block 855 len 8192 data: 2 2
12671 041/300 Read block 1707 len 36864
12672 042/300 Write block 4176 len 32768 data: 2 2 2 3 3 4 4 4
12673 043/300 Read block 3492 len 40960
12674 044/300 Write block 3104 len 28672 data: 3 3 2 2 2 2 2
12675 045/300 Read block 719 len 32768
12676 046/300 Read block 476 len 4096
12677 047/300 Write block 4320 len 32768 data: 2 2 2 3 3 3 3 4
12678 048/300 Write block 1185 len 32768 data: 2 2 2 2 3 4 4 4
12679 049/300 Read block 3953 len 12288
12680 050/300 Read block 101 len 40960
12681 051/300 Read block 941 len 32768
12682 052/300 Read block 3494 len 24576
12683 053/300 Write block 3388 len 24576 data: 5 4 3 3 3 3
12684 054/300 Read block 950 len 16384
12685 055/300 Read block 3099 len 16384
12686 056/300 Read block 1356 len 24576
12687 057/300 Read block 423 len 20480
12688 058/300 Read block 2020 len 40960
12689 059/300 Read block 1701 len 8192
12690 060/300 Write block 1594 len 32768 data: 6 7 7 6 5 5 5 5
12691 061/300 Read block 723 len 8192
12692 062/300 Write block 721 len 32768 data: 3 3 4 4 4 5 4 5
12693 063/300 Write block 2205 len 24576 data: 5 4 3 3 2 3
12694 064/300 Flush
12695 065/300 Read block 3480 len 16384
12696 066/300 Write block 4344 len 20480 data: 4 4 5 3 3
12697 067/300 Write block 2876 len 16384 data: 3 4 4 5
12698 068/300 Write block 2811 len 8192 data: 3 2
12699 069/300 Read block 2763 len 32768
12700 070/300 Read block 3679 len 24576
12701 071/300 Write block 1532 len 8192 data: 5 7
12702 072/300 Write block 3785 len 40960 data: 4 3 3 3 2 2 2 3 3 3
12703 073/300 Write block 1626 len 40960 data: 3 3 2 2 2 2 2 2 2 2
12704 074/300 Read block 245 len 4096
12705 075/300 Flush
12706 076/300 Read block 12 len 36864
12707 077/300 Write block 303 len 8192 data: 4 4
12708 078/300 Write block 1289 len 24576 data: 2 2 3 5 5 5
12709 079/300 Read block 1241 len 12288
12710 080/300 Read block 1496 len 24576
12711 081/300 Write block 766 len 24576 data: 4 5 5 5 4 5
12712 082/300 Write block 55 len 32768 data: 5 5 4 3 2 2 2 2
12713 083/300 Read block 810 len 28672
12714 084/300 Read block 1726 len 4096
12715 085/300 Read block 1579 len 40960
12716 086/300 Read block 1153 len 28672
12717 087/300 Read block 4600 len 24576
12718 088/300 Read block 3341 len 36864
12719 089/300 Flush
12720 090/300 Write block 3338 len 40960 data: 2 2 3 3 3 4 4 4 4 4
12721 091/300 Read block 2527 len 28672
12722 092/300 Flush
12723 093/300 Read block 281 len 16384
12724 094/300 Read block 2849 len 16384
12725 095/300 Write block 3930 len 24576 data: 3 3 4 4 4 4
12726 096/300 Write block 3392 len 36864 data: 4 4 4 4 3 4 4 4 5
12727 097/300 Write block 1159 len 40960 data: 3 4 4 5 4 4 2 2 2 2
12728 098/300 Read block 2269 len 40960
12729 099/300 Read block 1997 len 40960
12730 100/300 Read block 9 len 32768
12731 101/300 Read block 4220 len 8192
12732 102/300 Read block 361 len 24576
12733 103/300 Read block 268 len 16384
12734 104/300 Read block 701 len 16384
12735 105/300 Write block 2747 len 36864 data: 3 4 5 5 4 4 4 3 3
12736 106/300 Write block 1141 len 28672 data: 3 3 2 2 2 2 2
12737 107/300 Write block 468 len 8192 data: 3 3
12738 108/300 Read block 1823 len 8192
12739 109/300 Write block 4528 len 40960 data: 5 5 4 4 4 5 3 4 4 4
12740 110/300 Read block 1460 len 20480
12741 111/300 Write block 2552 len 24576 data: 4 4 4 4 5 4
12742 112/300 Read block 503 len 8192
12743 113/300 Write block 804 len 20480 data: 2 2 2 2 3
12744 114/300 Read block 2840 len 20480
12745 115/300 Write block 534 len 8192 data: 6 6
12746 116/300 Flush
12747 117/300 Read block 438 len 12288
12748 118/300 Read block 1555 len 12288
12749 119/300 Flush
12750 120/300 Read block 3710 len 40960
12751 121/300 Read block 4467 len 8192
12752 122/300 Read block 3385 len 40960
12753 123/300 Write block 842 len 24576 data: 2 2 2 4 3 2
12754 124/300 Read block 1997 len 12288
12755 125/300 Flush
12756 126/300 Read block 2507 len 16384
12757 127/300 Read block 4197 len 28672
12758 128/300 Read block 3318 len 40960
12759 129/300 Write block 3214 len 36864 data: 6 6 5 5 5 4 2 2 2
12760 130/300 Read block 4793 len 32768
12761 131/300 Read block 1734 len 36864
12762 132/300 Read block 3034 len 28672
12763 133/300 Read block 2623 len 28672
12764 134/300 Read block 3692 len 28672
12765 135/300 Read block 2081 len 20480
12766 136/300 Write block 3758 len 8192 data: 5 5
12767 137/300 Write block 1831 len 8192 data: 5 5
12768 138/300 Write block 278 len 4096 data: 3
12769 139/300 Read block 2225 len 12288
12770 140/300 Write block 3336 len 36864 data: 2 2 3 3 4 4 4 5 5
12771 141/300 Write block 1919 len 32768 data: 4 4 4 3 4 4 4 5
12772 142/300 Write block 314 len 36864 data: 6 6 6 5 3 3 3 2 3
12773 143/300 Read block 2330 len 28672
12774 144/300 Read block 2820 len 28672
12775 145/300 Read block 4901 len 32768
12776 146/300 Flush
12777 147/300 Read block 1571 len 12288
12778 148/300 Write block 4867 len 40960 data: 3 3 4 3 4 4 3 3 2 3
12779 149/300 Read block 4674 len 40960
12780 150/300 Read block 2537 len 4096
12781 151/300 Write block 2339 len 12288 data: 3 3 3
12782 152/300 Write block 2466 len 40960 data: 5 4 4 3 4 3 3 3 4 4
12783 153/300 Write block 1011 len 16384 data: 2 2 2 2
12784 154/300 Read block 211 len 16384
12785 155/300 Write block 3492 len 24576 data: 2 2 2 2 2 2
12786 156/300 Write block 2071 len 20480 data: 4 4 4 5 5
12787 157/300 Read block 4244 len 12288
12788 158/300 Write block 234 len 20480 data: 4 4 3 4 5
12789 159/300 Read block 3806 len 20480
12790 160/300 Write block 4966 len 40960 data: 3 3 3 3 3 3 3 3 2 2
12791 161/300 Write block 183 len 32768 data: 2 2 2 2 2 2 2 2
12792 162/300 Flush
12793 163/300 Read block 3801 len 40960
12794 164/300 Read block 2006 len 8192
12795 165/300 Read block 4512 len 12288
12796 166/300 Flush
12797 167/300 Write block 1867 len 24576 data: 4 3 3 4 4 4
12798 168/300 Read block 3481 len 8192
12799 169/300 Write block 4233 len 12288 data: 5 4 4
12800 170/300 Read block 478 len 32768
12801 171/300 Write block 3687 len 16384 data: 3 3 3 3
12802 172/300 Read block 4658 len 40960
12803 173/300 Write block 1288 len 8192 data: 2 3
12804 174/300 Read block 3115 len 40960
12805 175/300 Write block 2748 len 36864 data: 5 6 6 5 5 5 4 4 3
12806 176/300 Write block 666 len 36864 data: 4 3 3 3 3 4 4 3 3
12807 177/300 Write block 892 len 16384 data: 2 2 3 2
12808 178/300 Flush
12809 179/300 Read block 269 len 40960
12810 180/300 Write block 1955 len 40960 data: 3 3 4 4 3 3 2 2 2 2
12811 181/300 Read block 4892 len 16384
12812 182/300 Write block 3470 len 16384 data: 2 3 3 2
12813 183/300 Read block 4756 len 12288
12814 184/300 Write block 4058 len 12288 data: 3 3 3
12815 185/300 Write block 3186 len 20480 data: 4 3 4 4 3
12816 186/300 Read block 1716 len 40960
12817 187/300 Read block 1554 len 4096
12818 188/300 Flush
12819 189/300 Read block 2607 len 16384
12820 190/300 Write block 3210 len 12288 data: 7 7 8
12821 191/300 Read block 180 len 12288
12822 192/300 Read block 2528 len 36864
12823 193/300 Read block 4662 len 20480
12824 194/300 Read block 2076 len 12288
12825 195/300 Write block 1897 len 28672 data: 4 4 3 4 4 4 3
12826 196/300 Read block 2343 len 28672
12827 197/300 Read block 2876 len 40960
12828 198/300 Read block 4368 len 12288
12829 199/300 Read block 2210 len 24576
12830 200/300 Read block 3358 len 40960
12831 201/300 Write block 2532 len 12288 data: 4 3 4
12832 202/300 Read block 1445 len 12288
12833 203/300 Write block 2714 len 36864 data: 3 2 2 2 3 3 3 3 3
12834 204/300 Read block 2269 len 32768
12835 205/300 Flush
12836 206/300 Write block 554 len 40960 data: 3 4 4 2 2 2 2 2 3 4
12837 207/300 Write block 1687 len 40960 data: 3 3 3 3 3 3 2 2 2 2
12838 208/300 Read block 4627 len 24576
12839 209/300 Read block 3692 len 36864
12840 210/300 Read block 1612 len 36864
12841 211/300 Read block 4713 len 16384
12842 212/300 Read block 4733 len 32768
12843 213/300 Read block 1079 len 16384
12844 214/300 Read block 872 len 16384
12845 215/300 Write block 4402 len 8192 data: 2 2
12846 216/300 Read block 4810 len 40960
12847 217/300 Read block 1450 len 24576
12848 218/300 Read block 773 len 28672
12849 219/300 Read block 2158 len 40960
12850 220/300 Read block 45 len 24576
12851 221/300 Flush
12852 222/300 Write block 4742 len 16384 data: 3 4 5 5
12853 223/300 Read block 1680 len 8192
12854 224/300 Read block 3549 len 28672
12855 225/300 Read block 2208 len 32768
12856 226/300 Write block 4838 len 32768 data: 3 2 3 2 3 3 3 3
12857 227/300 Read block 4143 len 36864
12858 228/300 Read block 3752 len 12288
12859 229/300 Flush
12860 230/300 Write block 1018 len 12288 data: 2 2 2
12861 231/300 Write block 4956 len 12288 data: 3 2 2
12862 232/300 Read block 3016 len 12288
12863 233/300 Read block 3546 len 8192
12864 234/300 Write block 539 len 32768 data: 3 3 3 3 2 2 2 2
12865 235/300 Read block 3552 len 20480
12866 236/300 Write block 2815 len 8192 data: 3 3
12867 237/300 Read block 1687 len 4096
12868 238/300 Read block 4132 len 36864
12869 239/300 Read block 4436 len 36864
12870 240/300 Write block 2389 len 8192 data: 2 2
12871 241/300 Read block 459 len 20480
12872 242/300 Flush
12873 243/300 Read block 1480 len 16384
12874 244/300 Write block 123 len 28672 data: 6 6 7 6 5 4 5
12875 245/300 Read block 2242 len 40960
12876 246/300 Write block 8 len 24576 data: 2 2 2 2 2 2
12877 247/300 Read block 2938 len 36864
12878 248/300 Write block 2424 len 20480 data: 3 3 3 3 2
12879 249/300 Read block 4833 len 28672
12880 250/300 Read block 4779 len 4096
12881 251/300 Read block 2918 len 12288
12882 252/300 Write block 3390 len 40960 data: 4 4 5 5 5 5 4 5 5 5
12883 253/300 Read block 2844 len 28672
12884 254/300 Flush
12885 255/300 Read block 1812 len 20480
12886 256/300 Flush
12887 257/300 Read block 4286 len 28672
12888 258/300 Write block 2943 len 36864 data: 3 3 4 4 4 3 3 3 4
12889 259/300 Write block 4599 len 20480 data: 3 3 4 4 4
12890 260/300 Read block 2693 len 32768
12891 261/300 Read block 2788 len 16384
12892 262/300 Write block 2598 len 24576 data: 3 3 3 3 3 3
12893 263/300 Write block 3589 len 12288 data: 3 3 2
12894 264/300 Write block 3592 len 16384 data: 2 3 3 3
12895 265/300 Write block 2440 len 36864 data: 5 5 4 3 3 2 2 2 2
12896 266/300 Write block 4404 len 36864 data: 2 2 3 4 4 4 4 3 3
12897 267/300 Read block 4324 len 40960
12898 268/300 Write block 2215 len 12288 data: 3 3 2
12899 269/300 Write block 1634 len 36864 data: 3 3 2 3 3 3 3 2 2
12900 270/300 Read block 1034 len 8192
12901 271/300 Read block 2754 len 24576
12902 272/300 Flush
12903 273/300 Read block 3469 len 32768
12904 274/300 Write block 2577 len 20480 data: 6 6 6 6 5
12905 275/300 Write block 3783 len 32768 data: 5 3 5 4 4 4 3 3
12906 276/300 Write block 1317 len 40960 data: 2 2 3 3 3 3 3 3 3 4
12907 277/300 Read block 3908 len 24576
12908 278/300 Read block 1248 len 12288
12909 279/300 Write block 1998 len 24576 data: 3 4 4 4 4 5
12910 280/300 Write block 4877 len 8192 data: 3 2
12911 281/300 Read block 37 len 40960
12912 282/300 Read block 2686 len 20480
12913 283/300 Write block 456 len 20480 data: 2 2 2 2 3
12914 284/300 Write block 3665 len 4096 data: 2
12915 285/300 Write block 2802 len 32768 data: 2 2 4 4 4 4 3 3
12916 286/300 Read block 1855 len 4096
12917 287/300 Read block 1591 len 16384
12918 288/300 Flush
12919 289/300 Read block 2356 len 32768
12920 290/300 Write block 2608 len 8192 data: 3 4
12921 291/300 Read block 500 len 32768
12922 292/300 Write block 2240 len 20480 data: 2 2 2 2 2
12923 293/300 Write block 3406 len 12288 data: 2 2 2
12924 294/300 Write block 2696 len 24576 data: 2 2 2 2 2 2
12925 295/300 Read block 4397 len 8192
12926 296/300 Flush
12927 297/300 Read block 3240 len 4096
12928 298/300 Read block 3722 len 16384
12929 299/300 Write block 3987 len 4096 data: 6
12930 300/300 Read block 1567 len 12288
12931 301/300 Read block 1114 len 16384
12932 Main task has work to do, go find it
12933 got dsc Start(2)
12934 start 2
12935 Mar 17 22:37:27.640 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2daab2ad-4660-43ba-936e-7b13197f8081, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
12936 Replay: started 2, returned:Ok(())
12937 ----------------------------------------------------------------
12938 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12939 States: Active Active Offline
12940 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12941 4020 Acked Read 5 Done Done Sent false
12942 4021 Acked Write 6 Done Done Sent false
12943 4022 Acked Flush 0 Done Done Sent false
12944 4023 Acked Write 4 Done Done Sent false
12945 4024 Acked Read 3 Done Done Sent false
12946 4025 Acked Write 6 Done Done Sent false
12947 4026 Acked Write 4 Done Done Sent false
12948 4027 Acked Write 10 Done Done Sent false
12949 4028 Acked Read 5 Done Done Sent false
12950 4029 Acked Write 10 Done Done Sent false
12951 4030 Acked Write 2 Done Done Sent false
12952 4031 Acked Write 7 Done Done Sent false
12953 4032 Acked Read 2 Done Done Sent false
12954 4033 Acked Write 7 Done Done Sent false
12955 4034 Acked Read 7 Done Done Sent false
12956 4035 Acked Write 1 Done Done Sent false
12957 4036 Acked Read 6 Done Done Sent false
12958 4037 Acked Read 5 Done Done Sent false
12959 4038 Acked Read 9 Done Done Sent false
12960 4039 Acked Write 6 Done Done Sent false
12961 4040 Acked Write 5 Done Done Sent false
12962 4041 Acked Read 1 Done Done Sent false
12963 4042 Acked Read 4 Done Done Sent false
12964 4043 Acked Read 5 Done Done Sent false
12965 4044 Acked Write 3 Done Done Sent false
12966 4045 Acked Read 4 Done Done Sent false
12967 4046 Acked Write 9 Done Done Sent false
12968 4047 Acked Write 6 Done Done Sent false
12969 4048 Acked Write 7 Done Done Sent false
12970 4049 Acked Read 7 Done Done Sent false
12971 4050 Acked Read 2 Done Done Sent false
12972 4051 Acked Read 1 Done Done Sent false
12973 4052 Acked Read 6 Done Done Sent false
12974 4053 Acked Write 3 Done Done Sent false
12975 4054 Acked Read 1 Done Done Sent false
12976 4055 Acked Write 5 Done Done Sent false
12977 4056 Acked Flush 0 Done Done Sent false
12978 4057 Acked Read 10 Done Done Sent false
12979 4058 Acked Write 2 Done Done Sent false
12980 4059 Acked Write 2 Done Done Sent false
12981 4060 Acked Read 9 Done Done Sent false
12982 4061 Acked Write 8 Done Done Sent false
12983 4062 Acked Read 10 Done Done Sent false
12984 4063 Acked Write 7 Done Done Sent false
12985 4064 Acked Read 8 Done Done Sent false
12986 4065 Acked Read 1 Done Done Sent false
12987 4066 Acked Write 8 Done Done Sent false
12988 4067 Acked Write 8 Done Done Sent false
12989 4068 Acked Read 3 Done Done Sent false
12990 4069 Acked Read 10 Done Done Sent false
12991 4070 Acked Read 8 Done Done Sent false
12992 4071 Acked Read 6 Done Done Sent false
12993 4072 Acked Write 6 Done Done Sent false
12994 4073 Acked Read 4 Done Done Sent false
12995 4074 Acked Read 4 Done Done Sent false
12996 4075 Acked Read 6 Done Done Sent false
12997 4076 Acked Read 5 Done Done Sent false
12998 4077 Acked Read 10 Done Done Sent false
12999 4078 Acked Read 2 Done Done Sent false
13000 4079 Acked Write 8 Done Done Sent false
13001 4080 Acked Read 2 Done Done Sent false
13002 4081 Acked Write 8 Done Done Sent false
13003 4082 Acked Write 6 Done Done Sent false
13004 4083 Acked Flush 0 Done Done Sent false
13005 4084 Acked Read 4 Done Done Sent false
13006 4085 Acked Write 5 Done Done Sent false
13007 4086 Acked Write 4 Done Done Sent false
13008 4087 Acked Write 2 Done Done Sent false
13009 4088 Acked Read 8 Done Done Sent false
13010 4089 Acked Read 6 Done Done Sent false
13011 4090 Acked Write 2 Done Done Sent false
13012 4091 Acked Write 10 Done Done Sent false
13013 4092 Acked Write 10 Done Done Sent false
13014 4093 Acked Read 1 Done Done Sent false
13015 4094 Acked Flush 0 Done Done Sent false
13016 4095 Acked Read 9 Done Done Sent false
13017 4096 Acked Write 2 Done Done Sent false
13018 4097 Acked Write 6 Done Done Sent false
13019 4098 Acked Read 3 Done Done Sent false
13020 4099 Acked Read 6 Done Done Sent false
13021 4100 Acked Write 6 Done Done Sent false
13022 4101 Acked Write 8 Done Done Sent false
13023 4102 Acked Read 7 Done Done Sent false
13024 4103 Acked Read 1 Done Done Sent false
13025 4104 Acked Read 10 Done Done Sent false
13026 4105 Acked Read 7 Done Done Sent false
13027 4106 Acked Read 6 Done Done Sent false
13028 4107 Acked Read 9 Done Done Sent false
13029 4108 Acked Flush 0 Done Done Sent false
13030 4109 Acked Write 10 Done Done Sent false
13031 4110 Acked Read 7 Done Done Sent false
13032 4111 Acked Flush 0 Done Done Sent false
13033 4112 Acked Read 4 Done Done Sent false
13034 4113 Acked Read 4 Done Done Sent false
13035 4114 Acked Write 6 Done Done Sent false
13036 4115 Acked Write 9 Done Done Sent false
13037 4116 Acked Write 10 Done Done Sent false
13038 4117 Acked Read 10 Done Done Sent false
13039 4118 Acked Read 10 Done Done Sent false
13040 4119 Acked Read 8 Done Done Sent false
13041 4120 Acked Read 2 Done Done Sent false
13042 4121 Acked Read 6 Done Done Sent false
13043 4122 Acked Read 4 Done Done Sent false
13044 4123 Acked Read 4 Done Done Sent false
13045 4124 Acked Write 9 Done Done Sent false
13046 4125 Acked Write 7 Done Done Sent false
13047 4126 Acked Write 2 Done Done Sent false
13048 4127 Acked Read 2 Done Done Sent false
13049 4128 Acked Write 10 Done Done Sent false
13050 4129 Acked Read 5 Done Done Sent false
13051 4130 Acked Write 6 Done Done Sent false
13052 4131 Acked Read 2 Done Done Sent false
13053 4132 Acked Write 5 Done Done Sent false
13054 4133 Acked Read 5 Done Done Sent false
13055 4134 Acked Write 2 Done Done Sent false
13056 4135 Acked Flush 0 Done Done Sent false
13057 4136 Acked Read 3 Done Done Sent false
13058 4137 Acked Read 3 Done Done Sent false
13059 4138 Acked Flush 0 Done Done Sent false
13060 4139 Acked Read 10 Done Done Sent false
13061 4140 Acked Read 2 Done Done Sent false
13062 4141 Acked Read 10 Done Done Sent false
13063 4142 Acked Write 6 Done Done Sent false
13064 4143 Acked Read 3 Done Done Sent false
13065 4144 Acked Flush 0 Done Done Sent false
13066 4145 Acked Read 4 Done Done Sent false
13067 4146 Acked Read 7 Done Done Sent false
13068 4147 Acked Read 10 Done Done Sent false
13069 4148 Acked Write 9 Done Done Sent false
13070 4149 Acked Read 8 Done Done Sent false
13071 4150 Acked Read 9 Done Done Sent false
13072 4151 Acked Read 7 Done Done Sent false
13073 4152 Acked Read 7 Done Done Sent false
13074 4153 Acked Read 7 Done Done Sent false
13075 4154 Acked Read 5 Done Done Sent false
13076 4155 Acked Write 2 Done Done Sent false
13077 4156 Acked Write 2 Done Done Sent false
13078 4157 Acked Write 1 Done Done Sent false
13079 4158 Acked Read 3 Done Done Sent false
13080 4159 Acked Write 9 Done Done Sent false
13081 4160 Acked Write 8 Done Done Sent false
13082 4161 Acked Write 9 Done Done Sent false
13083 4162 Acked Read 7 Done Done Sent false
13084 4163 Acked Read 7 Done Done Sent false
13085 4164 Acked Read 8 Done Done Sent false
13086 4165 Acked Flush 0 Done Done Sent false
13087 4166 Acked Read 3 Done Done Sent false
13088 4167 Acked Write 10 Done Done Sent false
13089 4168 Acked Read 10 Done Done Sent false
13090 4169 Acked Read 1 Done Done Sent false
13091 4170 Acked Write 3 Done Done Sent false
13092 4171 Acked Write 10 Done Done Sent false
13093 4172 Acked Write 4 Done Done Sent false
13094 4173 Acked Read 4 Done Done Sent false
13095 4174 Acked Write 6 Done Done Sent false
13096 4175 Acked Write 5 Done Done Sent false
13097 4176 Acked Read 3 Done Done Sent false
13098 4177 Acked Write 5 Done Done Sent false
13099 4178 Acked Read 5 Done Done Sent false
13100 4179 Acked Write 10 Done Done Sent false
13101 4180 Acked Write 8 Done Done Sent false
13102 4181 Acked Flush 0 Done Done Sent false
13103 4182 Acked Read 10 Done Done Sent false
13104 4183 Acked Read 2 Done Done Sent false
13105 4184 Acked Read 3 Done Done Sent false
13106 4185 Acked Flush 0 Done Done Sent false
13107 4186 Acked Write 6 Done Done Sent false
13108 4187 Acked Read 2 Done Done Sent false
13109 4188 Acked Write 3 Done Done Sent false
13110 4189 Acked Read 8 Done Done Sent false
13111 4190 Acked Write 4 Done Done Sent false
13112 4191 Acked Read 10 Done Done Sent false
13113 4192 Acked Write 2 Done Done Sent false
13114 4193 Acked Read 10 Done Done Sent false
13115 4194 Acked Write 9 Done Done Sent false
13116 4195 Acked Write 9 Done Done Sent false
13117 4196 Acked Write 4 Done Done Sent false
13118 4197 Acked Flush 0 Done Done Sent false
13119 4198 Acked Read 10 Done Done Sent false
13120 4199 Acked Write 10 Done Done Sent false
13121 4200 Acked Read 4 Done Done Sent false
13122 4201 Acked Write 4 Done Done Sent false
13123 4202 Acked Read 3 Done Done Sent false
13124 4203 Acked Write 3 Done Done Sent false
13125 4204 Acked Write 5 Done Done Sent false
13126 4205 Acked Read 10 Done Done Sent false
13127 4206 Acked Read 1 Done Done Sent false
13128 4207 Acked Flush 0 Done Done Sent false
13129 4208 Acked Read 4 Done Done Sent false
13130 4209 Acked Write 3 Done Done Sent false
13131 4210 Acked Read 3 Done Done Sent false
13132 4211 Acked Read 9 Done Done Sent false
13133 4212 Acked Read 5 Done Done Sent false
13134 4213 Acked Read 3 Done Done Sent false
13135 4214 Acked Write 7 Done Done Sent false
13136 4215 Acked Read 7 Done Done Sent false
13137 4216 Acked Read 10 Done Done Sent false
13138 4217 Acked Read 3 Done Done Sent false
13139 4218 Acked Read 6 Done Done Sent false
13140 4219 Acked Read 10 Done Done Sent false
13141 4220 Acked Write 3 Done Done Sent false
13142 4221 Acked Read 3 Done Done Sent false
13143 4222 Acked Write 9 Done Done Sent false
13144 4223 Acked Read 8 Done Done Sent false
13145 4224 Acked Flush 0 Done Done Sent false
13146 4225 Acked Write 10 Done Done Sent false
13147 4226 Acked Write 10 Done Done Sent false
13148 4227 Acked Read 6 Done Done Sent false
13149 4228 Acked Read 9 Done Done Sent false
13150 4229 Acked Read 9 Done Done Sent false
13151 4230 Acked Read 4 Done Done Sent false
13152 4231 Acked Read 8 Done Done Sent false
13153 4232 Acked Read 4 Done Done Sent false
13154 4233 Acked Read 4 Done Done Sent false
13155 4234 Acked Write 2 Done Done Sent false
13156 4235 Acked Read 10 Done Done Sent false
13157 4236 Acked Read 6 Done Done Sent false
13158 4237 Acked Read 7 Done Done Sent false
13159 4238 Acked Read 10 Done Done Sent false
13160 4239 Acked Read 6 Done Done Sent false
13161 4240 Acked Flush 0 Done Done Sent false
13162 4241 Acked Write 4 Done Done Sent false
13163 4242 Acked Read 2 Done Done Sent false
13164 4243 Acked Read 7 Done Done Sent false
13165 4244 Acked Read 8 Done Done Sent false
13166 4245 Acked Write 8 Done Done Sent false
13167 4246 Acked Read 9 Done Done Sent false
13168 4247 Acked Read 3 Done Done Sent false
13169 4248 Acked Flush 0 Done Done Sent false
13170 4249 Acked Write 3 Done Done Sent false
13171 4250 Acked Write 3 Done Done Sent false
13172 4251 Acked Read 3 Done Done Sent false
13173 4252 Acked Read 2 Done Done Sent false
13174 4253 Acked Write 8 Done Done Sent false
13175 4254 Acked Read 5 Done Done Sent false
13176 4255 Acked Write 2 Done Done Sent false
13177 4256 Acked Read 1 Done Done Sent false
13178 4257 Acked Read 9 Done Done Sent false
13179 4258 Acked Read 9 Done Done Sent false
13180 4259 Acked Write 2 Done Done Sent false
13181 4260 Acked Read 5 Done Done Sent false
13182 4261 Acked Flush 0 Done Done Sent false
13183 4262 Acked Read 4 Done Done Sent false
13184 4263 Acked Write 7 Done Done Sent false
13185 4264 Acked Read 10 Done Done Sent false
13186 4265 Acked Write 6 Done Done Sent false
13187 4266 Acked Read 9 Done Done Sent false
13188 4267 Acked Write 5 Done Done Sent false
13189 4268 Acked Read 7 Done Done Sent false
13190 4269 Acked Read 1 Done Done Sent false
13191 4270 Acked Read 3 Done Done Sent false
13192 4271 Acked Write 10 Done Done Sent false
13193 4272 Acked Read 7 Done Done Sent false
13194 4273 Acked Flush 0 Done Done Sent false
13195 4274 Acked Read 5 Done Done Sent false
13196 4275 Acked Flush 0 Done Done Sent false
13197 4276 Acked Read 7 Done Done Sent false
13198 4277 Acked Write 9 Done Done Sent false
13199 4278 Acked Write 5 Done Done Sent false
13200 4279 Acked Read 8 Done Done Sent false
13201 4280 Acked Read 4 Done Done Sent false
13202 4281 Acked Write 6 Done Done Sent false
13203 4282 Acked Write 3 Done Done Sent false
13204 4283 Acked Write 4 Done Done Sent false
13205 4284 Acked Write 9 Done Done Sent false
13206 4285 Acked Write 9 Done Done Sent false
13207 4286 Acked Read 10 Done Done Sent false
13208 4287 Acked Write 3 Done Done Sent false
13209 4288 Acked Write 9 Done Done Sent false
13210 4289 Acked Read 2 Done Done Sent false
13211 4290 Acked Read 6 Done Done Sent false
13212 4291 Acked Flush 0 Done Done Sent false
13213 4292 Acked Read 8 Done Done Sent false
13214 4293 Acked Write 5 Done Done Sent false
13215 4294 Acked Write 8 Done Done Sent false
13216 4295 Acked Write 10 Done Done Sent false
13217 4296 Acked Read 6 Done Done Sent false
13218 4297 Acked Read 3 Done Done Sent false
13219 4298 Acked Write 6 Done Done Sent false
13220 4299 Acked Write 2 Done Done Sent false
13221 4300 Acked Read 10 Done Done Sent false
13222 4301 Acked Read 5 Done Done Sent false
13223 4302 Acked Write 5 Done Done Sent false
13224 4303 Acked Write 1 Done Done Sent false
13225 4304 Acked Write 8 Done Done Sent false
13226 4305 Acked Read 1 Done Done Sent false
13227 4306 Acked Read 4 Done Done Sent false
13228 4307 Acked Flush 0 Done Done Sent false
13229 4308 Acked Read 8 Done Done Sent false
13230 4309 Acked Write 2 Done Done Sent false
13231 4310 Acked Read 8 Done Done Sent false
13232 4311 Acked Write 5 Done Done Sent false
13233 4312 Acked Write 3 Done Done Sent false
13234 4313 Acked Write 6 Done Done Sent false
13235 4314 Acked Read 2 Done Done Sent false
13236 4315 Acked Flush 0 Done Done Sent false
13237 4316 Acked Read 1 Done Done Sent false
13238 4317 Acked Read 4 Done Done Sent false
13239 4318 Acked Write 1 Done Done Sent false
13240 4319 Acked Read 3 Done Done Sent false
13241 4320 Acked Read 4 Done Done Sent false
13242 STATES DS:0 DS:1 DS:2 TOTAL
13243 Sent 0 0 301 301
13244 Done 301 301 0 602
13245 Skipped 0 0 0 0
13246 Error 0 0 0 0
13247 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4019))
13248 Downstairs last five completed: 4019 4018 4017 4016 4015
13249 Upstairs last five completed: 4320 4319 4318 4317 4316
13250 CLIENT: Up:0 ds:301 act:2
132512025-03-17T22:37:28.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
132522025-03-17T22:37:29.586ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
132532025-03-17T22:37:30.588ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
13254 [8830] got start action
13255 [8830] I am going to restart
13256 Starting downstairs at port 8830
13257 Make output file at "/tmp/dsc/downstairs-8830.txt"
13258 [8830][2] reports Starting
13259 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1219)
13260 [8830][2] reports Running
132612025-03-17T22:37:31.590ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
13262 ----------------------------------------------------------------
13263 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13264 States: Active Active Offline
13265 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13266 4020 Acked Read 5 Done Done Sent false
13267 4021 Acked Write 6 Done Done Sent false
13268 4022 Acked Flush 0 Done Done Sent false
13269 4023 Acked Write 4 Done Done Sent false
13270 4024 Acked Read 3 Done Done Sent false
13271 4025 Acked Write 6 Done Done Sent false
13272 4026 Acked Write 4 Done Done Sent false
13273 4027 Acked Write 10 Done Done Sent false
13274 4028 Acked Read 5 Done Done Sent false
13275 4029 Acked Write 10 Done Done Sent false
13276 4030 Acked Write 2 Done Done Sent false
13277 4031 Acked Write 7 Done Done Sent false
13278 4032 Acked Read 2 Done Done Sent false
13279 4033 Acked Write 7 Done Done Sent false
13280 4034 Acked Read 7 Done Done Sent false
13281 4035 Acked Write 1 Done Done Sent false
13282 4036 Acked Read 6 Done Done Sent false
13283 4037 Acked Read 5 Done Done Sent false
13284 4038 Acked Read 9 Done Done Sent false
13285 4039 Acked Write 6 Done Done Sent false
13286 4040 Acked Write 5 Done Done Sent false
13287 4041 Acked Read 1 Done Done Sent false
13288 4042 Acked Read 4 Done Done Sent false
13289 4043 Acked Read 5 Done Done Sent false
13290 4044 Acked Write 3 Done Done Sent false
13291 4045 Acked Read 4 Done Done Sent false
13292 4046 Acked Write 9 Done Done Sent false
13293 4047 Acked Write 6 Done Done Sent false
13294 4048 Acked Write 7 Done Done Sent false
13295 4049 Acked Read 7 Done Done Sent false
13296 4050 Acked Read 2 Done Done Sent false
13297 4051 Acked Read 1 Done Done Sent false
13298 4052 Acked Read 6 Done Done Sent false
13299 4053 Acked Write 3 Done Done Sent false
13300 4054 Acked Read 1 Done Done Sent false
13301 4055 Acked Write 5 Done Done Sent false
13302 4056 Acked Flush 0 Done Done Sent false
13303 4057 Acked Read 10 Done Done Sent false
13304 4058 Acked Write 2 Done Done Sent false
13305 4059 Acked Write 2 Done Done Sent false
13306 4060 Acked Read 9 Done Done Sent false
13307 4061 Acked Write 8 Done Done Sent false
13308 4062 Acked Read 10 Done Done Sent false
13309 4063 Acked Write 7 Done Done Sent false
13310 4064 Acked Read 8 Done Done Sent false
13311 4065 Acked Read 1 Done Done Sent false
13312 4066 Acked Write 8 Done Done Sent false
13313 4067 Acked Write 8 Done Done Sent false
13314 4068 Acked Read 3 Done Done Sent false
13315 4069 Acked Read 10 Done Done Sent false
13316 4070 Acked Read 8 Done Done Sent false
13317 4071 Acked Read 6 Done Done Sent false
13318 4072 Acked Write 6 Done Done Sent false
13319 4073 Acked Read 4 Done Done Sent false
13320 4074 Acked Read 4 Done Done Sent false
13321 4075 Acked Read 6 Done Done Sent false
13322 4076 Acked Read 5 Done Done Sent false
13323 4077 Acked Read 10 Done Done Sent false
13324 4078 Acked Read 2 Done Done Sent false
13325 4079 Acked Write 8 Done Done Sent false
13326 4080 Acked Read 2 Done Done Sent false
13327 4081 Acked Write 8 Done Done Sent false
13328 4082 Acked Write 6 Done Done Sent false
13329 4083 Acked Flush 0 Done Done Sent false
13330 4084 Acked Read 4 Done Done Sent false
13331 4085 Acked Write 5 Done Done Sent false
13332 4086 Acked Write 4 Done Done Sent false
13333 4087 Acked Write 2 Done Done Sent false
13334 4088 Acked Read 8 Done Done Sent false
13335 4089 Acked Read 6 Done Done Sent false
13336 4090 Acked Write 2 Done Done Sent false
13337 4091 Acked Write 10 Done Done Sent false
13338 4092 Acked Write 10 Done Done Sent false
13339 4093 Acked Read 1 Done Done Sent false
13340 4094 Acked Flush 0 Done Done Sent false
13341 4095 Acked Read 9 Done Done Sent false
13342 4096 Acked Write 2 Done Done Sent false
13343 4097 Acked Write 6 Done Done Sent false
13344 4098 Acked Read 3 Done Done Sent false
13345 4099 Acked Read 6 Done Done Sent false
13346 4100 Acked Write 6 Done Done Sent false
13347 4101 Acked Write 8 Done Done Sent false
13348 4102 Acked Read 7 Done Done Sent false
13349 4103 Acked Read 1 Done Done Sent false
13350 4104 Acked Read 10 Done Done Sent false
13351 4105 Acked Read 7 Done Done Sent false
13352 4106 Acked Read 6 Done Done Sent false
13353 4107 Acked Read 9 Done Done Sent false
13354 4108 Acked Flush 0 Done Done Sent false
13355 4109 Acked Write 10 Done Done Sent false
13356 4110 Acked Read 7 Done Done Sent false
13357 4111 Acked Flush 0 Done Done Sent false
13358 4112 Acked Read 4 Done Done Sent false
13359 4113 Acked Read 4 Done Done Sent false
13360 4114 Acked Write 6 Done Done Sent false
13361 4115 Acked Write 9 Done Done Sent false
13362 4116 Acked Write 10 Done Done Sent false
13363 4117 Acked Read 10 Done Done Sent false
13364 4118 Acked Read 10 Done Done Sent false
13365 4119 Acked Read 8 Done Done Sent false
13366 4120 Acked Read 2 Done Done Sent false
13367 4121 Acked Read 6 Done Done Sent false
13368 4122 Acked Read 4 Done Done Sent false
13369 4123 Acked Read 4 Done Done Sent false
13370 4124 Acked Write 9 Done Done Sent false
13371 4125 Acked Write 7 Done Done Sent false
13372 4126 Acked Write 2 Done Done Sent false
13373 4127 Acked Read 2 Done Done Sent false
13374 4128 Acked Write 10 Done Done Sent false
13375 4129 Acked Read 5 Done Done Sent false
13376 4130 Acked Write 6 Done Done Sent false
13377 4131 Acked Read 2 Done Done Sent false
13378 4132 Acked Write 5 Done Done Sent false
13379 4133 Acked Read 5 Done Done Sent false
13380 4134 Acked Write 2 Done Done Sent false
13381 4135 Acked Flush 0 Done Done Sent false
13382 4136 Acked Read 3 Done Done Sent false
13383 4137 Acked Read 3 Done Done Sent false
13384 4138 Acked Flush 0 Done Done Sent false
13385 4139 Acked Read 10 Done Done Sent false
13386 4140 Acked Read 2 Done Done Sent false
13387 4141 Acked Read 10 Done Done Sent false
13388 4142 Acked Write 6 Done Done Sent false
13389 4143 Acked Read 3 Done Done Sent false
13390 4144 Acked Flush 0 Done Done Sent false
13391 4145 Acked Read 4 Done Done Sent false
13392 4146 Acked Read 7 Done Done Sent false
13393 4147 Acked Read 10 Done Done Sent false
13394 4148 Acked Write 9 Done Done Sent false
13395 4149 Acked Read 8 Done Done Sent false
13396 4150 Acked Read 9 Done Done Sent false
13397 4151 Acked Read 7 Done Done Sent false
13398 4152 Acked Read 7 Done Done Sent false
13399 4153 Acked Read 7 Done Done Sent false
13400 4154 Acked Read 5 Done Done Sent false
13401 4155 Acked Write 2 Done Done Sent false
13402 4156 Acked Write 2 Done Done Sent false
13403 4157 Acked Write 1 Done Done Sent false
13404 4158 Acked Read 3 Done Done Sent false
13405 4159 Acked Write 9 Done Done Sent false
13406 4160 Acked Write 8 Done Done Sent false
13407 4161 Acked Write 9 Done Done Sent false
13408 4162 Acked Read 7 Done Done Sent false
13409 4163 Acked Read 7 Done Done Sent false
13410 4164 Acked Read 8 Done Done Sent false
13411 4165 Acked Flush 0 Done Done Sent false
13412 4166 Acked Read 3 Done Done Sent false
13413 4167 Acked Write 10 Done Done Sent false
13414 4168 Acked Read 10 Done Done Sent false
13415 4169 Acked Read 1 Done Done Sent false
13416 4170 Acked Write 3 Done Done Sent false
13417 4171 Acked Write 10 Done Done Sent false
13418 4172 Acked Write 4 Done Done Sent false
13419 4173 Acked Read 4 Done Done Sent false
13420 4174 Acked Write 6 Done Done Sent false
13421 4175 Acked Write 5 Done Done Sent false
13422 4176 Acked Read 3 Done Done Sent false
13423 4177 Acked Write 5 Done Done Sent false
13424 4178 Acked Read 5 Done Done Sent false
13425 4179 Acked Write 10 Done Done Sent false
13426 4180 Acked Write 8 Done Done Sent false
13427 4181 Acked Flush 0 Done Done Sent false
13428 4182 Acked Read 10 Done Done Sent false
13429 4183 Acked Read 2 Done Done Sent false
13430 4184 Acked Read 3 Done Done Sent false
13431 4185 Acked Flush 0 Done Done Sent false
13432 4186 Acked Write 6 Done Done Sent false
13433 4187 Acked Read 2 Done Done Sent false
13434 4188 Acked Write 3 Done Done Sent false
13435 4189 Acked Read 8 Done Done Sent false
13436 4190 Acked Write 4 Done Done Sent false
13437 4191 Acked Read 10 Done Done Sent false
13438 4192 Acked Write 2 Done Done Sent false
13439 4193 Acked Read 10 Done Done Sent false
13440 4194 Acked Write 9 Done Done Sent false
13441 4195 Acked Write 9 Done Done Sent false
13442 4196 Acked Write 4 Done Done Sent false
13443 4197 Acked Flush 0 Done Done Sent false
13444 4198 Acked Read 10 Done Done Sent false
13445 4199 Acked Write 10 Done Done Sent false
13446 4200 Acked Read 4 Done Done Sent false
13447 4201 Acked Write 4 Done Done Sent false
13448 4202 Acked Read 3 Done Done Sent false
13449 4203 Acked Write 3 Done Done Sent false
13450 4204 Acked Write 5 Done Done Sent false
13451 4205 Acked Read 10 Done Done Sent false
13452 4206 Acked Read 1 Done Done Sent false
13453 4207 Acked Flush 0 Done Done Sent false
13454 4208 Acked Read 4 Done Done Sent false
13455 4209 Acked Write 3 Done Done Sent false
13456 4210 Acked Read 3 Done Done Sent false
13457 4211 Acked Read 9 Done Done Sent false
13458 4212 Acked Read 5 Done Done Sent false
13459 4213 Acked Read 3 Done Done Sent false
13460 4214 Acked Write 7 Done Done Sent false
13461 4215 Acked Read 7 Done Done Sent false
13462 4216 Acked Read 10 Done Done Sent false
13463 4217 Acked Read 3 Done Done Sent false
13464 4218 Acked Read 6 Done Done Sent false
13465 4219 Acked Read 10 Done Done Sent false
13466 4220 Acked Write 3 Done Done Sent false
13467 4221 Acked Read 3 Done Done Sent false
13468 4222 Acked Write 9 Done Done Sent false
13469 4223 Acked Read 8 Done Done Sent false
13470 4224 Acked Flush 0 Done Done Sent false
13471 4225 Acked Write 10 Done Done Sent false
13472 4226 Acked Write 10 Done Done Sent false
13473 4227 Acked Read 6 Done Done Sent false
13474 4228 Acked Read 9 Done Done Sent false
13475 4229 Acked Read 9 Done Done Sent false
13476 4230 Acked Read 4 Done Done Sent false
13477 4231 Acked Read 8 Done Done Sent false
13478 4232 Acked Read 4 Done Done Sent false
13479 4233 Acked Read 4 Done Done Sent false
13480 4234 Acked Write 2 Done Done Sent false
13481 4235 Acked Read 10 Done Done Sent false
13482 4236 Acked Read 6 Done Done Sent false
13483 4237 Acked Read 7 Done Done Sent false
13484 4238 Acked Read 10 Done Done Sent false
13485 4239 Acked Read 6 Done Done Sent false
13486 4240 Acked Flush 0 Done Done Sent false
13487 4241 Acked Write 4 Done Done Sent false
13488 4242 Acked Read 2 Done Done Sent false
13489 4243 Acked Read 7 Done Done Sent false
13490 4244 Acked Read 8 Done Done Sent false
13491 4245 Acked Write 8 Done Done Sent false
13492 4246 Acked Read 9 Done Done Sent false
13493 4247 Acked Read 3 Done Done Sent false
13494 4248 Acked Flush 0 Done Done Sent false
13495 4249 Acked Write 3 Done Done Sent false
13496 4250 Acked Write 3 Done Done Sent false
13497 4251 Acked Read 3 Done Done Sent false
13498 4252 Acked Read 2 Done Done Sent false
13499 4253 Acked Write 8 Done Done Sent false
13500 4254 Acked Read 5 Done Done Sent false
13501 4255 Acked Write 2 Done Done Sent false
13502 4256 Acked Read 1 Done Done Sent false
13503 4257 Acked Read 9 Done Done Sent false
13504 4258 Acked Read 9 Done Done Sent false
13505 4259 Acked Write 2 Done Done Sent false
13506 4260 Acked Read 5 Done Done Sent false
13507 4261 Acked Flush 0 Done Done Sent false
13508 4262 Acked Read 4 Done Done Sent false
13509 4263 Acked Write 7 Done Done Sent false
13510 4264 Acked Read 10 Done Done Sent false
13511 4265 Acked Write 6 Done Done Sent false
13512 4266 Acked Read 9 Done Done Sent false
13513 4267 Acked Write 5 Done Done Sent false
13514 4268 Acked Read 7 Done Done Sent false
13515 4269 Acked Read 1 Done Done Sent false
13516 4270 Acked Read 3 Done Done Sent false
13517 4271 Acked Write 10 Done Done Sent false
13518 4272 Acked Read 7 Done Done Sent false
13519 4273 Acked Flush 0 Done Done Sent false
13520 4274 Acked Read 5 Done Done Sent false
13521 4275 Acked Flush 0 Done Done Sent false
13522 4276 Acked Read 7 Done Done Sent false
13523 4277 Acked Write 9 Done Done Sent false
13524 4278 Acked Write 5 Done Done Sent false
13525 4279 Acked Read 8 Done Done Sent false
13526 4280 Acked Read 4 Done Done Sent false
13527 4281 Acked Write 6 Done Done Sent false
13528 4282 Acked Write 3 Done Done Sent false
13529 4283 Acked Write 4 Done Done Sent false
13530 4284 Acked Write 9 Done Done Sent false
13531 4285 Acked Write 9 Done Done Sent false
13532 4286 Acked Read 10 Done Done Sent false
13533 4287 Acked Write 3 Done Done Sent false
13534 4288 Acked Write 9 Done Done Sent false
13535 4289 Acked Read 2 Done Done Sent false
13536 4290 Acked Read 6 Done Done Sent false
13537 4291 Acked Flush 0 Done Done Sent false
13538 4292 Acked Read 8 Done Done Sent false
13539 4293 Acked Write 5 Done Done Sent false
13540 4294 Acked Write 8 Done Done Sent false
13541 4295 Acked Write 10 Done Done Sent false
13542 4296 Acked Read 6 Done Done Sent false
13543 4297 Acked Read 3 Done Done Sent false
13544 4298 Acked Write 6 Done Done Sent false
13545 4299 Acked Write 2 Done Done Sent false
13546 4300 Acked Read 10 Done Done Sent false
13547 4301 Acked Read 5 Done Done Sent false
13548 4302 Acked Write 5 Done Done Sent false
13549 4303 Acked Write 1 Done Done Sent false
13550 4304 Acked Write 8 Done Done Sent false
13551 4305 Acked Read 1 Done Done Sent false
13552 4306 Acked Read 4 Done Done Sent false
13553 4307 Acked Flush 0 Done Done Sent false
13554 4308 Acked Read 8 Done Done Sent false
13555 4309 Acked Write 2 Done Done Sent false
13556 4310 Acked Read 8 Done Done Sent false
13557 4311 Acked Write 5 Done Done Sent false
13558 4312 Acked Write 3 Done Done Sent false
13559 4313 Acked Write 6 Done Done Sent false
13560 4314 Acked Read 2 Done Done Sent false
13561 4315 Acked Flush 0 Done Done Sent false
13562 4316 Acked Read 1 Done Done Sent false
13563 4317 Acked Read 4 Done Done Sent false
13564 4318 Acked Write 1 Done Done Sent false
13565 4319 Acked Read 3 Done Done Sent false
13566 4320 Acked Read 4 Done Done Sent false
13567 STATES DS:0 DS:1 DS:2 TOTAL
13568 Sent 0 0 301 301
13569 Done 301 301 0 602
13570 Skipped 0 0 0 0
13571 Error 0 0 0 0
13572 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4019))
13573 Downstairs last five completed: 4019 4018 4017 4016 4015
13574 Upstairs last five completed: 4320 4319 4318 4317 4316
13575 CLIENT: Up:0 ds:301 act:2
135762025-03-17T22:37:32.591ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135772025-03-17T22:37:33.384ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135782025-03-17T22:37:33.385ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64172) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135792025-03-17T22:37:33.386ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135802025-03-17T22:37:33.386ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135812025-03-17T22:37:33.386ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135822025-03-17T22:37:33.386ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135832025-03-17T22:37:33.386ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135842025-03-17T22:37:33.386ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135852025-03-17T22:37:33.387ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
13586 ----------------------------------------------------------------
13587 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13588 Downstairs last five completed: 4321 4320 4319 4318 4317
13589 Upstairs last five completed: 4321 4320 4319 4318 4317
13590 CLIENT: Up:0 ds:0 act:3
13591 Replay: All jobs finished
13592 Main task has work to do, go find it
13593 got dsc Stop(0)
13594 stop 0
13595 Mar 17 22:37:35.645 INFO request completed, latency_us: 365[8810] Got stop action so:false kr:false
13596 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 34ef6ed9-9041-4738-83bf-ab6c2bf3743a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
135972025-03-17T22:37:35.645ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
135982025-03-17T22:37:35.645ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
13599 Mar 17 22:37:35.645 INFO request completed, latency_us: 161, {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:37:35.645979573Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
13600 response_code: 200, uri: /state/cid/0, method: GET, req_id: dd14a244-749f-4207-b5bd-0bc310f5b63e, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
136012025-03-17T22:37:35.646ZINFOcrucible: client task is exiting = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
13602 [8810] Exited with: ExitStatus(unix_wait_status(9))
13603 [8810][0] reports Exit
13604 Mar 17 22:37:39.648 INFO request completed, latency_us: 276, response_code: 200, uri: /state/cid/0, method: GET, req_id: b7fcfaa2-efae-4def-b803-4ab5298f72c8, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
13605 001/300 Read block 2390 len 4096
13606 002/300 Write block 3569 len 16384 data: 4 6 5 5
13607 003/300 Write block 3590 len 24576 data: 4 3 3 4 4 4
13608 004/300 Read block 580 len 12288
13609 005/300 Write block 3678 len 20480 data: 4 5 5 5 5
13610 006/300 Write block 459 len 8192 data: 3 4
13611 007/300 Read block 3003 len 32768
13612 008/300 Read block 2180 len 8192
13613 009/300 Write block 3081 len 36864 data: 2 2 2 2 2 3 2 3 2
13614 010/300 Read block 1304 len 32768
13615 011/300 Flush
13616 012/300 Read block 1989 len 32768
13617 013/300 Read block 1802 len 8192
13618 014/300 Write block 1150 len 28672 data: 3 3 4 3 3 3 3
13619 015/300 Read block 2947 len 8192
13620 016/300 Read block 2193 len 16384
13621 017/300 Read block 2715 len 32768
13622 018/300 Read block 4869 len 24576
13623 019/300 Read block 4979 len 4096
13624 020/300 Read block 2369 len 40960
13625 021/300 Flush
13626 022/300 Read block 1080 len 40960
13627 023/300 Write block 4025 len 16384 data: 5 5 5 5
13628 024/300 Read block 4485 len 4096
13629 025/300 Write block 2476 len 20480 data: 3 4 4 4 4
13630 026/300 Read block 1166 len 20480
13631 027/300 Read block 422 len 32768
13632 028/300 Flush
13633 029/300 Read block 4685 len 16384
13634 030/300 Read block 644 len 4096
13635 031/300 Write block 2030 len 32768 data: 6 6 6 6 6 5 4 4
13636 032/300 Read block 1850 len 20480
13637 033/300 Flush
13638 034/300 Read block 4900 len 4096
13639 035/300 Write block 2630 len 12288 data: 3 3 4
13640 036/300 Read block 3824 len 24576
13641 037/300 Read block 4754 len 24576
13642 038/300 Read block 1804 len 40960
13643 039/300 Write block 1162 len 4096 data: 6
13644 040/300 Write block 146 len 32768 data: 3 3 2 3 2 2 2 2
13645 041/300 Read block 2596 len 12288
13646 042/300 Write block 3108 len 16384 data: 3 3 3 2
13647 043/300 Write block 3708 len 8192 data: 2 3
13648 044/300 Write block 4541 len 20480 data: 3 2 2 2 2
13649 045/300 Read block 1191 len 4096
13650 046/300 Read block 3899 len 36864
13651 047/300 Read block 2944 len 36864
13652 048/300 Write block 302 len 36864 data: 4 5 5 4 4 4 5 4 4
13653 049/300 Read block 2312 len 28672
13654 050/300 Write block 3318 len 12288 data: 4 4 5
13655 051/300 Flush
13656 052/300 Read block 4728 len 12288
13657 053/300 Write block 766 len 16384 data: 5 6 6 6
13658 054/300 Read block 957 len 36864
13659 055/300 Write block 3900 len 16384 data: 4 5 4 3
13660 056/300 Flush
13661 057/300 Write block 1087 len 32768 data: 4 4 3 4 3 3 3 3
13662 058/300 Read block 1715 len 36864
13663 059/300 Read block 2339 len 24576
13664 060/300 Write block 903 len 40960 data: 4 4 4 4 4 5 4 4 3 2
13665 061/300 Write block 3042 len 20480 data: 2 2 2 2 2
13666 062/300 Flush
13667 063/300 Read block 2520 len 24576
13668 064/300 Read block 1110 len 16384
13669 065/300 Write block 3382 len 28672 data: 4 4 6 5 4 4 6
13670 066/300 Write block 4578 len 4096 data: 3
13671 067/300 Write block 3780 len 24576 data: 5 4 4 6 4 6
13672 068/300 Write block 702 len 28672 data: 5 5 6 7 7 6 6
13673 069/300 Read block 962 len 12288
13674 070/300 Write block 4828 len 20480 data: 3 2 2 2 2
13675 071/300 Read block 912 len 28672
13676 072/300 Write block 3839 len 36864 data: 5 4 3 3 3 3 4 3 3
13677 073/300 Read block 827 len 36864
13678 074/300 Read block 2312 len 12288
13679 075/300 Flush
13680 076/300 Flush
13681 077/300 Read block 3484 len 24576
13682 078/300 Read block 3244 len 12288
13683 079/300 Read block 4877 len 24576
13684 080/300 Read block 790 len 12288
13685 081/300 Read block 4450 len 8192
13686 082/300 Write block 2920 len 24576 data: 2 3 3 3 3 2
13687 083/300 Read block 2546 len 36864
13688 084/300 Write block 3792 len 36864 data: 4 4 4 3 2 2 2 2 2
13689 085/300 Flush
13690 086/300 Write block 4192 len 12288 data: 2 2 2
13691 087/300 Flush
13692 088/300 Read block 2877 len 32768
13693 089/300 Read block 1894 len 4096
13694 090/300 Write block 4957 len 36864 data: 3 3 2 2 2 2 2 2 2
13695 091/300 Read block 4619 len 8192
13696 092/300 Read block 1446 len 20480
13697 093/300 Write block 1655 len 28672 data: 4 5 5 4 4 5 4
13698 094/300 Write block 1869 len 12288 data: 4 5 5
13699 095/300 Read block 1614 len 40960
13700 096/300 Write block 444 len 32768 data: 4 3 4 4 4 5 5 4
13701 097/300 Write block 798 len 40960 data: 3 3 3 3 3 2 3 3 3 3
13702 098/300 Write block 4638 len 40960 data: 5 5 5 5 5 4 4 4 4 4
13703 099/300 Flush
13704 100/300 Flush
13705 101/300 Read block 3064 len 4096
13706 102/300 Write block 3792 len 8192 data: 5 5
13707 103/300 Read block 232 len 32768
13708 104/300 Read block 992 len 4096
13709 105/300 Write block 3728 len 4096 data: 4
13710 106/300 Read block 3194 len 12288
13711 107/300 Write block 4878 len 32768 data: 3 2 2 3 3 3 3 3
13712 108/300 Read block 1190 len 24576
13713 109/300 Flush
13714 110/300 Read block 30 len 4096
13715 111/300 Read block 1554 len 40960
13716 112/300 Read block 1754 len 40960
13717 113/300 Write block 4269 len 12288 data: 3 2 3
13718 114/300 Write block 11 len 28672 data: 3 3 3 2 2 2 2
13719 115/300 Write block 4314 len 4096 data: 4
13720 116/300 Write block 3728 len 16384 data: 5 5 4 5
13721 117/300 Write block 4203 len 28672 data: 2 2 2 4 4 4 3
13722 118/300 Read block 3449 len 16384
13723 119/300 Read block 4021 len 40960
13724 120/300 Write block 3920 len 12288 data: 2 3 3
13725 121/300 Write block 3420 len 12288 data: 5 6 6
13726 122/300 Read block 2806 len 12288
13727 123/300 Read block 729 len 24576
13728 124/300 Read block 6 len 20480
13729 125/300 Write block 438 len 32768 data: 2 3 3 3 3 4 5 4
13730 126/300 Write block 4827 len 20480 data: 3 4 3 3 3
13731 127/300 Write block 1557 len 16384 data: 3 3 2 3
13732 128/300 Read block 609 len 40960
13733 129/300 Read block 4006 len 20480
13734 130/300 Read block 1620 len 4096
13735 131/300 Read block 911 len 32768
13736 132/300 Read block 108 len 28672
13737 133/300 Read block 3336 len 8192
13738 134/300 Read block 983 len 16384
13739 135/300 Write block 2399 len 32768 data: 2 4 4 5 5 5 4 3
13740 136/300 Read block 3456 len 20480
13741 137/300 Read block 3696 len 4096
13742 138/300 Write block 3549 len 28672 data: 3 2 4 4 4 5 5
13743 139/300 Flush
13744 140/300 Write block 262 len 32768 data: 3 5 6 7 6 6 5 4
13745 141/300 Flush
13746 142/300 Read block 4396 len 16384
13747 143/300 Read block 3841 len 40960
13748 144/300 Read block 314 len 28672
13749 145/300 Write block 828 len 8192 data: 3 2
13750 146/300 Write block 4258 len 16384 data: 3 3 3 5
13751 147/300 Read block 2308 len 36864
13752 148/300 Read block 380 len 36864
13753 149/300 Flush
13754 150/300 Read block 3958 len 20480
13755 151/300 Write block 610 len 32768 data: 3 2 2 3 3 4 5 4
13756 152/300 Read block 3019 len 20480
13757 153/300 Flush
13758 154/300 Write block 4901 len 16384 data: 2 2 3 4
13759 155/300 Read block 4605 len 32768
13760 156/300 Write block 1621 len 36864 data: 3 4 4 4 4 4 4 3 3
13761 157/300 Flush
13762 158/300 Write block 3556 len 40960 data: 5 3 3 3 3 3 3 3 2 3
13763 159/300 Read block 3999 len 16384
13764 160/300 Write block 3544 len 28672 data: 2 2 4 4 4 4 3
13765 161/300 Write block 1258 len 8192 data: 3 4
13766 162/300 Read block 414 len 8192
13767 163/300 Write block 1176 len 8192 data: 3 3
13768 164/300 Read block 1998 len 16384
13769 165/300 Read block 4816 len 12288
13770 166/300 Write block 4575 len 8192 data: 4 4
13771 167/300 Read block 3945 len 24576
13772 168/300 Flush
13773 169/300 Read block 4477 len 36864
13774 170/300 Read block 4612 len 4096
13775 171/300 Flush
13776 172/300 Read block 2557 len 40960
13777 173/300 Read block 4811 len 24576
13778 174/300 Read block 82 len 12288
13779 175/300 Write block 1231 len 20480 data: 3 3 3 3 3
13780 176/300 Read block 1129 len 40960
13781 177/300 Write block 1585 len 16384 data: 2 2 2 2
13782 178/300 Write block 1241 len 24576 data: 3 3 3 3 2 2
13783 179/300 Read block 2629 len 20480
13784 180/300 Write block 1280 len 16384 data: 4 3 3 3
13785 181/300 Read block 3566 len 24576
13786 182/300 Write block 974 len 16384 data: 4 4 3 3
13787 183/300 Read block 2924 len 8192
13788 184/300 Read block 1974 len 16384
13789 185/300 Write block 2728 len 40960 data: 3 5 5 5 6 6 6 6 3 3
13790 186/300 Read block 4393 len 28672
13791 187/300 Flush
13792 188/300 Write block 4984 len 12288 data: 3 3 3
13793 189/300 Write block 1136 len 32768 data: 3 3 3 3 3 4 4 3
13794 190/300 Flush
13795 191/300 Write block 8 len 16384 data: 3 3 3 4
13796 192/300 Write block 2765 len 40960 data: 3 3 4 4 4 4 3 4 3 3
13797 193/300 Write block 3652 len 16384 data: 3 4 4 5
13798 194/300 Write block 1566 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13799 195/300 Read block 4111 len 16384
13800 196/300 Write block 1377 len 16384 data: 4 4 3 4
13801 197/300 Write block 154 len 8192 data: 2 3
13802 198/300 Read block 1451 len 28672
13803 199/300 Write block 3237 len 28672 data: 3 3 3 3 4 3 3
13804 200/300 Read block 2116 len 28672
13805 201/300 Read block 3649 len 40960
13806 202/300 Write block 3182 len 36864 data: 6 5 4 4 5 4 5 5 4
13807 203/300 Write block 2151 len 16384 data: 4 4 3 3
13808 204/300 Write block 4222 len 4096 data: 5
13809 205/300 Flush
13810 206/300 Read block 634 len 32768
13811 207/300 Read block 4817 len 16384
13812 208/300 Write block 4308 len 32768 data: 6 5 5 5 4 4 5 2
13813 209/300 Read block 335 len 24576
13814 210/300 Write block 4580 len 4096 data: 3
13815 211/300 Read block 4090 len 8192
13816 212/300 Write block 850 len 40960 data: 2 3 3 3 3 3 3 2 2 2
13817 213/300 Flush
13818 214/300 Read block 3753 len 16384
13819 215/300 Flush
13820 216/300 Write block 3150 len 12288 data: 4 4 5
13821 217/300 Read block 2505 len 32768
13822 218/300 Write block 4172 len 4096 data: 3
13823 219/300 Read block 571 len 36864
13824 220/300 Write block 781 len 24576 data: 4 5 5 5 4 2
13825 221/300 Read block 438 len 32768
13826 222/300 Read block 4869 len 40960
13827 223/300 Flush
13828 224/300 Read block 644 len 24576
13829 225/300 Read block 3625 len 40960
13830 226/300 Flush
13831 227/300 Read block 4964 len 8192
13832 228/300 Write block 3350 len 36864 data: 3 3 3 2 2 2 2 4 4
13833 229/300 Write block 4162 len 24576 data: 3 4 4 4 5 6
13834 230/300 Read block 106 len 40960
13835 231/300 Read block 4213 len 20480
13836 232/300 Read block 705 len 32768
13837 233/300 Read block 1257 len 8192
13838 234/300 Write block 4238 len 40960 data: 3 3 3 3 3 3 3 3 3 4
13839 235/300 Flush
13840 236/300 Write block 2752 len 24576 data: 6 6 5 5 4 3
13841 237/300 Read block 180 len 16384
13842 238/300 Read block 2492 len 4096
13843 239/300 Flush
13844 240/300 Write block 551 len 40960 data: 4 4 4 4 5 5 3 3 3 3
13845 241/300 Read block 1859 len 16384
13846 242/300 Read block 3677 len 12288
13847 243/300 Read block 1678 len 32768
13848 244/300 Flush
13849 245/300 Read block 1068 len 12288
13850 246/300 Read block 3373 len 8192
13851 247/300 Read block 3316 len 32768
13852 248/300 Read block 3197 len 16384
13853 249/300 Write block 3406 len 40960 data: 3 3 3 5 5 4 5 5 5 5
13854 250/300 Write block 4890 len 24576 data: 2 2 2 2 2 2
13855 251/300 Flush
13856 252/300 Read block 3977 len 32768
13857 253/300 Read block 3550 len 36864
13858 254/300 Read block 731 len 36864
13859 255/300 Write block 1357 len 12288 data: 4 4 4
13860 256/300 Write block 1088 len 20480 data: 5 4 5 4 4
13861 257/300 Read block 4674 len 12288
13862 258/300 Flush
13863 259/300 Write block 2152 len 4096 data: 5
13864 260/300 Read block 3809 len 28672
13865 261/300 Read block 1751 len 32768
13866 262/300 Read block 3473 len 24576
13867 263/300 Write block 1635 len 28672 data: 4 3 4 4 4 4 3
13868 264/300 Flush
13869 265/300 Read block 2850 len 28672
13870 266/300 Read block 674 len 36864
13871 267/300 Flush
13872 268/300 Flush
13873 269/300 Write block 3771 len 24576 data: 2 2 3 3 3 4
13874 270/300 Read block 3249 len 36864
13875 271/300 Read block 1387 len 16384
13876 272/300 Write block 170 len 16384 data: 2 3 3 3
13877 273/300 Write block 4910 len 32768 data: 6 6 5 5 5 6 7 8
13878 274/300 Write block 1507 len 28672 data: 4 4 4 3 3 5 5
13879 275/300 Write block 4074 len 16384 data: 4 4 3 3
13880 276/300 Write block 2963 len 28672 data: 3 2 2 2 2 2 2
13881 277/300 Read block 3114 len 24576
13882 278/300 Write block 4843 len 32768 data: 4 4 4 4 4 3 3 3
13883 279/300 Write block 3718 len 36864 data: 3 4 3 4 4 4 3 4 4
13884 280/300 Read block 2607 len 32768
13885 281/300 Read block 4094 len 24576
13886 282/300 Write block 2978 len 24576 data: 5 6 6 5 4 4
13887 283/300 Read block 4470 len 32768
13888 284/300 Read block 1456 len 24576
13889 285/300 Write block 3003 len 32768 data: 2 2 2 2 2 2 2 3
13890 286/300 Write block 1156 len 24576 data: 4 3 3 4 5 5
13891 287/300 Write block 1496 len 36864 data: 3 4 5 5 4 3 3 3 4
13892 288/300 Flush
13893 289/300 Read block 2941 len 4096
13894 290/300 Write block 269 len 36864 data: 5 4 4 3 3 3 4 4 3
13895 291/300 Read block 4646 len 4096
13896 292/300 Read block 3383 len 4096
13897 293/300 Read block 1045 len 16384
13898 294/300 Read block 2612 len 40960
13899 295/300 Write block 1424 len 8192 data: 3 4
13900 296/300 Read block 352 len 8192
13901 297/300 Write block 4464 len 20480 data: 5 5 4 3 2
13902 298/300 Write block 846 len 32768 data: 4 3 2 2 3 4 4 4
13903 299/300 Flush
13904 300/300 Read block 2152 len 4096
13905 301/300 Write block 4858 len 40960 data: 2 2 2 2 2 3 3 3 3 4
13906 Main task has work to do, go find it
13907 got dsc Start(0)
13908 start 0
13909 Mar 17 22:37:39.895 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/0, method: POST, req_id: f010fbf9-b3fb-446a-8299-104725c6feab, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
13910 Replay: started 0, returned:Ok(())
13911 ----------------------------------------------------------------
13912 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13913 States: Offline Active Active
13914 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13915 4322 Acked Read 1 Sent Done Done false
13916 4323 Acked Write 4 Sent Done Done false
13917 4324 Acked Write 6 Sent Done Done false
13918 4325 Acked Read 3 Sent Done Done false
13919 4326 Acked Write 5 Sent Done Done false
13920 4327 Acked Write 2 Sent Done Done false
13921 4328 Acked Read 8 Sent Done Done false
13922 4329 Acked Read 2 Sent Done Done false
13923 4330 Acked Write 9 Sent Done Done false
13924 4331 Acked Read 8 Sent Done Done false
13925 4332 Acked Flush 0 Sent Done Done false
13926 4333 Acked Read 8 Sent Done Done false
13927 4334 Acked Read 2 Sent Done Done false
13928 4335 Acked Write 7 Sent Done Done false
13929 4336 Acked Read 2 Sent Done Done false
13930 4337 Acked Read 4 Sent Done Done false
13931 4338 Acked Read 8 Sent Done Done false
13932 4339 Acked Read 6 Sent Done Done false
13933 4340 Acked Read 1 Sent Done Done false
13934 4341 Acked Read 10 Sent Done Done false
13935 4342 Acked Flush 0 Sent Done Done false
13936 4343 Acked Read 10 Sent Done Done false
13937 4344 Acked Write 4 Sent Done Done false
13938 4345 Acked Read 1 Sent Done Done false
13939 4346 Acked Write 5 Sent Done Done false
13940 4347 Acked Read 5 Sent Done Done false
13941 4348 Acked Read 8 Sent Done Done false
13942 4349 Acked Flush 0 Sent Done Done false
13943 4350 Acked Read 4 Sent Done Done false
13944 4351 Acked Read 1 Sent Done Done false
13945 4352 Acked Write 8 Sent Done Done false
13946 4353 Acked Read 5 Sent Done Done false
13947 4354 Acked Flush 0 Sent Done Done false
13948 4355 Acked Read 1 Sent Done Done false
13949 4356 Acked Write 3 Sent Done Done false
13950 4357 Acked Read 6 Sent Done Done false
13951 4358 Acked Read 6 Sent Done Done false
13952 4359 Acked Read 10 Sent Done Done false
13953 4360 Acked Write 1 Sent Done Done false
13954 4361 Acked Write 8 Sent Done Done false
13955 4362 Acked Read 3 Sent Done Done false
13956 4363 Acked Write 4 Sent Done Done false
13957 4364 Acked Write 2 Sent Done Done false
13958 4365 Acked Write 5 Sent Done Done false
13959 4366 Acked Read 1 Sent Done Done false
13960 4367 Acked Read 9 Sent Done Done false
13961 4368 Acked Read 9 Sent Done Done false
13962 4369 Acked Write 9 Sent Done Done false
13963 4370 Acked Read 7 Sent Done Done false
13964 4371 Acked Write 3 Sent Done Done false
13965 4372 Acked Flush 0 Sent Done Done false
13966 4373 Acked Read 3 Sent Done Done false
13967 4374 Acked Write 4 Sent Done Done false
13968 4375 Acked Read 9 Sent Done Done false
13969 4376 Acked Write 4 Sent Done Done false
13970 4377 Acked Flush 0 Sent Done Done false
13971 4378 Acked Write 8 Sent Done Done false
13972 4379 Acked Read 9 Sent Done Done false
13973 4380 Acked Read 6 Sent Done Done false
13974 4381 Acked Write 10 Sent Done Done false
13975 4382 Acked Write 5 Sent Done Done false
13976 4383 Acked Flush 0 Sent Done Done false
13977 4384 Acked Read 6 Sent Done Done false
13978 4385 Acked Read 4 Sent Done Done false
13979 4386 Acked Write 7 Sent Done Done false
13980 4387 Acked Write 1 Sent Done Done false
13981 4388 Acked Write 6 Sent Done Done false
13982 4389 Acked Write 7 Sent Done Done false
13983 4390 Acked Read 3 Sent Done Done false
13984 4391 Acked Write 5 Sent Done Done false
13985 4392 Acked Read 7 Sent Done Done false
13986 4393 Acked Write 9 Sent Done Done false
13987 4394 Acked Read 9 Sent Done Done false
13988 4395 Acked Read 3 Sent Done Done false
13989 4396 Acked Flush 0 Sent Done Done false
13990 4397 Acked Flush 0 Sent Done Done false
13991 4398 Acked Read 6 Sent Done Done false
13992 4399 Acked Read 3 Sent Done Done false
13993 4400 Acked Read 6 Sent Done Done false
13994 4401 Acked Read 3 Sent Done Done false
13995 4402 Acked Read 2 Sent Done Done false
13996 4403 Acked Write 6 Sent Done Done false
13997 4404 Acked Read 9 Sent Done Done false
13998 4405 Acked Write 9 Sent Done Done false
13999 4406 Acked Flush 0 Sent Done Done false
14000 4407 Acked Write 3 Sent Done Done false
14001 4408 Acked Flush 0 Sent Done Done false
14002 4409 Acked Read 8 Sent Done Done false
14003 4410 Acked Read 1 Sent Done Done false
14004 4411 Acked Write 9 Sent Done Done false
14005 4412 Acked Read 2 Sent Done Done false
14006 4413 Acked Read 5 Sent Done Done false
14007 4414 Acked Write 7 Sent Done Done false
14008 4415 Acked Write 3 Sent Done Done false
14009 4416 Acked Read 10 Sent Done Done false
14010 4417 Acked Write 8 Sent Done Done false
14011 4418 Acked Write 10 Sent Done Done false
14012 4419 Acked Write 10 Sent Done Done false
14013 4420 Acked Flush 0 Sent Done Done false
14014 4421 Acked Flush 0 Sent Done Done false
14015 4422 Acked Read 1 Sent Done Done false
14016 4423 Acked Write 2 Sent Done Done false
14017 4424 Acked Read 8 Sent Done Done false
14018 4425 Acked Read 1 Sent Done Done false
14019 4426 Acked Write 1 Sent Done Done false
14020 4427 Acked Read 3 Sent Done Done false
14021 4428 Acked Write 8 Sent Done Done false
14022 4429 Acked Read 6 Sent Done Done false
14023 4430 Acked Flush 0 Sent Done Done false
14024 4431 Acked Read 1 Sent Done Done false
14025 4432 Acked Read 10 Sent Done Done false
14026 4433 Acked Read 10 Sent Done Done false
14027 4434 Acked Write 3 Sent Done Done false
14028 4435 Acked Write 7 Sent Done Done false
14029 4436 Acked Write 1 Sent Done Done false
14030 4437 Acked Write 4 Sent Done Done false
14031 4438 Acked Write 7 Sent Done Done false
14032 4439 Acked Read 4 Sent Done Done false
14033 4440 Acked Read 10 Sent Done Done false
14034 4441 Acked Write 3 Sent Done Done false
14035 4442 Acked Write 3 Sent Done Done false
14036 4443 Acked Read 3 Sent Done Done false
14037 4444 Acked Read 6 Sent Done Done false
14038 4445 Acked Read 5 Sent Done Done false
14039 4446 Acked Write 8 Sent Done Done false
14040 4447 Acked Write 5 Sent Done Done false
14041 4448 Acked Write 4 Sent Done Done false
14042 4449 Acked Read 10 Sent Done Done false
14043 4450 Acked Read 5 Sent Done Done false
14044 4451 Acked Read 1 Sent Done Done false
14045 4452 Acked Read 8 Sent Done Done false
14046 4453 Acked Read 7 Sent Done Done false
14047 4454 Acked Read 2 Sent Done Done false
14048 4455 Acked Read 4 Sent Done Done false
14049 4456 Acked Write 8 Sent Done Done false
14050 4457 Acked Read 5 Sent Done Done false
14051 4458 Acked Read 1 Sent Done Done false
14052 4459 Acked Write 7 Sent Done Done false
14053 4460 Acked Flush 0 Sent Done Done false
14054 4461 Acked Write 8 Sent Done Done false
14055 4462 Acked Flush 0 Sent Done Done false
14056 4463 Acked Read 4 Sent Done Done false
14057 4464 Acked Read 10 Sent Done Done false
14058 4465 Acked Read 7 Sent Done Done false
14059 4466 Acked Write 2 Sent Done Done false
14060 4467 Acked Write 4 Sent Done Done false
14061 4468 Acked Read 9 Sent Done Done false
14062 4469 Acked Read 9 Sent Done Done false
14063 4470 Acked Flush 0 Sent Done Done false
14064 4471 Acked Read 5 Sent Done Done false
14065 4472 Acked Write 8 Sent Done Done false
14066 4473 Acked Read 5 Sent Done Done false
14067 4474 Acked Flush 0 Sent Done Done false
14068 4475 Acked Write 4 Sent Done Done false
14069 4476 Acked Read 8 Sent Done Done false
14070 4477 Acked Write 9 Sent Done Done false
14071 4478 Acked Flush 0 Sent Done Done false
14072 4479 Acked Write 10 Sent Done Done false
14073 4480 Acked Read 4 Sent Done Done false
14074 4481 Acked Write 7 Sent Done Done false
14075 4482 Acked Write 2 Sent Done Done false
14076 4483 Acked Read 2 Sent Done Done false
14077 4484 Acked Write 2 Sent Done Done false
14078 4485 Acked Read 4 Sent Done Done false
14079 4486 Acked Read 3 Sent Done Done false
14080 4487 Acked Write 2 Sent Done Done false
14081 4488 Acked Read 6 Sent Done Done false
14082 4489 Acked Flush 0 Sent Done Done false
14083 4490 Acked Read 9 Sent Done Done false
14084 4491 Acked Read 1 Sent Done Done false
14085 4492 Acked Flush 0 Sent Done Done false
14086 4493 Acked Read 10 Sent Done Done false
14087 4494 Acked Read 6 Sent Done Done false
14088 4495 Acked Read 3 Sent Done Done false
14089 4496 Acked Write 5 Sent Done Done false
14090 4497 Acked Read 10 Sent Done Done false
14091 4498 Acked Write 4 Sent Done Done false
14092 4499 Acked Write 6 Sent Done Done false
14093 4500 Acked Read 5 Sent Done Done false
14094 4501 Acked Write 4 Sent Done Done false
14095 4502 Acked Read 6 Sent Done Done false
14096 4503 Acked Write 4 Sent Done Done false
14097 4504 Acked Read 2 Sent Done Done false
14098 4505 Acked Read 4 Sent Done Done false
14099 4506 Acked Write 10 Sent Done Done false
14100 4507 Acked Read 7 Sent Done Done false
14101 4508 Acked Flush 0 Sent Done Done false
14102 4509 Acked Write 3 Sent Done Done false
14103 4510 Acked Write 8 Sent Done Done false
14104 4511 Acked Flush 0 Sent Done Done false
14105 4512 Acked Write 4 Sent Done Done false
14106 4513 Acked Write 10 Sent Done Done false
14107 4514 Acked Write 4 Sent Done Done false
14108 4515 Acked Write 10 Sent Done Done false
14109 4516 Acked Read 4 Sent Done Done false
14110 4517 Acked Write 4 Sent Done Done false
14111 4518 Acked Write 2 Sent Done Done false
14112 4519 Acked Read 7 Sent Done Done false
14113 4520 Acked Write 7 Sent Done Done false
14114 4521 Acked Read 7 Sent Done Done false
14115 4522 Acked Read 10 Sent Done Done false
14116 4523 Acked Write 9 Sent Done Done false
14117 4524 Acked Write 4 Sent Done Done false
14118 4525 Acked Write 1 Sent Done Done false
14119 4526 Acked Flush 0 Sent Done Done false
14120 4527 Acked Read 8 Sent Done Done false
14121 4528 Acked Read 4 Sent Done Done false
14122 4529 Acked Write 8 Sent Done Done false
14123 4530 Acked Read 6 Sent Done Done false
14124 4531 Acked Write 1 Sent Done Done false
14125 4532 Acked Read 2 Sent Done Done false
14126 4533 Acked Write 10 Sent Done Done false
14127 4534 Acked Flush 0 Sent Done Done false
14128 4535 Acked Read 4 Sent Done Done false
14129 4536 Acked Flush 0 Sent Done Done false
14130 4537 Acked Write 3 Sent Done Done false
14131 4538 Acked Read 8 Sent Done Done false
14132 4539 Acked Write 1 Sent Done Done false
14133 4540 Acked Read 9 Sent Done Done false
14134 4541 Acked Write 6 Sent Done Done false
14135 4542 Acked Read 8 Sent Done Done false
14136 4543 Acked Read 10 Sent Done Done false
14137 4544 Acked Flush 0 Sent Done Done false
14138 4545 Acked Read 6 Sent Done Done false
14139 4546 Acked Read 10 Sent Done Done false
14140 4547 Acked Flush 0 Sent Done Done false
14141 4548 Acked Read 2 Sent Done Done false
14142 4549 Acked Write 9 Sent Done Done false
14143 4550 Acked Write 6 Sent Done Done false
14144 4551 Acked Read 10 Sent Done Done false
14145 4552 Acked Read 5 Sent Done Done false
14146 4553 Acked Read 8 Sent Done Done false
14147 4554 Acked Read 2 Sent Done Done false
14148 4555 Acked Write 10 Sent Done Done false
14149 4556 Acked Flush 0 Sent Done Done false
14150 4557 Acked Write 6 Sent Done Done false
14151 4558 Acked Read 4 Sent Done Done false
14152 4559 Acked Read 1 Sent Done Done false
14153 4560 Acked Flush 0 Sent Done Done false
14154 4561 Acked Write 10 Sent Done Done false
14155 4562 Acked Read 4 Sent Done Done false
14156 4563 Acked Read 3 Sent Done Done false
14157 4564 Acked Read 8 Sent Done Done false
14158 4565 Acked Flush 0 Sent Done Done false
14159 4566 Acked Read 3 Sent Done Done false
14160 4567 Acked Read 2 Sent Done Done false
14161 4568 Acked Read 8 Sent Done Done false
14162 4569 Acked Read 4 Sent Done Done false
14163 4570 Acked Write 10 Sent Done Done false
14164 4571 Acked Write 6 Sent Done Done false
14165 4572 Acked Flush 0 Sent Done Done false
14166 4573 Acked Read 8 Sent Done Done false
14167 4574 Acked Read 9 Sent Done Done false
14168 4575 Acked Read 9 Sent Done Done false
14169 4576 Acked Write 3 Sent Done Done false
14170 4577 Acked Write 5 Sent Done Done false
14171 4578 Acked Read 3 Sent Done Done false
14172 4579 Acked Flush 0 Sent Done Done false
14173 4580 Acked Write 1 Sent Done Done false
14174 4581 Acked Read 7 Sent Done Done false
14175 4582 Acked Read 8 Sent Done Done false
14176 4583 Acked Read 6 Sent Done Done false
14177 4584 Acked Write 7 Sent Done Done false
14178 4585 Acked Flush 0 Sent Done Done false
14179 4586 Acked Read 7 Sent Done Done false
14180 4587 Acked Read 9 Sent Done Done false
14181 4588 Acked Flush 0 Sent Done Done false
14182 4589 Acked Flush 0 Sent Done Done false
14183 4590 Acked Write 6 Sent Done Done false
14184 4591 Acked Read 9 Sent Done Done false
14185 4592 Acked Read 4 Sent Done Done false
14186 4593 Acked Write 4 Sent Done Done false
14187 4594 Acked Write 8 Sent Done Done false
14188 4595 Acked Write 7 Sent Done Done false
14189 4596 Acked Write 4 Sent Done Done false
14190 4597 Acked Write 7 Sent Done Done false
14191 4598 Acked Read 6 Sent Done Done false
14192 4599 Acked Write 8 Sent Done Done false
14193 4600 Acked Write 9 Sent Done Done false
14194 4601 Acked Read 8 Sent Done Done false
14195 4602 Acked Read 6 Sent Done Done false
14196 4603 Acked Write 6 Sent Done Done false
14197 4604 Acked Read 8 Sent Done Done false
14198 4605 Acked Read 6 Sent Done Done false
14199 4606 Acked Write 8 Sent Done Done false
14200 4607 Acked Write 6 Sent Done Done false
14201 4608 Acked Write 9 Sent Done Done false
14202 4609 Acked Flush 0 Sent Done Done false
14203 4610 Acked Read 1 Sent Done Done false
14204 4611 Acked Write 9 Sent Done Done false
14205 4612 Acked Read 1 Sent Done Done false
14206 4613 Acked Read 1 Sent Done Done false
14207 4614 Acked Read 4 Sent Done Done false
14208 4615 Acked Read 10 Sent Done Done false
14209 4616 Acked Write 2 Sent Done Done false
14210 4617 Acked Read 2 Sent Done Done false
14211 4618 Acked Write 5 Sent Done Done false
14212 4619 Acked Write 8 Sent Done Done false
14213 4620 Acked Flush 0 Sent Done Done false
14214 4621 Acked Read 1 Sent Done Done false
14215 4622 Acked Write 10 Sent Done Done false
14216 STATES DS:0 DS:1 DS:2 TOTAL
14217 Sent 301 0 0 301
14218 Done 0 301 301 602
14219 Skipped 0 0 0 0
14220 Error 0 0 0 0
14221 Last Flush: Some(JobId(4321)) Some(JobId(4620)) Some(JobId(4620))
14222 Downstairs last five completed: 4321 4320 4319 4318 4317
14223 Upstairs last five completed: 4622 4621 4620 4619 4618
14224 CLIENT: Up:0 ds:301 act:2
142252025-03-17T22:37:40.606ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
142262025-03-17T22:37:41.608ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
14227 [8810] got start action
14228 [8810] I am going to restart
14229 Starting downstairs at port 8810
14230 Make output file at "/tmp/dsc/downstairs-8810.txt"
14231 [8810][0] reports Starting
14232 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
14233 [8810][0] reports Running
142342025-03-17T22:37:42.610ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
142352025-03-17T22:37:43.888ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
14236 ----------------------------------------------------------------
14237 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14238 States: Offline Active Active
14239 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14240 4322 Acked Read 1 Sent Done Done false
14241 4323 Acked Write 4 Sent Done Done false
14242 4324 Acked Write 6 Sent Done Done false
14243 4325 Acked Read 3 Sent Done Done false
14244 4326 Acked Write 5 Sent Done Done false
14245 4327 Acked Write 2 Sent Done Done false
14246 4328 Acked Read 8 Sent Done Done false
14247 4329 Acked Read 2 Sent Done Done false
14248 4330 Acked Write 9 Sent Done Done false
14249 4331 Acked Read 8 Sent Done Done false
14250 4332 Acked Flush 0 Sent Done Done false
14251 4333 Acked Read 8 Sent Done Done false
14252 4334 Acked Read 2 Sent Done Done false
14253 4335 Acked Write 7 Sent Done Done false
14254 4336 Acked Read 2 Sent Done Done false
14255 4337 Acked Read 4 Sent Done Done false
14256 4338 Acked Read 8 Sent Done Done false
14257 4339 Acked Read 6 Sent Done Done false
14258 4340 Acked Read 1 Sent Done Done false
14259 4341 Acked Read 10 Sent Done Done false
14260 4342 Acked Flush 0 Sent Done Done false
14261 4343 Acked Read 10 Sent Done Done false
14262 4344 Acked Write 4 Sent Done Done false
14263 4345 Acked Read 1 Sent Done Done false
14264 4346 Acked Write 5 Sent Done Done false
14265 4347 Acked Read 5 Sent Done Done false
14266 4348 Acked Read 8 Sent Done Done false
14267 4349 Acked Flush 0 Sent Done Done false
14268 4350 Acked Read 4 Sent Done Done false
14269 4351 Acked Read 1 Sent Done Done false
14270 4352 Acked Write 8 Sent Done Done false
14271 4353 Acked Read 5 Sent Done Done false
14272 4354 Acked Flush 0 Sent Done Done false
14273 4355 Acked Read 1 Sent Done Done false
14274 4356 Acked Write 3 Sent Done Done false
14275 4357 Acked Read 6 Sent Done Done false
14276 4358 Acked Read 6 Sent Done Done false
14277 4359 Acked Read 10 Sent Done Done false
14278 4360 Acked Write 1 Sent Done Done false
14279 4361 Acked Write 8 Sent Done Done false
14280 4362 Acked Read 3 Sent Done Done false
14281 4363 Acked Write 4 Sent Done Done false
14282 4364 Acked Write 2 Sent Done Done false
14283 4365 Acked Write 5 Sent Done Done false
14284 4366 Acked Read 1 Sent Done Done false
14285 4367 Acked Read 9 Sent Done Done false
14286 4368 Acked Read 9 Sent Done Done false
14287 4369 Acked Write 9 Sent Done Done false
14288 4370 Acked Read 7 Sent Done Done false
14289 4371 Acked Write 3 Sent Done Done false
14290 4372 Acked Flush 0 Sent Done Done false
14291 4373 Acked Read 3 Sent Done Done false
14292 4374 Acked Write 4 Sent Done Done false
14293 4375 Acked Read 9 Sent Done Done false
14294 4376 Acked Write 4 Sent Done Done false
14295 4377 Acked Flush 0 Sent Done Done false
14296 4378 Acked Write 8 Sent Done Done false
14297 4379 Acked Read 9 Sent Done Done false
14298 4380 Acked Read 6 Sent Done Done false
14299 4381 Acked Write 10 Sent Done Done false
14300 4382 Acked Write 5 Sent Done Done false
14301 4383 Acked Flush 0 Sent Done Done false
14302 4384 Acked Read 6 Sent Done Done false
14303 4385 Acked Read 4 Sent Done Done false
14304 4386 Acked Write 7 Sent Done Done false
14305 4387 Acked Write 1 Sent Done Done false
14306 4388 Acked Write 6 Sent Done Done false
14307 4389 Acked Write 7 Sent Done Done false
14308 4390 Acked Read 3 Sent Done Done false
14309 4391 Acked Write 5 Sent Done Done false
14310 4392 Acked Read 7 Sent Done Done false
14311 4393 Acked Write 9 Sent Done Done false
14312 4394 Acked Read 9 Sent Done Done false
14313 4395 Acked Read 3 Sent Done Done false
14314 4396 Acked Flush 0 Sent Done Done false
14315 4397 Acked Flush 0 Sent Done Done false
14316 4398 Acked Read 6 Sent Done Done false
14317 4399 Acked Read 3 Sent Done Done false
14318 4400 Acked Read 6 Sent Done Done false
14319 4401 Acked Read 3 Sent Done Done false
14320 4402 Acked Read 2 Sent Done Done false
14321 4403 Acked Write 6 Sent Done Done false
14322 4404 Acked Read 9 Sent Done Done false
14323 4405 Acked Write 9 Sent Done Done false
14324 4406 Acked Flush 0 Sent Done Done false
14325 4407 Acked Write 3 Sent Done Done false
14326 4408 Acked Flush 0 Sent Done Done false
14327 4409 Acked Read 8 Sent Done Done false
14328 4410 Acked Read 1 Sent Done Done false
14329 4411 Acked Write 9 Sent Done Done false
14330 4412 Acked Read 2 Sent Done Done false
14331 4413 Acked Read 5 Sent Done Done false
14332 4414 Acked Write 7 Sent Done Done false
14333 4415 Acked Write 3 Sent Done Done false
14334 4416 Acked Read 10 Sent Done Done false
14335 4417 Acked Write 8 Sent Done Done false
14336 4418 Acked Write 10 Sent Done Done false
14337 4419 Acked Write 10 Sent Done Done false
14338 4420 Acked Flush 0 Sent Done Done false
14339 4421 Acked Flush 0 Sent Done Done false
14340 4422 Acked Read 1 Sent Done Done false
14341 4423 Acked Write 2 Sent Done Done false
14342 4424 Acked Read 8 Sent Done Done false
14343 4425 Acked Read 1 Sent Done Done false
14344 4426 Acked Write 1 Sent Done Done false
14345 4427 Acked Read 3 Sent Done Done false
14346 4428 Acked Write 8 Sent Done Done false
14347 4429 Acked Read 6 Sent Done Done false
14348 4430 Acked Flush 0 Sent Done Done false
14349 4431 Acked Read 1 Sent Done Done false
14350 4432 Acked Read 10 Sent Done Done false
14351 4433 Acked Read 10 Sent Done Done false
14352 4434 Acked Write 3 Sent Done Done false
14353 4435 Acked Write 7 Sent Done Done false
14354 4436 Acked Write 1 Sent Done Done false
14355 4437 Acked Write 4 Sent Done Done false
14356 4438 Acked Write 7 Sent Done Done false
14357 4439 Acked Read 4 Sent Done Done false
14358 4440 Acked Read 10 Sent Done Done false
14359 4441 Acked Write 3 Sent Done Done false
14360 4442 Acked Write 3 Sent Done Done false
14361 4443 Acked Read 3 Sent Done Done false
14362 4444 Acked Read 6 Sent Done Done false
14363 4445 Acked Read 5 Sent Done Done false
14364 4446 Acked Write 8 Sent Done Done false
14365 4447 Acked Write 5 Sent Done Done false
14366 4448 Acked Write 4 Sent Done Done false
14367 4449 Acked Read 10 Sent Done Done false
14368 4450 Acked Read 5 Sent Done Done false
14369 4451 Acked Read 1 Sent Done Done false
14370 4452 Acked Read 8 Sent Done Done false
14371 4453 Acked Read 7 Sent Done Done false
14372 4454 Acked Read 2 Sent Done Done false
14373 4455 Acked Read 4 Sent Done Done false
14374 4456 Acked Write 8 Sent Done Done false
14375 4457 Acked Read 5 Sent Done Done false
14376 4458 Acked Read 1 Sent Done Done false
14377 4459 Acked Write 7 Sent Done Done false
14378 4460 Acked Flush 0 Sent Done Done false
14379 4461 Acked Write 8 Sent Done Done false
14380 4462 Acked Flush 0 Sent Done Done false
14381 4463 Acked Read 4 Sent Done Done false
14382 4464 Acked Read 10 Sent Done Done false
14383 4465 Acked Read 7 Sent Done Done false
14384 4466 Acked Write 2 Sent Done Done false
14385 4467 Acked Write 4 Sent Done Done false
14386 4468 Acked Read 9 Sent Done Done false
14387 4469 Acked Read 9 Sent Done Done false
14388 4470 Acked Flush 0 Sent Done Done false
14389 4471 Acked Read 5 Sent Done Done false
14390 4472 Acked Write 8 Sent Done Done false
14391 4473 Acked Read 5 Sent Done Done false
14392 4474 Acked Flush 0 Sent Done Done false
14393 4475 Acked Write 4 Sent Done Done false
14394 4476 Acked Read 8 Sent Done Done false
14395 4477 Acked Write 9 Sent Done Done false
14396 4478 Acked Flush 0 Sent Done Done false
14397 4479 Acked Write 10 Sent Done Done false
14398 4480 Acked Read 4 Sent Done Done false
14399 4481 Acked Write 7 Sent Done Done false
14400 4482 Acked Write 2 Sent Done Done false
14401 4483 Acked Read 2 Sent Done Done false
14402 4484 Acked Write 2 Sent Done Done false
14403 4485 Acked Read 4 Sent Done Done false
14404 4486 Acked Read 3 Sent Done Done false
14405 4487 Acked Write 2 Sent Done Done false
14406 4488 Acked Read 6 Sent Done Done false
14407 4489 Acked Flush 0 Sent Done Done false
14408 4490 Acked Read 9 Sent Done Done false
14409 4491 Acked Read 1 Sent Done Done false
14410 4492 Acked Flush 0 Sent Done Done false
14411 4493 Acked Read 10 Sent Done Done false
14412 4494 Acked Read 6 Sent Done Done false
14413 4495 Acked Read 3 Sent Done Done false
14414 4496 Acked Write 5 Sent Done Done false
14415 4497 Acked Read 10 Sent Done Done false
14416 4498 Acked Write 4 Sent Done Done false
14417 4499 Acked Write 6 Sent Done Done false
14418 4500 Acked Read 5 Sent Done Done false
14419 4501 Acked Write 4 Sent Done Done false
14420 4502 Acked Read 6 Sent Done Done false
14421 4503 Acked Write 4 Sent Done Done false
14422 4504 Acked Read 2 Sent Done Done false
14423 4505 Acked Read 4 Sent Done Done false
14424 4506 Acked Write 10 Sent Done Done false
14425 4507 Acked Read 7 Sent Done Done false
14426 4508 Acked Flush 0 Sent Done Done false
14427 4509 Acked Write 3 Sent Done Done false
14428 4510 Acked Write 8 Sent Done Done false
14429 4511 Acked Flush 0 Sent Done Done false
14430 4512 Acked Write 4 Sent Done Done false
14431 4513 Acked Write 10 Sent Done Done false
14432 4514 Acked Write 4 Sent Done Done false
14433 4515 Acked Write 10 Sent Done Done false
14434 4516 Acked Read 4 Sent Done Done false
14435 4517 Acked Write 4 Sent Done Done false
14436 4518 Acked Write 2 Sent Done Done false
14437 4519 Acked Read 7 Sent Done Done false
14438 4520 Acked Write 7 Sent Done Done false
14439 4521 Acked Read 7 Sent Done Done false
14440 4522 Acked Read 10 Sent Done Done false
14441 4523 Acked Write 9 Sent Done Done false
14442 4524 Acked Write 4 Sent Done Done false
14443 4525 Acked Write 1 Sent Done Done false
14444 4526 Acked Flush 0 Sent Done Done false
14445 4527 Acked Read 8 Sent Done Done false
14446 4528 Acked Read 4 Sent Done Done false
14447 4529 Acked Write 8 Sent Done Done false
14448 4530 Acked Read 6 Sent Done Done false
14449 4531 Acked Write 1 Sent Done Done false
14450 4532 Acked Read 2 Sent Done Done false
14451 4533 Acked Write 10 Sent Done Done false
14452 4534 Acked Flush 0 Sent Done Done false
14453 4535 Acked Read 4 Sent Done Done false
14454 4536 Acked Flush 0 Sent Done Done false
14455 4537 Acked Write 3 Sent Done Done false
14456 4538 Acked Read 8 Sent Done Done false
14457 4539 Acked Write 1 Sent Done Done false
14458 4540 Acked Read 9 Sent Done Done false
14459 4541 Acked Write 6 Sent Done Done false
14460 4542 Acked Read 8 Sent Done Done false
14461 4543 Acked Read 10 Sent Done Done false
14462 4544 Acked Flush 0 Sent Done Done false
14463 4545 Acked Read 6 Sent Done Done false
14464 4546 Acked Read 10 Sent Done Done false
14465 4547 Acked Flush 0 Sent Done Done false
14466 4548 Acked Read 2 Sent Done Done false
14467 4549 Acked Write 9 Sent Done Done false
14468 4550 Acked Write 6 Sent Done Done false
14469 4551 Acked Read 10 Sent Done Done false
14470 4552 Acked Read 5 Sent Done Done false
14471 4553 Acked Read 8 Sent Done Done false
14472 4554 Acked Read 2 Sent Done Done false
14473 4555 Acked Write 10 Sent Done Done false
14474 4556 Acked Flush 0 Sent Done Done false
14475 4557 Acked Write 6 Sent Done Done false
14476 4558 Acked Read 4 Sent Done Done false
14477 4559 Acked Read 1 Sent Done Done false
14478 4560 Acked Flush 0 Sent Done Done false
14479 4561 Acked Write 10 Sent Done Done false
14480 4562 Acked Read 4 Sent Done Done false
14481 4563 Acked Read 3 Sent Done Done false
14482 4564 Acked Read 8 Sent Done Done false
14483 4565 Acked Flush 0 Sent Done Done false
14484 4566 Acked Read 3 Sent Done Done false
14485 4567 Acked Read 2 Sent Done Done false
14486 4568 Acked Read 8 Sent Done Done false
14487 4569 Acked Read 4 Sent Done Done false
14488 4570 Acked Write 10 Sent Done Done false
14489 4571 Acked Write 6 Sent Done Done false
14490 4572 Acked Flush 0 Sent Done Done false
14491 4573 Acked Read 8 Sent Done Done false
14492 4574 Acked Read 9 Sent Done Done false
14493 4575 Acked Read 9 Sent Done Done false
14494 4576 Acked Write 3 Sent Done Done false
14495 4577 Acked Write 5 Sent Done Done false
14496 4578 Acked Read 3 Sent Done Done false
14497 4579 Acked Flush 0 Sent Done Done false
14498 4580 Acked Write 1 Sent Done Done false
14499 4581 Acked Read 7 Sent Done Done false
14500 4582 Acked Read 8 Sent Done Done false
14501 4583 Acked Read 6 Sent Done Done false
14502 4584 Acked Write 7 Sent Done Done false
14503 4585 Acked Flush 0 Sent Done Done false
14504 4586 Acked Read 7 Sent Done Done false
14505 4587 Acked Read 9 Sent Done Done false
14506 4588 Acked Flush 0 Sent Done Done false
14507 4589 Acked Flush 0 Sent Done Done false
14508 4590 Acked Write 6 Sent Done Done false
14509 4591 Acked Read 9 Sent Done Done false
14510 4592 Acked Read 4 Sent Done Done false
14511 4593 Acked Write 4 Sent Done Done false
14512 4594 Acked Write 8 Sent Done Done false
14513 4595 Acked Write 7 Sent Done Done false
14514 4596 Acked Write 4 Sent Done Done false
14515 4597 Acked Write 7 Sent Done Done false
14516 4598 Acked Read 6 Sent Done Done false
14517 4599 Acked Write 8 Sent Done Done false
14518 4600 Acked Write 9 Sent Done Done false
14519 4601 Acked Read 8 Sent Done Done false
14520 4602 Acked Read 6 Sent Done Done false
14521 4603 Acked Write 6 Sent Done Done false
14522 4604 Acked Read 8 Sent Done Done false
14523 4605 Acked Read 6 Sent Done Done false
14524 4606 Acked Write 8 Sent Done Done false
14525 4607 Acked Write 6 Sent Done Done false
14526 4608 Acked Write 9 Sent Done Done false
14527 4609 Acked Flush 0 Sent Done Done false
14528 4610 Acked Read 1 Sent Done Done false
14529 4611 Acked Write 9 Sent Done Done false
14530 4612 Acked Read 1 Sent Done Done false
14531 4613 Acked Read 1 Sent Done Done false
14532 4614 Acked Read 4 Sent Done Done false
14533 4615 Acked Read 10 Sent Done Done false
14534 4616 Acked Write 2 Sent Done Done false
14535 4617 Acked Read 2 Sent Done Done false
14536 4618 Acked Write 5 Sent Done Done false
14537 4619 Acked Write 8 Sent Done Done false
14538 4620 Acked Flush 0 Sent Done Done false
14539 4621 Acked Read 1 Sent Done Done false
14540 4622 Acked Write 10 Sent Done Done false
14541 STATES DS:0 DS:1 DS:2 TOTAL
14542 Sent 301 0 0 301
14543 Done 0 301 301 602
14544 Skipped 0 0 0 0
14545 Error 0 0 0 0
14546 Last Flush: Some(JobId(4321)) Some(JobId(4620)) Some(JobId(4620))
14547 Downstairs last five completed: 4321 4320 4319 4318 4317
14548 Upstairs last five completed: 4622 4621 4620 4619 4618
14549 CLIENT: Up:0 ds:301 act:2
145502025-03-17T22:37:44.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145512025-03-17T22:37:45.613ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145522025-03-17T22:37:45.647ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145532025-03-17T22:37:45.647ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58314) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145542025-03-17T22:37:45.648ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145552025-03-17T22:37:45.648ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145562025-03-17T22:37:45.649ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145572025-03-17T22:37:45.649ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145582025-03-17T22:37:45.649ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145592025-03-17T22:37:45.649ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145602025-03-17T22:37:45.649ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
14561 ----------------------------------------------------------------
14562 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14563 Downstairs last five completed: 4623 4622 4621 4620 4619
14564 Upstairs last five completed: 4623 4622 4621 4620 4619
14565 CLIENT: Up:0 ds:0 act:3
14566 Replay: All jobs finished
14567 Main task has work to do, go find it
14568 got dsc Stop(1)
14569 stop 1
14570 Mar 17 22:37:47.899 INFO request completed, latency_us: 524, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 70ba3431-e477-4a96-aa8c-16c276efe56f, remote_addr: 127.0.[8820] Got stop action so:false kr:false
14571 0.1:35180, local_addr: 127.0.0.1:9998
14572 Mar 17 22:37:47.900 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/1, method: GET, req_id: efb9ec15-8961-41d1-8473-3b7147abb735, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
145732025-03-17T22:37:47.900ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145742025-03-17T22:37:47.900ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145752025-03-17T22:37:47.900ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
145762025-03-17T22:37:47.900ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
14577 [8820] Exited with: ExitStatus(unix_wait_status(9))
14578 [8820][1] reports Exit
14579 Mar 17 22:37:51.902 INFO request completed, latency_us: 309, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4ba61068-9381-45b0-b501-34bb2a9c2266, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
14580 001/300 Write block 3878 len 28672 data: 7 6 5 6 7 5 4
14581 002/300 Read block 3722 len 40960
14582 003/300 Flush
14583 004/300 Read block 2108 len 40960
14584 005/300 Read block 1735 len 24576
14585 006/300 Read block 1749 len 32768
14586 007/300 Write block 816 len 8192 data: 6 6
14587 008/300 Read block 2800 len 28672
14588 009/300 Write block 3088 len 12288 data: 4 3 3
14589 010/300 Read block 4787 len 12288
14590 011/300 Read block 138 len 12288
14591 012/300 Read block 204 len 12288
14592 013/300 Read block 2508 len 4096
14593 014/300 Flush
14594 015/300 Read block 4156 len 16384
14595 016/300 Write block 4340 len 36864 data: 5 4 4 3 5 5 6 4 4
14596 017/300 Write block 3232 len 16384 data: 3 3 3 3
14597 018/300 Read block 2735 len 16384
14598 019/300 Write block 4487 len 32768 data: 2 3 3 3 2 2 4 4
14599 020/300 Write block 4008 len 36864 data: 3 4 4 4 5 5 4 4 4
14600 021/300 Read block 2277 len 28672
14601 022/300 Read block 976 len 36864
14602 023/300 Write block 4604 len 40960 data: 4 4 3 2 2 2 2 3 5 5
14603 024/300 Write block 4599 len 36864 data: 4 4 5 5 5 5 5 4 3
14604 025/300 Write block 2007 len 36864 data: 4 2 3 3 3 3 3 3 3
14605 026/300 Flush
14606 027/300 Read block 515 len 32768
14607 028/300 Write block 3545 len 28672 data: 3 5 5 5 5 4 5
14608 029/300 Read block 4835 len 8192
14609 030/300 Read block 194 len 20480
14610 031/300 Write block 671 len 40960 data: 5 5 4 4 5 5 6 5 4 4
14611 032/300 Read block 3063 len 36864
14612 033/300 Write block 2739 len 40960 data: 4 4 5 5 3 3 2 3 4 6
14613 034/300 Read block 478 len 28672
14614 035/300 Read block 1135 len 32768
14615 036/300 Read block 4108 len 8192
14616 037/300 Flush
14617 038/300 Write block 2058 len 4096 data: 2
14618 039/300 Flush
14619 040/300 Read block 2973 len 36864
14620 041/300 Write block 981 len 12288 data: 5 6 6
14621 042/300 Write block 745 len 24576 data: 4 4 5 7 5 5
14622 043/300 Write block 3543 len 12288 data: 2 3 4
14623 044/300 Write block 3275 len 28672 data: 2 2 2 2 2 3 3
14624 045/300 Flush
14625 046/300 Write block 4003 len 4096 data: 4
14626 047/300 Write block 3214 len 8192 data: 7 7
14627 048/300 Read block 125 len 24576
14628 049/300 Read block 2200 len 24576
14629 050/300 Flush
14630 051/300 Read block 503 len 36864
14631 052/300 Read block 3679 len 40960
14632 053/300 Write block 3890 len 28672 data: 3 4 3 3 3 3 4
14633 054/300 Write block 1372 len 28672 data: 2 2 3 4 4 5 5
14634 055/300 Write block 2224 len 4096 data: 2
14635 056/300 Read block 4139 len 32768
14636 057/300 Read block 1283 len 12288
14637 058/300 Write block 3077 len 20480 data: 3 3 3 2 3
14638 059/300 Write block 1731 len 24576 data: 4 4 4 4 3 4
14639 060/300 Write block 3276 len 12288 data: 3 3 3
14640 061/300 Read block 1977 len 32768
14641 062/300 Flush
14642 063/300 Read block 3705 len 8192
14643 064/300 Read block 3298 len 40960
14644 065/300 Read block 3275 len 40960
14645 066/300 Write block 3235 len 24576 data: 4 3 4 4 4 4
14646 067/300 Read block 4054 len 8192
14647 068/300 Write block 66 len 8192 data: 4 4
14648 069/300 Read block 3877 len 28672
14649 070/300 Read block 122 len 20480
14650 071/300 Read block 3971 len 28672
14651 072/300 Write block 3405 len 36864 data: 3 4 4 4 6 6 5 6 6
14652 073/300 Read block 3145 len 28672
14653 074/300 Read block 3561 len 28672
14654 075/300 Write block 2112 len 40960 data: 2 2 3 4 4 4 3 4 4 2
14655 076/300 Read block 271 len 40960
14656 077/300 Read block 2588 len 20480
14657 078/300 Read block 3610 len 4096
14658 079/300 Read block 2591 len 40960
14659 080/300 Read block 1429 len 28672
14660 081/300 Write block 421 len 20480 data: 5 5 4 4 5
14661 082/300 Read block 4540 len 32768
14662 083/300 Read block 3313 len 36864
14663 084/300 Read block 2372 len 24576
14664 085/300 Write block 1904 len 4096 data: 4
14665 086/300 Write block 2129 len 12288 data: 2 2 2
14666 087/300 Read block 3389 len 32768
14667 088/300 Flush
14668 089/300 Read block 4453 len 40960
14669 090/300 Read block 3653 len 4096
14670 091/300 Flush
14671 092/300 Read block 4709 len 40960
14672 093/300 Read block 4503 len 20480
14673 094/300 Read block 3946 len 28672
14674 095/300 Read block 1918 len 24576
14675 096/300 Write block 4221 len 20480 data: 4 6 4 4 4
14676 097/300 Write block 2600 len 16384 data: 4 4 4 4
14677 098/300 Read block 4171 len 36864
14678 099/300 Write block 1811 len 8192 data: 4 4
14679 100/300 Write block 3882 len 32768 data: 8 6 5 4 4 4 4 3
14680 101/300 Write block 1409 len 40960 data: 2 2 3 3 3 5 6 6 6 6
14681 102/300 Write block 2477 len 20480 data: 5 5 5 5 5
14682 103/300 Write block 1878 len 24576 data: 4 2 3 3 4 3
14683 104/300 Read block 3320 len 12288
14684 105/300 Write block 3066 len 16384 data: 2 2 2 3
14685 106/300 Read block 2567 len 32768
14686 107/300 Write block 2530 len 36864 data: 4 4 5 4 5 4 4 4 4
14687 108/300 Write block 2944 len 4096 data: 4
14688 109/300 Read block 4366 len 16384
14689 110/300 Read block 556 len 24576
14690 111/300 Read block 3422 len 16384
14691 112/300 Read block 2100 len 16384
14692 113/300 Flush
14693 114/300 Read block 3092 len 16384
14694 115/300 Read block 3192 len 28672
14695 116/300 Read block 4097 len 24576
14696 117/300 Read block 4359 len 4096
14697 118/300 Write block 4230 len 32768 data: 4 5 6 6 5 5 4 4
14698 119/300 Read block 3123 len 16384
14699 120/300 Read block 4322 len 24576
14700 121/300 Write block 3687 len 20480 data: 4 4 4 4 3
14701 122/300 Write block 4844 len 12288 data: 5 5 5
14702 123/300 Flush
14703 124/300 Write block 1982 len 32768 data: 3 4 4 4 4 4 4 3
14704 125/300 Write block 2125 len 36864 data: 2 2 2 2 3 3 3 3 3
14705 126/300 Read block 259 len 28672
14706 127/300 Write block 2916 len 36864 data: 4 5 2 2 3 4 4 4 4
14707 128/300 Read block 4151 len 40960
14708 129/300 Read block 4086 len 12288
14709 130/300 Read block 2207 len 36864
14710 131/300 Write block 199 len 28672 data: 4 4 3 2 2 2 2
14711 132/300 Read block 1915 len 32768
14712 133/300 Write block 2779 len 24576 data: 2 2 2 2 3 3
14713 134/300 Read block 1176 len 8192
14714 135/300 Write block 3105 len 16384 data: 4 3 3 4
14715 136/300 Write block 640 len 8192 data: 3 3
14716 137/300 Flush
14717 138/300 Read block 3237 len 4096
14718 139/300 Flush
14719 140/300 Read block 2203 len 36864
14720 141/300 Read block 4687 len 36864
14721 142/300 Read block 3378 len 24576
14722 143/300 Write block 1979 len 20480 data: 2 2 2 4 5
14723 144/300 Read block 517 len 36864
14724 145/300 Flush
14725 146/300 Read block 1241 len 24576
14726 147/300 Read block 4815 len 12288
14727 148/300 Flush
14728 149/300 Read block 415 len 40960
14729 150/300 Write block 1217 len 16384 data: 2 2 2 2
14730 151/300 Write block 4989 len 12288 data: 3 2 2
14731 152/300 Read block 4209 len 12288
14732 153/300 Write block 436 len 4096 data: 3
14733 154/300 Read block 1030 len 12288
14734 155/300 Write block 4933 len 20480 data: 3 4 4 4 4
14735 156/300 Flush
14736 157/300 Read block 2659 len 4096
14737 158/300 Write block 1599 len 4096 data: 6
14738 159/300 Write block 3962 len 40960 data: 4 4 4 4 4 2 2 2 2 2
14739 160/300 Write block 2037 len 28672 data: 5 5 5 6 5 6 6
14740 161/300 Write block 864 len 12288 data: 2 3 3
14741 162/300 Flush
14742 163/300 Read block 3881 len 36864
14743 164/300 Read block 1864 len 16384
14744 165/300 Write block 1722 len 28672 data: 4 4 4 5 5 4 4
14745 166/300 Write block 4114 len 28672 data: 5 5 5 4 5 5 4
14746 167/300 Read block 4324 len 20480
14747 168/300 Write block 3467 len 24576 data: 2 2 2 3 4 4
14748 169/300 Read block 3978 len 8192
14749 170/300 Write block 2096 len 28672 data: 3 3 3 4 4 3 3
14750 171/300 Read block 2215 len 40960
14751 172/300 Read block 4346 len 12288
14752 173/300 Write block 1243 len 20480 data: 4 4 3 3 2
14753 174/300 Write block 1423 len 12288 data: 3 4 5
14754 175/300 Read block 1556 len 4096
14755 176/300 Write block 3941 len 28672 data: 4 4 4 3 4 4 3
14756 177/300 Flush
14757 178/300 Read block 1089 len 36864
14758 179/300 Write block 4653 len 36864 data: 3 3 3 3 2 3 4 4 4
14759 180/300 Write block 4431 len 4096 data: 4
14760 181/300 Read block 3913 len 16384
14761 182/300 Read block 3439 len 8192
14762 183/300 Read block 4772 len 12288
14763 184/300 Flush
14764 185/300 Write block 3739 len 4096 data: 4
14765 186/300 Read block 4364 len 12288
14766 187/300 Read block 4483 len 40960
14767 188/300 Write block 3944 len 16384 data: 4 5 5 4
14768 189/300 Write block 4353 len 24576 data: 8 7 6 6 6 5
14769 190/300 Read block 3483 len 40960
14770 191/300 Write block 569 len 4096 data: 3
14771 192/300 Write block 2196 len 40960 data: 3 2 2 2 2 3 4 4 4 6
14772 193/300 Read block 2804 len 36864
14773 194/300 Write block 7 len 4096 data: 2
14774 195/300 Write block 1435 len 16384 data: 4 6 5 3
14775 196/300 Write block 428 len 32768 data: 3 3 3 4 4 4 4 3
14776 197/300 Flush
14777 198/300 Flush
14778 199/300 Read block 486 len 8192
14779 200/300 Read block 1924 len 28672
14780 201/300 Flush
14781 202/300 Read block 4470 len 40960
14782 203/300 Read block 2113 len 12288
14783 204/300 Read block 1897 len 28672
14784 205/300 Flush
14785 206/300 Flush
14786 207/300 Read block 3434 len 32768
14787 208/300 Read block 1275 len 16384
14788 209/300 Write block 2676 len 8192 data: 2 2
14789 210/300 Write block 2659 len 16384 data: 3 3 3 4
14790 211/300 Read block 4017 len 32768
14791 212/300 Write block 1267 len 28672 data: 4 2 2 2 3 3 3
14792 213/300 Read block 1870 len 36864
14793 214/300 Read block 4801 len 12288
14794 215/300 Write block 4049 len 28672 data: 2 2 2 2 2 2 2
14795 216/300 Flush
14796 217/300 Write block 1815 len 16384 data: 2 2 3 4
14797 218/300 Read block 1552 len 4096
14798 219/300 Write block 436 len 20480 data: 4 2 3 4 4
14799 220/300 Write block 1896 len 4096 data: 4
14800 221/300 Write block 384 len 28672 data: 4 5 4 4 4 3 3
14801 222/300 Read block 3890 len 20480
14802 223/300 Read block 1072 len 28672
14803 224/300 Write block 1969 len 16384 data: 2 2 2 2
14804 225/300 Read block 2601 len 16384
14805 226/300 Read block 4503 len 8192
14806 227/300 Read block 880 len 24576
14807 228/300 Read block 386 len 24576
14808 229/300 Read block 2632 len 4096
14809 230/300 Write block 1096 len 28672 data: 2 2 2 3 4 4 4
14810 231/300 Read block 2515 len 32768
14811 232/300 Flush
14812 233/300 Read block 4525 len 40960
14813 234/300 Flush
14814 235/300 Read block 2355 len 40960
14815 236/300 Write block 2528 len 32768 data: 3 3 5 5 6 5 6 5
14816 237/300 Flush
14817 238/300 Write block 3022 len 36864 data: 3 3 3 3 3 3 3 4 4
14818 239/300 Read block 1317 len 8192
14819 240/300 Flush
14820 241/300 Read block 901 len 12288
14821 242/300 Read block 657 len 4096
14822 243/300 Write block 2195 len 8192 data: 3 4
14823 244/300 Read block 3936 len 40960
14824 245/300 Read block 1247 len 20480
14825 246/300 Flush
14826 247/300 Read block 3976 len 24576
14827 248/300 Write block 3943 len 20480 data: 5 5 6 6 5
14828 249/300 Write block 4687 len 40960 data: 2 4 4 4 4 3 3 3 3 4
14829 250/300 Read block 648 len 36864
14830 251/300 Flush
14831 252/300 Read block 4617 len 24576
14832 253/300 Read block 56 len 40960
14833 254/300 Write block 3011 len 28672 data: 3 3 3 2 3 4 5
14834 255/300 Read block 3222 len 32768
14835 256/300 Read block 3405 len 16384
14836 257/300 Read block 1660 len 20480
14837 258/300 Read block 3894 len 20480
14838 259/300 Write block 1017 len 12288 data: 2 3 3
14839 260/300 Write block 1532 len 4096 data: 6
14840 261/300 Write block 3158 len 36864 data: 2 2 2 4 4 4 4 3 3
14841 262/300 Flush
14842 263/300 Write block 2976 len 40960 data: 4 4 6 7 7 6 5 5 4 4
14843 264/300 Read block 3254 len 40960
14844 265/300 Flush
14845 266/300 Read block 611 len 32768
14846 267/300 Write block 4866 len 40960 data: 4 5 4 5 4 5 5 4 4 3
14847 268/300 Flush
14848 269/300 Read block 459 len 36864
14849 270/300 Read block 3169 len 4096
14850 271/300 Write block 1681 len 24576 data: 2 2 3 3 3 3
14851 272/300 Read block 1671 len 4096
14852 273/300 Write block 1543 len 16384 data: 3 3 4 4
14853 274/300 Write block 3465 len 28672 data: 2 2 3 3 3 4 5
14854 275/300 Read block 2224 len 8192
14855 276/300 Read block 2224 len 36864
14856 277/300 Read block 2987 len 12288
14857 278/300 Write block 1419 len 32768 data: 4 3 3 3 4 5 6 4
14858 279/300 Read block 3206 len 36864
14859 280/300 Write block 1879 len 12288 data: 3 4 4
14860 281/300 Flush
14861 282/300 Write block 4212 len 28672 data: 2 2 3 3 5 5 5
14862 283/300 Read block 3507 len 16384
14863 284/300 Read block 1804 len 20480
14864 285/300 Read block 4838 len 32768
14865 286/300 Read block 2228 len 12288
14866 287/300 Read block 1271 len 24576
14867 288/300 Flush
14868 289/300 Read block 4363 len 40960
14869 290/300 Write block 3098 len 4096 data: 3
14870 291/300 Flush
14871 292/300 Write block 2675 len 28672 data: 3 3 3 2 2 2 2
14872 293/300 Read block 1514 len 28672
14873 294/300 Read block 4249 len 4096
14874 295/300 Write block 1212 len 24576 data: 3 2 2 2 2 3
14875 296/300 Read block 3920 len 36864
14876 297/300 Read block 3964 len 20480
14877 298/300 Read block 623 len 24576
14878 299/300 Write block 4533 len 40960 data: 6 4 5 5 5 4 4 3 4 3
14879 300/300 Read block 1010 len 12288
14880 301/300 Read block 4775 len 20480
14881 Main task has work to do, go find it
14882 got dsc Start(1)
14883 start 1
14884 Mar 17 22:37:52.156 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/1, method: POST, req_id: e1abdb4e-d278-4bfb-a42c-d31e3813abcf, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
14885 Replay: started 1, returned:Ok(())
14886 ----------------------------------------------------------------
14887 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14888 States: Active Offline Active
14889 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14890 4624 Acked Write 7 Done Sent Done false
14891 4625 Acked Read 10 Done Sent Done false
14892 4626 Acked Flush 0 Done Sent Done false
14893 4627 Acked Read 10 Done Sent Done false
14894 4628 Acked Read 6 Done Sent Done false
14895 4629 Acked Read 8 Done Sent Done false
14896 4630 Acked Write 2 Done Sent Done false
14897 4631 Acked Read 7 Done Sent Done false
14898 4632 Acked Write 3 Done Sent Done false
14899 4633 Acked Read 3 Done Sent Done false
14900 4634 Acked Read 3 Done Sent Done false
14901 4635 Acked Read 3 Done Sent Done false
14902 4636 Acked Read 1 Done Sent Done false
14903 4637 Acked Flush 0 Done Sent Done false
14904 4638 Acked Read 4 Done Sent Done false
14905 4639 Acked Write 9 Done Sent Done false
14906 4640 Acked Write 4 Done Sent Done false
14907 4641 Acked Read 4 Done Sent Done false
14908 4642 Acked Write 8 Done Sent Done false
14909 4643 Acked Write 9 Done Sent Done false
14910 4644 Acked Read 7 Done Sent Done false
14911 4645 Acked Read 9 Done Sent Done false
14912 4646 Acked Write 10 Done Sent Done false
14913 4647 Acked Write 9 Done Sent Done false
14914 4648 Acked Write 9 Done Sent Done false
14915 4649 Acked Flush 0 Done Sent Done false
14916 4650 Acked Read 8 Done Sent Done false
14917 4651 Acked Write 7 Done Sent Done false
14918 4652 Acked Read 2 Done Sent Done false
14919 4653 Acked Read 5 Done Sent Done false
14920 4654 Acked Write 10 Done Sent Done false
14921 4655 Acked Read 9 Done Sent Done false
14922 4656 Acked Write 10 Done Sent Done false
14923 4657 Acked Read 7 Done Sent Done false
14924 4658 Acked Read 8 Done Sent Done false
14925 4659 Acked Read 2 Done Sent Done false
14926 4660 Acked Flush 0 Done Sent Done false
14927 4661 Acked Write 1 Done Sent Done false
14928 4662 Acked Flush 0 Done Sent Done false
14929 4663 Acked Read 9 Done Sent Done false
14930 4664 Acked Write 3 Done Sent Done false
14931 4665 Acked Write 6 Done Sent Done false
14932 4666 Acked Write 3 Done Sent Done false
14933 4667 Acked Write 7 Done Sent Done false
14934 4668 Acked Flush 0 Done Sent Done false
14935 4669 Acked Write 1 Done Sent Done false
14936 4670 Acked Write 2 Done Sent Done false
14937 4671 Acked Read 6 Done Sent Done false
14938 4672 Acked Read 6 Done Sent Done false
14939 4673 Acked Flush 0 Done Sent Done false
14940 4674 Acked Read 9 Done Sent Done false
14941 4675 Acked Read 10 Done Sent Done false
14942 4676 Acked Write 7 Done Sent Done false
14943 4677 Acked Write 7 Done Sent Done false
14944 4678 Acked Write 1 Done Sent Done false
14945 4679 Acked Read 8 Done Sent Done false
14946 4680 Acked Read 3 Done Sent Done false
14947 4681 Acked Write 5 Done Sent Done false
14948 4682 Acked Write 6 Done Sent Done false
14949 4683 Acked Write 3 Done Sent Done false
14950 4684 Acked Read 8 Done Sent Done false
14951 4685 Acked Flush 0 Done Sent Done false
14952 4686 Acked Read 2 Done Sent Done false
14953 4687 Acked Read 10 Done Sent Done false
14954 4688 Acked Read 10 Done Sent Done false
14955 4689 Acked Write 6 Done Sent Done false
14956 4690 Acked Read 2 Done Sent Done false
14957 4691 Acked Write 2 Done Sent Done false
14958 4692 Acked Read 7 Done Sent Done false
14959 4693 Acked Read 5 Done Sent Done false
14960 4694 Acked Read 7 Done Sent Done false
14961 4695 Acked Write 9 Done Sent Done false
14962 4696 Acked Read 7 Done Sent Done false
14963 4697 Acked Read 7 Done Sent Done false
14964 4698 Acked Write 10 Done Sent Done false
14965 4699 Acked Read 10 Done Sent Done false
14966 4700 Acked Read 5 Done Sent Done false
14967 4701 Acked Read 1 Done Sent Done false
14968 4702 Acked Read 10 Done Sent Done false
14969 4703 Acked Read 7 Done Sent Done false
14970 4704 Acked Write 5 Done Sent Done false
14971 4705 Acked Read 8 Done Sent Done false
14972 4706 Acked Read 9 Done Sent Done false
14973 4707 Acked Read 6 Done Sent Done false
14974 4708 Acked Write 1 Done Sent Done false
14975 4709 Acked Write 3 Done Sent Done false
14976 4710 Acked Read 8 Done Sent Done false
14977 4711 Acked Flush 0 Done Sent Done false
14978 4712 Acked Read 10 Done Sent Done false
14979 4713 Acked Read 1 Done Sent Done false
14980 4714 Acked Flush 0 Done Sent Done false
14981 4715 Acked Read 10 Done Sent Done false
14982 4716 Acked Read 5 Done Sent Done false
14983 4717 Acked Read 7 Done Sent Done false
14984 4718 Acked Read 6 Done Sent Done false
14985 4719 Acked Write 5 Done Sent Done false
14986 4720 Acked Write 4 Done Sent Done false
14987 4721 Acked Read 9 Done Sent Done false
14988 4722 Acked Write 2 Done Sent Done false
14989 4723 Acked Write 8 Done Sent Done false
14990 4724 Acked Write 10 Done Sent Done false
14991 4725 Acked Write 5 Done Sent Done false
14992 4726 Acked Write 6 Done Sent Done false
14993 4727 Acked Read 3 Done Sent Done false
14994 4728 Acked Write 4 Done Sent Done false
14995 4729 Acked Read 8 Done Sent Done false
14996 4730 Acked Write 9 Done Sent Done false
14997 4731 Acked Write 1 Done Sent Done false
14998 4732 Acked Read 4 Done Sent Done false
14999 4733 Acked Read 6 Done Sent Done false
15000 4734 Acked Read 4 Done Sent Done false
15001 4735 Acked Read 4 Done Sent Done false
15002 4736 Acked Flush 0 Done Sent Done false
15003 4737 Acked Read 4 Done Sent Done false
15004 4738 Acked Read 7 Done Sent Done false
15005 4739 Acked Read 6 Done Sent Done false
15006 4740 Acked Read 1 Done Sent Done false
15007 4741 Acked Write 8 Done Sent Done false
15008 4742 Acked Read 4 Done Sent Done false
15009 4743 Acked Read 6 Done Sent Done false
15010 4744 Acked Write 5 Done Sent Done false
15011 4745 Acked Write 3 Done Sent Done false
15012 4746 Acked Flush 0 Done Sent Done false
15013 4747 Acked Write 8 Done Sent Done false
15014 4748 Acked Write 9 Done Sent Done false
15015 4749 Acked Read 7 Done Sent Done false
15016 4750 Acked Write 9 Done Sent Done false
15017 4751 Acked Read 10 Done Sent Done false
15018 4752 Acked Read 3 Done Sent Done false
15019 4753 Acked Read 9 Done Sent Done false
15020 4754 Acked Write 7 Done Sent Done false
15021 4755 Acked Read 8 Done Sent Done false
15022 4756 Acked Write 6 Done Sent Done false
15023 4757 Acked Read 2 Done Sent Done false
15024 4758 Acked Write 4 Done Sent Done false
15025 4759 Acked Write 2 Done Sent Done false
15026 4760 Acked Flush 0 Done Sent Done false
15027 4761 Acked Read 1 Done Sent Done false
15028 4762 Acked Flush 0 Done Sent Done false
15029 4763 Acked Read 9 Done Sent Done false
15030 4764 Acked Read 9 Done Sent Done false
15031 4765 Acked Read 6 Done Sent Done false
15032 4766 Acked Write 5 Done Sent Done false
15033 4767 Acked Read 9 Done Sent Done false
15034 4768 Acked Flush 0 Done Sent Done false
15035 4769 Acked Read 6 Done Sent Done false
15036 4770 Acked Read 3 Done Sent Done false
15037 4771 Acked Flush 0 Done Sent Done false
15038 4772 Acked Read 10 Done Sent Done false
15039 4773 Acked Write 4 Done Sent Done false
15040 4774 Acked Write 3 Done Sent Done false
15041 4775 Acked Read 3 Done Sent Done false
15042 4776 Acked Write 1 Done Sent Done false
15043 4777 Acked Read 3 Done Sent Done false
15044 4778 Acked Write 5 Done Sent Done false
15045 4779 Acked Flush 0 Done Sent Done false
15046 4780 Acked Read 1 Done Sent Done false
15047 4781 Acked Write 1 Done Sent Done false
15048 4782 Acked Write 10 Done Sent Done false
15049 4783 Acked Write 7 Done Sent Done false
15050 4784 Acked Write 3 Done Sent Done false
15051 4785 Acked Flush 0 Done Sent Done false
15052 4786 Acked Read 9 Done Sent Done false
15053 4787 Acked Read 4 Done Sent Done false
15054 4788 Acked Write 7 Done Sent Done false
15055 4789 Acked Write 7 Done Sent Done false
15056 4790 Acked Read 5 Done Sent Done false
15057 4791 Acked Write 6 Done Sent Done false
15058 4792 Acked Read 2 Done Sent Done false
15059 4793 Acked Write 7 Done Sent Done false
15060 4794 Acked Read 10 Done Sent Done false
15061 4795 Acked Read 3 Done Sent Done false
15062 4796 Acked Write 5 Done Sent Done false
15063 4797 Acked Write 3 Done Sent Done false
15064 4798 Acked Read 1 Done Sent Done false
15065 4799 Acked Write 7 Done Sent Done false
15066 4800 Acked Flush 0 Done Sent Done false
15067 4801 Acked Read 9 Done Sent Done false
15068 4802 Acked Write 9 Done Sent Done false
15069 4803 Acked Write 1 Done Sent Done false
15070 4804 Acked Read 4 Done Sent Done false
15071 4805 Acked Read 2 Done Sent Done false
15072 4806 Acked Read 3 Done Sent Done false
15073 4807 Acked Flush 0 Done Sent Done false
15074 4808 Acked Write 1 Done Sent Done false
15075 4809 Acked Read 3 Done Sent Done false
15076 4810 Acked Read 10 Done Sent Done false
15077 4811 Acked Write 4 Done Sent Done false
15078 4812 Acked Write 6 Done Sent Done false
15079 4813 Acked Read 10 Done Sent Done false
15080 4814 Acked Write 1 Done Sent Done false
15081 4815 Acked Write 10 Done Sent Done false
15082 4816 Acked Read 9 Done Sent Done false
15083 4817 Acked Write 1 Done Sent Done false
15084 4818 Acked Write 4 Done Sent Done false
15085 4819 Acked Write 8 Done Sent Done false
15086 4820 Acked Flush 0 Done Sent Done false
15087 4821 Acked Flush 0 Done Sent Done false
15088 4822 Acked Read 2 Done Sent Done false
15089 4823 Acked Read 7 Done Sent Done false
15090 4824 Acked Flush 0 Done Sent Done false
15091 4825 Acked Read 10 Done Sent Done false
15092 4826 Acked Read 3 Done Sent Done false
15093 4827 Acked Read 7 Done Sent Done false
15094 4828 Acked Flush 0 Done Sent Done false
15095 4829 Acked Flush 0 Done Sent Done false
15096 4830 Acked Read 8 Done Sent Done false
15097 4831 Acked Read 4 Done Sent Done false
15098 4832 Acked Write 2 Done Sent Done false
15099 4833 Acked Write 4 Done Sent Done false
15100 4834 Acked Read 8 Done Sent Done false
15101 4835 Acked Write 7 Done Sent Done false
15102 4836 Acked Read 9 Done Sent Done false
15103 4837 Acked Read 3 Done Sent Done false
15104 4838 Acked Write 7 Done Sent Done false
15105 4839 Acked Flush 0 Done Sent Done false
15106 4840 Acked Write 4 Done Sent Done false
15107 4841 Acked Read 1 Done Sent Done false
15108 4842 Acked Write 5 Done Sent Done false
15109 4843 Acked Write 1 Done Sent Done false
15110 4844 Acked Write 7 Done Sent Done false
15111 4845 Acked Read 5 Done Sent Done false
15112 4846 Acked Read 7 Done Sent Done false
15113 4847 Acked Write 4 Done Sent Done false
15114 4848 Acked Read 4 Done Sent Done false
15115 4849 Acked Read 2 Done Sent Done false
15116 4850 Acked Read 6 Done Sent Done false
15117 4851 Acked Read 6 Done Sent Done false
15118 4852 Acked Read 1 Done Sent Done false
15119 4853 Acked Write 7 Done Sent Done false
15120 4854 Acked Read 8 Done Sent Done false
15121 4855 Acked Flush 0 Done Sent Done false
15122 4856 Acked Read 10 Done Sent Done false
15123 4857 Acked Flush 0 Done Sent Done false
15124 4858 Acked Read 10 Done Sent Done false
15125 4859 Acked Write 8 Done Sent Done false
15126 4860 Acked Flush 0 Done Sent Done false
15127 4861 Acked Write 9 Done Sent Done false
15128 4862 Acked Read 2 Done Sent Done false
15129 4863 Acked Flush 0 Done Sent Done false
15130 4864 Acked Read 3 Done Sent Done false
15131 4865 Acked Read 1 Done Sent Done false
15132 4866 Acked Write 2 Done Sent Done false
15133 4867 Acked Read 10 Done Sent Done false
15134 4868 Acked Read 5 Done Sent Done false
15135 4869 Acked Flush 0 Done Sent Done false
15136 4870 Acked Read 6 Done Sent Done false
15137 4871 Acked Write 5 Done Sent Done false
15138 4872 Acked Write 10 Done Sent Done false
15139 4873 Acked Read 9 Done Sent Done false
15140 4874 Acked Flush 0 Done Sent Done false
15141 4875 Acked Read 6 Done Sent Done false
15142 4876 Acked Read 10 Done Sent Done false
15143 4877 Acked Write 7 Done Sent Done false
15144 4878 Acked Read 8 Done Sent Done false
15145 4879 Acked Read 4 Done Sent Done false
15146 4880 Acked Read 5 Done Sent Done false
15147 4881 Acked Read 5 Done Sent Done false
15148 4882 Acked Write 3 Done Sent Done false
15149 4883 Acked Write 1 Done Sent Done false
15150 4884 Acked Write 9 Done Sent Done false
15151 4885 Acked Flush 0 Done Sent Done false
15152 4886 Acked Write 10 Done Sent Done false
15153 4887 Acked Read 10 Done Sent Done false
15154 4888 Acked Flush 0 Done Sent Done false
15155 4889 Acked Read 8 Done Sent Done false
15156 4890 Acked Write 10 Done Sent Done false
15157 4891 Acked Flush 0 Done Sent Done false
15158 4892 Acked Read 9 Done Sent Done false
15159 4893 Acked Read 1 Done Sent Done false
15160 4894 Acked Write 6 Done Sent Done false
15161 4895 Acked Read 1 Done Sent Done false
15162 4896 Acked Write 4 Done Sent Done false
15163 4897 Acked Write 7 Done Sent Done false
15164 4898 Acked Read 2 Done Sent Done false
15165 4899 Acked Read 9 Done Sent Done false
15166 4900 Acked Read 3 Done Sent Done false
15167 4901 Acked Write 8 Done Sent Done false
15168 4902 Acked Read 9 Done Sent Done false
15169 4903 Acked Write 3 Done Sent Done false
15170 4904 Acked Flush 0 Done Sent Done false
15171 4905 Acked Write 7 Done Sent Done false
15172 4906 Acked Read 4 Done Sent Done false
15173 4907 Acked Read 5 Done Sent Done false
15174 4908 Acked Read 8 Done Sent Done false
15175 4909 Acked Read 3 Done Sent Done false
15176 4910 Acked Read 6 Done Sent Done false
15177 4911 Acked Flush 0 Done Sent Done false
15178 4912 Acked Read 10 Done Sent Done false
15179 4913 Acked Write 1 Done Sent Done false
15180 4914 Acked Flush 0 Done Sent Done false
15181 4915 Acked Write 7 Done Sent Done false
15182 4916 Acked Read 7 Done Sent Done false
15183 4917 Acked Read 1 Done Sent Done false
15184 4918 Acked Write 6 Done Sent Done false
15185 4919 Acked Read 9 Done Sent Done false
15186 4920 Acked Read 5 Done Sent Done false
15187 4921 Acked Read 6 Done Sent Done false
15188 4922 Acked Write 10 Done Sent Done false
15189 4923 Acked Read 3 Done Sent Done false
15190 4924 Acked Read 5 Done Sent Done false
15191 STATES DS:0 DS:1 DS:2 TOTAL
15192 Sent 0 301 0 301
15193 Done 301 0 301 602
15194 Skipped 0 0 0 0
15195 Error 0 0 0 0
15196 Last Flush: Some(JobId(4914)) Some(JobId(4623)) Some(JobId(4914))
15197 Downstairs last five completed: 4623 4622 4621 4620 4619
15198 Upstairs last five completed: 4924 4923 4922 4921 4920
15199 CLIENT: Up:0 ds:301 act:2
152002025-03-17T22:37:52.658ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
152012025-03-17T22:37:53.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
152022025-03-17T22:37:54.630ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
152032025-03-17T22:37:55.632ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
15204 ----------------------------------------------------------------
15205 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15206 States: Active Offline Active
15207 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15208 4624 Acked Write 7 Done Sent Done false
15209 4625 Acked Read 10 Done Sent Done false
15210 4626 Acked Flush 0 Done Sent Done false
15211 4627 Acked Read 10 Done Sent Done false
15212 4628 Acked Read 6 Done Sent Done false
15213 4629 Acked Read 8 Done Sent Done false
15214 4630 Acked Write 2 Done Sent Done false
15215 4631 Acked Read 7 Done Sent Done false
15216 4632 Acked Write 3 Done Sent Done false
15217 4633 Acked Read 3 Done Sent Done false
15218 4634 Acked Read 3 Done Sent Done false
15219 4635 Acked Read 3 Done Sent Done false
15220 4636 Acked Read 1 Done Sent Done false
15221 4637 Acked Flush 0 Done Sent Done false
15222 4638 Acked Read 4 Done Sent Done false
15223 4639 Acked Write 9 Done Sent Done false
15224 4640 Acked Write 4 Done Sent Done false
15225 4641 Acked Read 4 Done Sent Done false
15226 4642 Acked Write 8 Done Sent Done false
15227 4643 Acked Write 9 Done Sent Done false
15228 4644 Acked Read 7 Done Sent Done false
15229 4645 Acked Read 9 Done Sent Done false
15230 4646 Acked Write 10 Done Sent Done false
15231 4647 Acked Write 9 Done Sent Done false
15232 4648 Acked Write 9 Done Sent Done false
15233 4649 Acked Flush 0 Done Sent Done false
15234 4650 Acked Read 8 Done Sent Done false
15235 4651 Acked Write 7 Done Sent Done false
15236 4652 Acked Read 2 Done Sent Done false
15237 4653 Acked Read 5 Done Sent Done false
15238 4654 Acked Write 10 Done Sent Done false
15239 4655 Acked Read 9 Done Sent Done false
15240 4656 Acked Write 10 Done Sent Done false
15241 4657 Acked Read 7 Done Sent Done false
15242 4658 Acked Read 8 Done Sent Done false
15243 4659 Acked Read 2 Done Sent Done false
15244 4660 Acked Flush 0 Done Sent Done false
15245 4661 Acked Write 1 Done Sent Done false
15246 4662 Acked Flush 0 Done Sent Done false
15247 4663 Acked Read 9 Done Sent Done false
15248 4664 Acked Write 3 Done Sent Done false
15249 4665 Acked Write 6 Done Sent Done false
15250 4666 Acked Write 3 Done Sent Done false
15251 4667 Acked Write 7 Done Sent Done false
15252 4668 Acked Flush 0 Done Sent Done false
15253 4669 Acked Write 1 Done Sent Done false
15254 4670 Acked Write 2 Done Sent Done false
15255 4671 Acked Read 6 Done Sent Done false
15256 4672 Acked Read 6 Done Sent Done false
15257 4673 Acked Flush 0 Done Sent Done false
15258 4674 Acked Read 9 Done Sent Done false
15259 4675 Acked Read 10 Done Sent Done false
15260 4676 Acked Write 7 Done Sent Done false
15261 4677 Acked Write 7 Done Sent Done false
15262 4678 Acked Write 1 Done Sent Done false
15263 4679 Acked Read 8 Done Sent Done false
15264 4680 Acked Read 3 Done Sent Done false
15265 4681 Acked Write 5 Done Sent Done false
15266 4682 Acked Write 6 Done Sent Done false
15267 4683 Acked Write 3 Done Sent Done false
15268 4684 Acked Read 8 Done Sent Done false
15269 4685 Acked Flush 0 Done Sent Done false
15270 4686 Acked Read 2 Done Sent Done false
15271 4687 Acked Read 10 Done Sent Done false
15272 4688 Acked Read 10 Done Sent Done false
15273 4689 Acked Write 6 Done Sent Done false
15274 4690 Acked Read 2 Done Sent Done false
15275 4691 Acked Write 2 Done Sent Done false
15276 4692 Acked Read 7 Done Sent Done false
15277 4693 Acked Read 5 Done Sent Done false
15278 4694 Acked Read 7 Done Sent Done false
15279 4695 Acked Write 9 Done Sent Done false
15280 4696 Acked Read 7 Done Sent Done false
15281 4697 Acked Read 7 Done Sent Done false
15282 4698 Acked Write 10 Done Sent Done false
15283 4699 Acked Read 10 Done Sent Done false
15284 4700 Acked Read 5 Done Sent Done false
15285 4701 Acked Read 1 Done Sent Done false
15286 4702 Acked Read 10 Done Sent Done false
15287 4703 Acked Read 7 Done Sent Done false
15288 4704 Acked Write 5 Done Sent Done false
15289 4705 Acked Read 8 Done Sent Done false
15290 4706 Acked Read 9 Done Sent Done false
15291 4707 Acked Read 6 Done Sent Done false
15292 4708 Acked Write 1 Done Sent Done false
15293 4709 Acked Write 3 Done Sent Done false
15294 4710 Acked Read 8 Done Sent Done false
15295 4711 Acked Flush 0 Done Sent Done false
15296 4712 Acked Read 10 Done Sent Done false
15297 4713 Acked Read 1 Done Sent Done false
15298 4714 Acked Flush 0 Done Sent Done false
15299 4715 Acked Read 10 Done Sent Done false
15300 4716 Acked Read 5 Done Sent Done false
15301 4717 Acked Read 7 Done Sent Done false
15302 4718 Acked Read 6 Done Sent Done false
15303 4719 Acked Write 5 Done Sent Done false
15304 4720 Acked Write 4 Done Sent Done false
15305 4721 Acked Read 9 Done Sent Done false
15306 4722 Acked Write 2 Done Sent Done false
15307 4723 Acked Write 8 Done Sent Done false
15308 4724 Acked Write 10 Done Sent Done false
15309 4725 Acked Write 5 Done Sent Done false
15310 4726 Acked Write 6 Done Sent Done false
15311 4727 Acked Read 3 Done Sent Done false
15312 4728 Acked Write 4 Done Sent Done false
15313 4729 Acked Read 8 Done Sent Done false
15314 4730 Acked Write 9 Done Sent Done false
15315 4731 Acked Write 1 Done Sent Done false
15316 4732 Acked Read 4 Done Sent Done false
15317 4733 Acked Read 6 Done Sent Done false
15318 4734 Acked Read 4 Done Sent Done false
15319 4735 Acked Read 4 Done Sent Done false
15320 4736 Acked Flush 0 Done Sent Done false
15321 4737 Acked Read 4 Done Sent Done false
15322 4738 Acked Read 7 Done Sent Done false
15323 4739 Acked Read 6 Done Sent Done false
15324 4740 Acked Read 1 Done Sent Done false
15325 4741 Acked Write 8 Done Sent Done false
15326 4742 Acked Read 4 Done Sent Done false
15327 4743 Acked Read 6 Done Sent Done false
15328 4744 Acked Write 5 Done Sent Done false
15329 4745 Acked Write 3 Done Sent Done false
15330 4746 Acked Flush 0 Done Sent Done false
15331 4747 Acked Write 8 Done Sent Done false
15332 4748 Acked Write 9 Done Sent Done false
15333 4749 Acked Read 7 Done Sent Done false
15334 4750 Acked Write 9 Done Sent Done false
15335 4751 Acked Read 10 Done Sent Done false
15336 4752 Acked Read 3 Done Sent Done false
15337 4753 Acked Read 9 Done Sent Done false
15338 4754 Acked Write 7 Done Sent Done false
15339 4755 Acked Read 8 Done Sent Done false
15340 4756 Acked Write 6 Done Sent Done false
15341 4757 Acked Read 2 Done Sent Done false
15342 4758 Acked Write 4 Done Sent Done false
15343 4759 Acked Write 2 Done Sent Done false
15344 4760 Acked Flush 0 Done Sent Done false
15345 4761 Acked Read 1 Done Sent Done false
15346 4762 Acked Flush 0 Done Sent Done false
15347 4763 Acked Read 9 Done Sent Done false
15348 4764 Acked Read 9 Done Sent Done false
15349 4765 Acked Read 6 Done Sent Done false
15350 4766 Acked Write 5 Done Sent Done false
15351 4767 Acked Read 9 Done Sent Done false
15352 4768 Acked Flush 0 Done Sent Done false
15353 4769 Acked Read 6 Done Sent Done false
15354 4770 Acked Read 3 Done Sent Done false
15355 4771 Acked Flush 0 Done Sent Done false
15356 4772 Acked Read 10 Done Sent Done false
15357 4773 Acked Write 4 Done Sent Done false
15358 4774 Acked Write 3 Done Sent Done false
15359 4775 Acked Read 3 Done Sent Done false
15360 4776 Acked Write 1 Done Sent Done false
15361 4777 Acked Read 3 Done Sent Done false
15362 4778 Acked Write 5 Done Sent Done false
15363 4779 Acked Flush 0 Done Sent Done false
15364 4780 Acked Read 1 Done Sent Done false
15365 4781 Acked Write 1 Done Sent Done false
15366 4782 Acked Write 10 Done Sent Done false
15367 4783 Acked Write 7 Done Sent Done false
15368 4784 Acked Write 3 Done Sent Done false
15369 4785 Acked Flush 0 Done Sent Done false
15370 4786 Acked Read 9 Done Sent Done false
15371 4787 Acked Read 4 Done Sent Done false
15372 4788 Acked Write 7 Done Sent Done false
15373 4789 Acked Write 7 Done Sent Done false
15374 4790 Acked Read 5 Done Sent Done false
15375 4791 Acked Write 6 Done Sent Done false
15376 4792 Acked Read 2 Done Sent Done false
15377 4793 Acked Write 7 Done Sent Done false
15378 4794 Acked Read 10 Done Sent Done false
15379 4795 Acked Read 3 Done Sent Done false
15380 4796 Acked Write 5 Done Sent Done false
15381 4797 Acked Write 3 Done Sent Done false
15382 4798 Acked Read 1 Done Sent Done false
15383 4799 Acked Write 7 Done Sent Done false
15384 4800 Acked Flush 0 Done Sent Done false
15385 4801 Acked Read 9 Done Sent Done false
15386 4802 Acked Write 9 Done Sent Done false
15387 4803 Acked Write 1 Done Sent Done false
15388 4804 Acked Read 4 Done Sent Done false
15389 4805 Acked Read 2 Done Sent Done false
15390 4806 Acked Read 3 Done Sent Done false
15391 4807 Acked Flush 0 Done Sent Done false
15392 4808 Acked Write 1 Done Sent Done false
15393 4809 Acked Read 3 Done Sent Done false
15394 4810 Acked Read 10 Done Sent Done false
15395 4811 Acked Write 4 Done Sent Done false
15396 4812 Acked Write 6 Done Sent Done false
15397 4813 Acked Read 10 Done Sent Done false
15398 4814 Acked Write 1 Done Sent Done false
15399 4815 Acked Write 10 Done Sent Done false
15400 4816 Acked Read 9 Done Sent Done false
15401 4817 Acked Write 1 Done Sent Done false
15402 4818 Acked Write 4 Done Sent Done false
15403 4819 Acked Write 8 Done Sent Done false
15404 4820 Acked Flush 0 Done Sent Done false
15405 4821 Acked Flush 0 Done Sent Done false
15406 4822 Acked Read 2 Done Sent Done false
15407 4823 Acked Read 7 Done Sent Done false
15408 4824 Acked Flush 0 Done Sent Done false
15409 4825 Acked Read 10 Done Sent Done false
15410 4826 Acked Read 3 Done Sent Done false
15411 4827 Acked Read 7 Done Sent Done false
15412 4828 Acked Flush 0 Done Sent Done false
15413 4829 Acked Flush 0 Done Sent Done false
15414 4830 Acked Read 8 Done Sent Done false
15415 4831 Acked Read 4 Done Sent Done false
15416 4832 Acked Write 2 Done Sent Done false
15417 4833 Acked Write 4 Done Sent Done false
15418 4834 Acked Read 8 Done Sent Done false
15419 4835 Acked Write 7 Done Sent Done false
15420 4836 Acked Read 9 Done Sent Done false
15421 4837 Acked Read 3 Done Sent Done false
15422 4838 Acked Write 7 Done Sent Done false
15423 4839 Acked Flush 0 Done Sent Done false
15424 4840 Acked Write 4 Done Sent Done false
15425 4841 Acked Read 1 Done Sent Done false
15426 4842 Acked Write 5 Done Sent Done false
15427 4843 Acked Write 1 Done Sent Done false
15428 4844 Acked Write 7 Done Sent Done false
15429 4845 Acked Read 5 Done Sent Done false
15430 4846 Acked Read 7 Done Sent Done false
15431 4847 Acked Write 4 Done Sent Done false
15432 4848 Acked Read 4 Done Sent Done false
15433 4849 Acked Read 2 Done Sent Done false
15434 4850 Acked Read 6 Done Sent Done false
15435 4851 Acked Read 6 Done Sent Done false
15436 4852 Acked Read 1 Done Sent Done false
15437 4853 Acked Write 7 Done Sent Done false
15438 4854 Acked Read 8 Done Sent Done false
15439 4855 Acked Flush 0 Done Sent Done false
15440 4856 Acked Read 10 Done Sent Done false
15441 4857 Acked Flush 0 Done Sent Done false
15442 4858 Acked Read 10 Done Sent Done false
15443 4859 Acked Write 8 Done Sent Done false
15444 4860 Acked Flush 0 Done Sent Done false
15445 4861 Acked Write 9 Done Sent Done false
15446 4862 Acked Read 2 Done Sent Done false
15447 4863 Acked Flush 0 Done Sent Done false
15448 4864 Acked Read 3 Done Sent Done false
15449 4865 Acked Read 1 Done Sent Done false
15450 4866 Acked Write 2 Done Sent Done false
15451 4867 Acked Read 10 Done Sent Done false
15452 4868 Acked Read 5 Done Sent Done false
15453 4869 Acked Flush 0 Done Sent Done false
15454 4870 Acked Read 6 Done Sent Done false
15455 4871 Acked Write 5 Done Sent Done false
15456 4872 Acked Write 10 Done Sent Done false
15457 4873 Acked Read 9 Done Sent Done false
15458 4874 Acked Flush 0 Done Sent Done false
15459 4875 Acked Read 6 Done Sent Done false
15460 4876 Acked Read 10 Done Sent Done false
15461 4877 Acked Write 7 Done Sent Done false
15462 4878 Acked Read 8 Done Sent Done false
15463 4879 Acked Read 4 Done Sent Done false
15464 4880 Acked Read 5 Done Sent Done false
15465 4881 Acked Read 5 Done Sent Done false
15466 4882 Acked Write 3 Done Sent Done false
15467 4883 Acked Write 1 Done Sent Done false
15468 4884 Acked Write 9 Done Sent Done false
15469 4885 Acked Flush 0 Done Sent Done false
15470 4886 Acked Write 10 Done Sent Done false
15471 4887 Acked Read 10 Done Sent Done false
15472 4888 Acked Flush 0 Done Sent Done false
15473 4889 Acked Read 8 Done Sent Done false
15474 4890 Acked Write 10 Done Sent Done false
15475 4891 Acked Flush 0 Done Sent Done false
15476 4892 Acked Read 9 Done Sent Done false
15477 4893 Acked Read 1 Done Sent Done false
15478 4894 Acked Write 6 Done Sent Done false
15479 4895 Acked Read 1 Done Sent Done false
15480 4896 Acked Write 4 Done Sent Done false
15481 4897 Acked Write 7 Done Sent Done false
15482 4898 Acked Read 2 Done Sent Done false
15483 4899 Acked Read 9 Done Sent Done false
15484 4900 Acked Read 3 Done Sent Done false
15485 4901 Acked Write 8 Done Sent Done false
15486 4902 Acked Read 9 Done Sent Done false
15487 4903 Acked Write 3 Done Sent Done false
15488 4904 Acked Flush 0 Done Sent Done false
15489 4905 Acked Write 7 Done Sent Done false
15490 4906 Acked Read 4 Done Sent Done false
15491 4907 Acked Read 5 Done Sent Done false
15492 4908 Acked Read 8 Done Sent Done false
15493 4909 Acked Read 3 Done Sent Done false
15494 4910 Acked Read 6 Done Sent Done false
15495 4911 Acked Flush 0 Done Sent Done false
15496 4912 Acked Read 10 Done Sent Done false
15497 4913 Acked Write 1 Done Sent Done false
15498 4914 Acked Flush 0 Done Sent Done false
15499 4915 Acked Write 7 Done Sent Done false
15500 4916 Acked Read 7 Done Sent Done false
15501 4917 Acked Read 1 Done Sent Done false
15502 4918 Acked Write 6 Done Sent Done false
15503 4919 Acked Read 9 Done Sent Done false
15504 4920 Acked Read 5 Done Sent Done false
15505 4921 Acked Read 6 Done Sent Done false
15506 4922 Acked Write 10 Done Sent Done false
15507 4923 Acked Read 3 Done Sent Done false
15508 4924 Acked Read 5 Done Sent Done false
15509 STATES DS:0 DS:1 DS:2 TOTAL
15510 Sent 0 301 0 301
15511 Done 301 0 301 602
15512 Skipped 0 0 0 0
15513 Error 0 0 0 0
15514 Last Flush: Some(JobId(4914)) Some(JobId(4623)) Some(JobId(4914))
15515 Downstairs last five completed: 4623 4622 4621 4620 4619
15516 Upstairs last five completed: 4924 4923 4922 4921 4920
15517 CLIENT: Up:0 ds:301 act:2
155182025-03-17T22:37:56.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155192025-03-17T22:37:57.635ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155202025-03-17T22:37:57.901ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155212025-03-17T22:37:57.901ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155222025-03-17T22:37:57.901ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155232025-03-17T22:37:57.901ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155242025-03-17T22:37:57.902ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155252025-03-17T22:37:58.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
155262025-03-17T22:37:59.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
15527 [8820] got start action
15528 [8820] I am going to restart
15529 Starting downstairs at port 8820
15530 Make output file at "/tmp/dsc/downstairs-8820.txt"
15531 [8820][1] reports Starting
15532 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221)
15533 [8820][1] reports Running
15534 ----------------------------------------------------------------
15535 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15536 States: Active Offline Active
15537 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15538 4624 Acked Write 7 Done Sent Done false
15539 4625 Acked Read 10 Done Sent Done false
15540 4626 Acked Flush 0 Done Sent Done false
15541 4627 Acked Read 10 Done Sent Done false
15542 4628 Acked Read 6 Done Sent Done false
15543 4629 Acked Read 8 Done Sent Done false
15544 4630 Acked Write 2 Done Sent Done false
15545 4631 Acked Read 7 Done Sent Done false
15546 4632 Acked Write 3 Done Sent Done false
15547 4633 Acked Read 3 Done Sent Done false
15548 4634 Acked Read 3 Done Sent Done false
15549 4635 Acked Read 3 Done Sent Done false
15550 4636 Acked Read 1 Done Sent Done false
15551 4637 Acked Flush 0 Done Sent Done false
15552 4638 Acked Read 4 Done Sent Done false
15553 4639 Acked Write 9 Done Sent Done false
15554 4640 Acked Write 4 Done Sent Done false
15555 4641 Acked Read 4 Done Sent Done false
15556 4642 Acked Write 8 Done Sent Done false
15557 4643 Acked Write 9 Done Sent Done false
15558 4644 Acked Read 7 Done Sent Done false
15559 4645 Acked Read 9 Done Sent Done false
15560 4646 Acked Write 10 Done Sent Done false
15561 4647 Acked Write 9 Done Sent Done false
15562 4648 Acked Write 9 Done Sent Done false
15563 4649 Acked Flush 0 Done Sent Done false
15564 4650 Acked Read 8 Done Sent Done false
15565 4651 Acked Write 7 Done Sent Done false
15566 4652 Acked Read 2 Done Sent Done false
15567 4653 Acked Read 5 Done Sent Done false
15568 4654 Acked Write 10 Done Sent Done false
15569 4655 Acked Read 9 Done Sent Done false
15570 4656 Acked Write 10 Done Sent Done false
15571 4657 Acked Read 7 Done Sent Done false
15572 4658 Acked Read 8 Done Sent Done false
15573 4659 Acked Read 2 Done Sent Done false
15574 4660 Acked Flush 0 Done Sent Done false
15575 4661 Acked Write 1 Done Sent Done false
15576 4662 Acked Flush 0 Done Sent Done false
15577 4663 Acked Read 9 Done Sent Done false
15578 4664 Acked Write 3 Done Sent Done false
15579 4665 Acked Write 6 Done Sent Done false
15580 4666 Acked Write 3 Done Sent Done false
15581 4667 Acked Write 7 Done Sent Done false
15582 4668 Acked Flush 0 Done Sent Done false
15583 4669 Acked Write 1 Done Sent Done false
15584 4670 Acked Write 2 Done Sent Done false
15585 4671 Acked Read 6 Done Sent Done false
15586 4672 Acked Read 6 Done Sent Done false
15587 4673 Acked Flush 0 Done Sent Done false
15588 4674 Acked Read 9 Done Sent Done false
15589 4675 Acked Read 10 Done Sent Done false
15590 4676 Acked Write 7 Done Sent Done false
15591 4677 Acked Write 7 Done Sent Done false
15592 4678 Acked Write 1 Done Sent Done false
15593 4679 Acked Read 8 Done Sent Done false
15594 4680 Acked Read 3 Done Sent Done false
15595 4681 Acked Write 5 Done Sent Done false
15596 4682 Acked Write 6 Done Sent Done false
15597 4683 Acked Write 3 Done Sent Done false
15598 4684 Acked Read 8 Done Sent Done false
15599 4685 Acked Flush 0 Done Sent Done false
15600 4686 Acked Read 2 Done Sent Done false
15601 4687 Acked Read 10 Done Sent Done false
15602 4688 Acked Read 10 Done Sent Done false
15603 4689 Acked Write 6 Done Sent Done false
15604 4690 Acked Read 2 Done Sent Done false
15605 4691 Acked Write 2 Done Sent Done false
15606 4692 Acked Read 7 Done Sent Done false
15607 4693 Acked Read 5 Done Sent Done false
15608 4694 Acked Read 7 Done Sent Done false
15609 4695 Acked Write 9 Done Sent Done false
15610 4696 Acked Read 7 Done Sent Done false
15611 4697 Acked Read 7 Done Sent Done false
15612 4698 Acked Write 10 Done Sent Done false
15613 4699 Acked Read 10 Done Sent Done false
15614 4700 Acked Read 5 Done Sent Done false
15615 4701 Acked Read 1 Done Sent Done false
15616 4702 Acked Read 10 Done Sent Done false
15617 4703 Acked Read 7 Done Sent Done false
15618 4704 Acked Write 5 Done Sent Done false
15619 4705 Acked Read 8 Done Sent Done false
15620 4706 Acked Read 9 Done Sent Done false
15621 4707 Acked Read 6 Done Sent Done false
15622 4708 Acked Write 1 Done Sent Done false
15623 4709 Acked Write 3 Done Sent Done false
15624 4710 Acked Read 8 Done Sent Done false
15625 4711 Acked Flush 0 Done Sent Done false
15626 4712 Acked Read 10 Done Sent Done false
15627 4713 Acked Read 1 Done Sent Done false
15628 4714 Acked Flush 0 Done Sent Done false
15629 4715 Acked Read 10 Done Sent Done false
15630 4716 Acked Read 5 Done Sent Done false
15631 4717 Acked Read 7 Done Sent Done false
15632 4718 Acked Read 6 Done Sent Done false
15633 4719 Acked Write 5 Done Sent Done false
15634 4720 Acked Write 4 Done Sent Done false
15635 4721 Acked Read 9 Done Sent Done false
15636 4722 Acked Write 2 Done Sent Done false
15637 4723 Acked Write 8 Done Sent Done false
15638 4724 Acked Write 10 Done Sent Done false
15639 4725 Acked Write 5 Done Sent Done false
15640 4726 Acked Write 6 Done Sent Done false
15641 4727 Acked Read 3 Done Sent Done false
15642 4728 Acked Write 4 Done Sent Done false
15643 4729 Acked Read 8 Done Sent Done false
15644 4730 Acked Write 9 Done Sent Done false
15645 4731 Acked Write 1 Done Sent Done false
15646 4732 Acked Read 4 Done Sent Done false
15647 4733 Acked Read 6 Done Sent Done false
15648 4734 Acked Read 4 Done Sent Done false
15649 4735 Acked Read 4 Done Sent Done false
15650 4736 Acked Flush 0 Done Sent Done false
15651 4737 Acked Read 4 Done Sent Done false
15652 4738 Acked Read 7 Done Sent Done false
15653 4739 Acked Read 6 Done Sent Done false
15654 4740 Acked Read 1 Done Sent Done false
15655 4741 Acked Write 8 Done Sent Done false
15656 4742 Acked Read 4 Done Sent Done false
15657 4743 Acked Read 6 Done Sent Done false
15658 4744 Acked Write 5 Done Sent Done false
15659 4745 Acked Write 3 Done Sent Done false
15660 4746 Acked Flush 0 Done Sent Done false
15661 4747 Acked Write 8 Done Sent Done false
15662 4748 Acked Write 9 Done Sent Done false
15663 4749 Acked Read 7 Done Sent Done false
15664 4750 Acked Write 9 Done Sent Done false
15665 4751 Acked Read 10 Done Sent Done false
15666 4752 Acked Read 3 Done Sent Done false
15667 4753 Acked Read 9 Done Sent Done false
15668 4754 Acked Write 7 Done Sent Done false
15669 4755 Acked Read 8 Done Sent Done false
15670 4756 Acked Write 6 Done Sent Done false
15671 4757 Acked Read 2 Done Sent Done false
15672 4758 Acked Write 4 Done Sent Done false
15673 4759 Acked Write 2 Done Sent Done false
15674 4760 Acked Flush 0 Done Sent Done false
15675 4761 Acked Read 1 Done Sent Done false
15676 4762 Acked Flush 0 Done Sent Done false
15677 4763 Acked Read 9 Done Sent Done false
15678 4764 Acked Read 9 Done Sent Done false
15679 4765 Acked Read 6 Done Sent Done false
15680 4766 Acked Write 5 Done Sent Done false
15681 4767 Acked Read 9 Done Sent Done false
15682 4768 Acked Flush 0 Done Sent Done false
15683 4769 Acked Read 6 Done Sent Done false
15684 4770 Acked Read 3 Done Sent Done false
15685 4771 Acked Flush 0 Done Sent Done false
15686 4772 Acked Read 10 Done Sent Done false
15687 4773 Acked Write 4 Done Sent Done false
15688 4774 Acked Write 3 Done Sent Done false
15689 4775 Acked Read 3 Done Sent Done false
15690 4776 Acked Write 1 Done Sent Done false
15691 4777 Acked Read 3 Done Sent Done false
15692 4778 Acked Write 5 Done Sent Done false
15693 4779 Acked Flush 0 Done Sent Done false
15694 4780 Acked Read 1 Done Sent Done false
15695 4781 Acked Write 1 Done Sent Done false
15696 4782 Acked Write 10 Done Sent Done false
15697 4783 Acked Write 7 Done Sent Done false
15698 4784 Acked Write 3 Done Sent Done false
15699 4785 Acked Flush 0 Done Sent Done false
15700 4786 Acked Read 9 Done Sent Done false
15701 4787 Acked Read 4 Done Sent Done false
15702 4788 Acked Write 7 Done Sent Done false
15703 4789 Acked Write 7 Done Sent Done false
15704 4790 Acked Read 5 Done Sent Done false
15705 4791 Acked Write 6 Done Sent Done false
15706 4792 Acked Read 2 Done Sent Done false
15707 4793 Acked Write 7 Done Sent Done false
15708 4794 Acked Read 10 Done Sent Done false
15709 4795 Acked Read 3 Done Sent Done false
15710 4796 Acked Write 5 Done Sent Done false
15711 4797 Acked Write 3 Done Sent Done false
15712 4798 Acked Read 1 Done Sent Done false
15713 4799 Acked Write 7 Done Sent Done false
15714 4800 Acked Flush 0 Done Sent Done false
15715 4801 Acked Read 9 Done Sent Done false
15716 4802 Acked Write 9 Done Sent Done false
15717 4803 Acked Write 1 Done Sent Done false
15718 4804 Acked Read 4 Done Sent Done false
15719 4805 Acked Read 2 Done Sent Done false
15720 4806 Acked Read 3 Done Sent Done false
15721 4807 Acked Flush 0 Done Sent Done false
15722 4808 Acked Write 1 Done Sent Done false
15723 4809 Acked Read 3 Done Sent Done false
15724 4810 Acked Read 10 Done Sent Done false
15725 4811 Acked Write 4 Done Sent Done false
15726 4812 Acked Write 6 Done Sent Done false
15727 4813 Acked Read 10 Done Sent Done false
15728 4814 Acked Write 1 Done Sent Done false
15729 4815 Acked Write 10 Done Sent Done false
15730 4816 Acked Read 9 Done Sent Done false
15731 4817 Acked Write 1 Done Sent Done false
15732 4818 Acked Write 4 Done Sent Done false
15733 4819 Acked Write 8 Done Sent Done false
15734 4820 Acked Flush 0 Done Sent Done false
15735 4821 Acked Flush 0 Done Sent Done false
15736 4822 Acked Read 2 Done Sent Done false
15737 4823 Acked Read 7 Done Sent Done false
15738 4824 Acked Flush 0 Done Sent Done false
15739 4825 Acked Read 10 Done Sent Done false
15740 4826 Acked Read 3 Done Sent Done false
15741 4827 Acked Read 7 Done Sent Done false
15742 4828 Acked Flush 0 Done Sent Done false
15743 4829 Acked Flush 0 Done Sent Done false
15744 4830 Acked Read 8 Done Sent Done false
15745 4831 Acked Read 4 Done Sent Done false
15746 4832 Acked Write 2 Done Sent Done false
15747 4833 Acked Write 4 Done Sent Done false
15748 4834 Acked Read 8 Done Sent Done false
15749 4835 Acked Write 7 Done Sent Done false
15750 4836 Acked Read 9 Done Sent Done false
15751 4837 Acked Read 3 Done Sent Done false
15752 4838 Acked Write 7 Done Sent Done false
15753 4839 Acked Flush 0 Done Sent Done false
15754 4840 Acked Write 4 Done Sent Done false
15755 4841 Acked Read 1 Done Sent Done false
15756 4842 Acked Write 5 Done Sent Done false
15757 4843 Acked Write 1 Done Sent Done false
15758 4844 Acked Write 7 Done Sent Done false
15759 4845 Acked Read 5 Done Sent Done false
15760 4846 Acked Read 7 Done Sent Done false
15761 4847 Acked Write 4 Done Sent Done false
15762 4848 Acked Read 4 Done Sent Done false
15763 4849 Acked Read 2 Done Sent Done false
15764 4850 Acked Read 6 Done Sent Done false
15765 4851 Acked Read 6 Done Sent Done false
15766 4852 Acked Read 1 Done Sent Done false
15767 4853 Acked Write 7 Done Sent Done false
15768 4854 Acked Read 8 Done Sent Done false
15769 4855 Acked Flush 0 Done Sent Done false
15770 4856 Acked Read 10 Done Sent Done false
15771 4857 Acked Flush 0 Done Sent Done false
15772 4858 Acked Read 10 Done Sent Done false
15773 4859 Acked Write 8 Done Sent Done false
15774 4860 Acked Flush 0 Done Sent Done false
15775 4861 Acked Write 9 Done Sent Done false
15776 4862 Acked Read 2 Done Sent Done false
15777 4863 Acked Flush 0 Done Sent Done false
15778 4864 Acked Read 3 Done Sent Done false
15779 4865 Acked Read 1 Done Sent Done false
15780 4866 Acked Write 2 Done Sent Done false
15781 4867 Acked Read 10 Done Sent Done false
15782 4868 Acked Read 5 Done Sent Done false
15783 4869 Acked Flush 0 Done Sent Done false
15784 4870 Acked Read 6 Done Sent Done false
15785 4871 Acked Write 5 Done Sent Done false
15786 4872 Acked Write 10 Done Sent Done false
15787 4873 Acked Read 9 Done Sent Done false
15788 4874 Acked Flush 0 Done Sent Done false
15789 4875 Acked Read 6 Done Sent Done false
15790 4876 Acked Read 10 Done Sent Done false
15791 4877 Acked Write 7 Done Sent Done false
15792 4878 Acked Read 8 Done Sent Done false
15793 4879 Acked Read 4 Done Sent Done false
15794 4880 Acked Read 5 Done Sent Done false
15795 4881 Acked Read 5 Done Sent Done false
15796 4882 Acked Write 3 Done Sent Done false
15797 4883 Acked Write 1 Done Sent Done false
15798 4884 Acked Write 9 Done Sent Done false
15799 4885 Acked Flush 0 Done Sent Done false
15800 4886 Acked Write 10 Done Sent Done false
15801 4887 Acked Read 10 Done Sent Done false
15802 4888 Acked Flush 0 Done Sent Done false
15803 4889 Acked Read 8 Done Sent Done false
15804 4890 Acked Write 10 Done Sent Done false
15805 4891 Acked Flush 0 Done Sent Done false
15806 4892 Acked Read 9 Done Sent Done false
15807 4893 Acked Read 1 Done Sent Done false
15808 4894 Acked Write 6 Done Sent Done false
15809 4895 Acked Read 1 Done Sent Done false
15810 4896 Acked Write 4 Done Sent Done false
15811 4897 Acked Write 7 Done Sent Done false
15812 4898 Acked Read 2 Done Sent Done false
15813 4899 Acked Read 9 Done Sent Done false
15814 4900 Acked Read 3 Done Sent Done false
15815 4901 Acked Write 8 Done Sent Done false
15816 4902 Acked Read 9 Done Sent Done false
15817 4903 Acked Write 3 Done Sent Done false
15818 4904 Acked Flush 0 Done Sent Done false
15819 4905 Acked Write 7 Done Sent Done false
15820 4906 Acked Read 4 Done Sent Done false
15821 4907 Acked Read 5 Done Sent Done false
15822 4908 Acked Read 8 Done Sent Done false
15823 4909 Acked Read 3 Done Sent Done false
15824 4910 Acked Read 6 Done Sent Done false
15825 4911 Acked Flush 0 Done Sent Done false
15826 4912 Acked Read 10 Done Sent Done false
15827 4913 Acked Write 1 Done Sent Done false
15828 4914 Acked Flush 0 Done Sent Done false
15829 4915 Acked Write 7 Done Sent Done false
15830 4916 Acked Read 7 Done Sent Done false
15831 4917 Acked Read 1 Done Sent Done false
15832 4918 Acked Write 6 Done Sent Done false
15833 4919 Acked Read 9 Done Sent Done false
15834 4920 Acked Read 5 Done Sent Done false
15835 4921 Acked Read 6 Done Sent Done false
15836 4922 Acked Write 10 Done Sent Done false
15837 4923 Acked Read 3 Done Sent Done false
15838 4924 Acked Read 5 Done Sent Done false
15839 STATES DS:0 DS:1 DS:2 TOTAL
15840 Sent 0 301 0 301
15841 Done 301 0 301 602
15842 Skipped 0 0 0 0
15843 Error 0 0 0 0
15844 Last Flush: Some(JobId(4914)) Some(JobId(4623)) Some(JobId(4914))
15845 Downstairs last five completed: 4623 4622 4621 4620 4619
15846 Upstairs last five completed: 4924 4923 4922 4921 4920
15847 CLIENT: Up:0 ds:301 act:2
158482025-03-17T22:38:01.641ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
158492025-03-17T22:38:02.643ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
158502025-03-17T22:38:03.892ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
15851 ----------------------------------------------------------------
15852 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15853 States: Active Offline Active
15854 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15855 4624 Acked Write 7 Done Sent Done false
15856 4625 Acked Read 10 Done Sent Done false
15857 4626 Acked Flush 0 Done Sent Done false
15858 4627 Acked Read 10 Done Sent Done false
15859 4628 Acked Read 6 Done Sent Done false
15860 4629 Acked Read 8 Done Sent Done false
15861 4630 Acked Write 2 Done Sent Done false
15862 4631 Acked Read 7 Done Sent Done false
15863 4632 Acked Write 3 Done Sent Done false
15864 4633 Acked Read 3 Done Sent Done false
15865 4634 Acked Read 3 Done Sent Done false
15866 4635 Acked Read 3 Done Sent Done false
15867 4636 Acked Read 1 Done Sent Done false
15868 4637 Acked Flush 0 Done Sent Done false
15869 4638 Acked Read 4 Done Sent Done false
15870 4639 Acked Write 9 Done Sent Done false
15871 4640 Acked Write 4 Done Sent Done false
15872 4641 Acked Read 4 Done Sent Done false
15873 4642 Acked Write 8 Done Sent Done false
15874 4643 Acked Write 9 Done Sent Done false
15875 4644 Acked Read 7 Done Sent Done false
15876 4645 Acked Read 9 Done Sent Done false
15877 4646 Acked Write 10 Done Sent Done false
15878 4647 Acked Write 9 Done Sent Done false
15879 4648 Acked Write 9 Done Sent Done false
15880 4649 Acked Flush 0 Done Sent Done false
15881 4650 Acked Read 8 Done Sent Done false
15882 4651 Acked Write 7 Done Sent Done false
15883 4652 Acked Read 2 Done Sent Done false
15884 4653 Acked Read 5 Done Sent Done false
15885 4654 Acked Write 10 Done Sent Done false
15886 4655 Acked Read 9 Done Sent Done false
15887 4656 Acked Write 10 Done Sent Done false
15888 4657 Acked Read 7 Done Sent Done false
15889 4658 Acked Read 8 Done Sent Done false
15890 4659 Acked Read 2 Done Sent Done false
15891 4660 Acked Flush 0 Done Sent Done false
15892 4661 Acked Write 1 Done Sent Done false
15893 4662 Acked Flush 0 Done Sent Done false
15894 4663 Acked Read 9 Done Sent Done false
15895 4664 Acked Write 3 Done Sent Done false
15896 4665 Acked Write 6 Done Sent Done false
15897 4666 Acked Write 3 Done Sent Done false
15898 4667 Acked Write 7 Done Sent Done false
15899 4668 Acked Flush 0 Done Sent Done false
15900 4669 Acked Write 1 Done Sent Done false
15901 4670 Acked Write 2 Done Sent Done false
15902 4671 Acked Read 6 Done Sent Done false
15903 4672 Acked Read 6 Done Sent Done false
15904 4673 Acked Flush 0 Done Sent Done false
15905 4674 Acked Read 9 Done Sent Done false
15906 4675 Acked Read 10 Done Sent Done false
15907 4676 Acked Write 7 Done Sent Done false
15908 4677 Acked Write 7 Done Sent Done false
15909 4678 Acked Write 1 Done Sent Done false
15910 4679 Acked Read 8 Done Sent Done false
15911 4680 Acked Read 3 Done Sent Done false
15912 4681 Acked Write 5 Done Sent Done false
15913 4682 Acked Write 6 Done Sent Done false
15914 4683 Acked Write 3 Done Sent Done false
15915 4684 Acked Read 8 Done Sent Done false
15916 4685 Acked Flush 0 Done Sent Done false
15917 4686 Acked Read 2 Done Sent Done false
15918 4687 Acked Read 10 Done Sent Done false
15919 4688 Acked Read 10 Done Sent Done false
15920 4689 Acked Write 6 Done Sent Done false
15921 4690 Acked Read 2 Done Sent Done false
15922 4691 Acked Write 2 Done Sent Done false
15923 4692 Acked Read 7 Done Sent Done false
15924 4693 Acked Read 5 Done Sent Done false
15925 4694 Acked Read 7 Done Sent Done false
15926 4695 Acked Write 9 Done Sent Done false
15927 4696 Acked Read 7 Done Sent Done false
15928 4697 Acked Read 7 Done Sent Done false
15929 4698 Acked Write 10 Done Sent Done false
15930 4699 Acked Read 10 Done Sent Done false
15931 4700 Acked Read 5 Done Sent Done false
15932 4701 Acked Read 1 Done Sent Done false
15933 4702 Acked Read 10 Done Sent Done false
15934 4703 Acked Read 7 Done Sent Done false
15935 4704 Acked Write 5 Done Sent Done false
15936 4705 Acked Read 8 Done Sent Done false
15937 4706 Acked Read 9 Done Sent Done false
15938 4707 Acked Read 6 Done Sent Done false
15939 4708 Acked Write 1 Done Sent Done false
15940 4709 Acked Write 3 Done Sent Done false
15941 4710 Acked Read 8 Done Sent Done false
15942 4711 Acked Flush 0 Done Sent Done false
15943 4712 Acked Read 10 Done Sent Done false
15944 4713 Acked Read 1 Done Sent Done false
15945 4714 Acked Flush 0 Done Sent Done false
15946 4715 Acked Read 10 Done Sent Done false
15947 4716 Acked Read 5 Done Sent Done false
15948 4717 Acked Read 7 Done Sent Done false
15949 4718 Acked Read 6 Done Sent Done false
15950 4719 Acked Write 5 Done Sent Done false
15951 4720 Acked Write 4 Done Sent Done false
15952 4721 Acked Read 9 Done Sent Done false
15953 4722 Acked Write 2 Done Sent Done false
15954 4723 Acked Write 8 Done Sent Done false
15955 4724 Acked Write 10 Done Sent Done false
15956 4725 Acked Write 5 Done Sent Done false
15957 4726 Acked Write 6 Done Sent Done false
15958 4727 Acked Read 3 Done Sent Done false
15959 4728 Acked Write 4 Done Sent Done false
15960 4729 Acked Read 8 Done Sent Done false
15961 4730 Acked Write 9 Done Sent Done false
15962 4731 Acked Write 1 Done Sent Done false
15963 4732 Acked Read 4 Done Sent Done false
15964 4733 Acked Read 6 Done Sent Done false
15965 4734 Acked Read 4 Done Sent Done false
15966 4735 Acked Read 4 Done Sent Done false
15967 4736 Acked Flush 0 Done Sent Done false
15968 4737 Acked Read 4 Done Sent Done false
15969 4738 Acked Read 7 Done Sent Done false
15970 4739 Acked Read 6 Done Sent Done false
15971 4740 Acked Read 1 Done Sent Done false
15972 4741 Acked Write 8 Done Sent Done false
15973 4742 Acked Read 4 Done Sent Done false
15974 4743 Acked Read 6 Done Sent Done false
15975 4744 Acked Write 5 Done Sent Done false
15976 4745 Acked Write 3 Done Sent Done false
15977 4746 Acked Flush 0 Done Sent Done false
15978 4747 Acked Write 8 Done Sent Done false
15979 4748 Acked Write 9 Done Sent Done false
15980 4749 Acked Read 7 Done Sent Done false
15981 4750 Acked Write 9 Done Sent Done false
15982 4751 Acked Read 10 Done Sent Done false
15983 4752 Acked Read 3 Done Sent Done false
15984 4753 Acked Read 9 Done Sent Done false
15985 4754 Acked Write 7 Done Sent Done false
15986 4755 Acked Read 8 Done Sent Done false
15987 4756 Acked Write 6 Done Sent Done false
15988 4757 Acked Read 2 Done Sent Done false
15989 4758 Acked Write 4 Done Sent Done false
15990 4759 Acked Write 2 Done Sent Done false
15991 4760 Acked Flush 0 Done Sent Done false
15992 4761 Acked Read 1 Done Sent Done false
15993 4762 Acked Flush 0 Done Sent Done false
15994 4763 Acked Read 9 Done Sent Done false
15995 4764 Acked Read 9 Done Sent Done false
15996 4765 Acked Read 6 Done Sent Done false
15997 4766 Acked Write 5 Done Sent Done false
15998 4767 Acked Read 9 Done Sent Done false
15999 4768 Acked Flush 0 Done Sent Done false
16000 4769 Acked Read 6 Done Sent Done false
16001 4770 Acked Read 3 Done Sent Done false
16002 4771 Acked Flush 0 Done Sent Done false
16003 4772 Acked Read 10 Done Sent Done false
16004 4773 Acked Write 4 Done Sent Done false
16005 4774 Acked Write 3 Done Sent Done false
16006 4775 Acked Read 3 Done Sent Done false
16007 4776 Acked Write 1 Done Sent Done false
16008 4777 Acked Read 3 Done Sent Done false
16009 4778 Acked Write 5 Done Sent Done false
16010 4779 Acked Flush 0 Done Sent Done false
16011 4780 Acked Read 1 Done Sent Done false
16012 4781 Acked Write 1 Done Sent Done false
16013 4782 Acked Write 10 Done Sent Done false
16014 4783 Acked Write 7 Done Sent Done false
16015 4784 Acked Write 3 Done Sent Done false
16016 4785 Acked Flush 0 Done Sent Done false
16017 4786 Acked Read 9 Done Sent Done false
16018 4787 Acked Read 4 Done Sent Done false
16019 4788 Acked Write 7 Done Sent Done false
16020 4789 Acked Write 7 Done Sent Done false
16021 4790 Acked Read 5 Done Sent Done false
16022 4791 Acked Write 6 Done Sent Done false
16023 4792 Acked Read 2 Done Sent Done false
16024 4793 Acked Write 7 Done Sent Done false
16025 4794 Acked Read 10 Done Sent Done false
16026 4795 Acked Read 3 Done Sent Done false
16027 4796 Acked Write 5 Done Sent Done false
16028 4797 Acked Write 3 Done Sent Done false
16029 4798 Acked Read 1 Done Sent Done false
16030 4799 Acked Write 7 Done Sent Done false
16031 4800 Acked Flush 0 Done Sent Done false
16032 4801 Acked Read 9 Done Sent Done false
16033 4802 Acked Write 9 Done Sent Done false
16034 4803 Acked Write 1 Done Sent Done false
16035 4804 Acked Read 4 Done Sent Done false
16036 4805 Acked Read 2 Done Sent Done false
16037 4806 Acked Read 3 Done Sent Done false
16038 4807 Acked Flush 0 Done Sent Done false
16039 4808 Acked Write 1 Done Sent Done false
16040 4809 Acked Read 3 Done Sent Done false
16041 4810 Acked Read 10 Done Sent Done false
16042 4811 Acked Write 4 Done Sent Done false
16043 4812 Acked Write 6 Done Sent Done false
16044 4813 Acked Read 10 Done Sent Done false
16045 4814 Acked Write 1 Done Sent Done false
16046 4815 Acked Write 10 Done Sent Done false
16047 4816 Acked Read 9 Done Sent Done false
16048 4817 Acked Write 1 Done Sent Done false
16049 4818 Acked Write 4 Done Sent Done false
16050 4819 Acked Write 8 Done Sent Done false
16051 4820 Acked Flush 0 Done Sent Done false
16052 4821 Acked Flush 0 Done Sent Done false
16053 4822 Acked Read 2 Done Sent Done false
16054 4823 Acked Read 7 Done Sent Done false
16055 4824 Acked Flush 0 Done Sent Done false
16056 4825 Acked Read 10 Done Sent Done false
16057 4826 Acked Read 3 Done Sent Done false
16058 4827 Acked Read 7 Done Sent Done false
16059 4828 Acked Flush 0 Done Sent Done false
16060 4829 Acked Flush 0 Done Sent Done false
16061 4830 Acked Read 8 Done Sent Done false
16062 4831 Acked Read 4 Done Sent Done false
16063 4832 Acked Write 2 Done Sent Done false
16064 4833 Acked Write 4 Done Sent Done false
16065 4834 Acked Read 8 Done Sent Done false
16066 4835 Acked Write 7 Done Sent Done false
16067 4836 Acked Read 9 Done Sent Done false
16068 4837 Acked Read 3 Done Sent Done false
16069 4838 Acked Write 7 Done Sent Done false
16070 4839 Acked Flush 0 Done Sent Done false
16071 4840 Acked Write 4 Done Sent Done false
16072 4841 Acked Read 1 Done Sent Done false
16073 4842 Acked Write 5 Done Sent Done false
16074 4843 Acked Write 1 Done Sent Done false
16075 4844 Acked Write 7 Done Sent Done false
16076 4845 Acked Read 5 Done Sent Done false
16077 4846 Acked Read 7 Done Sent Done false
16078 4847 Acked Write 4 Done Sent Done false
16079 4848 Acked Read 4 Done Sent Done false
16080 4849 Acked Read 2 Done Sent Done false
16081 4850 Acked Read 6 Done Sent Done false
16082 4851 Acked Read 6 Done Sent Done false
16083 4852 Acked Read 1 Done Sent Done false
16084 4853 Acked Write 7 Done Sent Done false
16085 4854 Acked Read 8 Done Sent Done false
16086 4855 Acked Flush 0 Done Sent Done false
16087 4856 Acked Read 10 Done Sent Done false
16088 4857 Acked Flush 0 Done Sent Done false
16089 4858 Acked Read 10 Done Sent Done false
16090 4859 Acked Write 8 Done Sent Done false
16091 4860 Acked Flush 0 Done Sent Done false
16092 4861 Acked Write 9 Done Sent Done false
16093 4862 Acked Read 2 Done Sent Done false
16094 4863 Acked Flush 0 Done Sent Done false
16095 4864 Acked Read 3 Done Sent Done false
16096 4865 Acked Read 1 Done Sent Done false
16097 4866 Acked Write 2 Done Sent Done false
16098 4867 Acked Read 10 Done Sent Done false
16099 4868 Acked Read 5 Done Sent Done false
16100 4869 Acked Flush 0 Done Sent Done false
16101 4870 Acked Read 6 Done Sent Done false
16102 4871 Acked Write 5 Done Sent Done false
16103 4872 Acked Write 10 Done Sent Done false
16104 4873 Acked Read 9 Done Sent Done false
16105 4874 Acked Flush 0 Done Sent Done false
16106 4875 Acked Read 6 Done Sent Done false
16107 4876 Acked Read 10 Done Sent Done false
16108 4877 Acked Write 7 Done Sent Done false
16109 4878 Acked Read 8 Done Sent Done false
16110 4879 Acked Read 4 Done Sent Done false
16111 4880 Acked Read 5 Done Sent Done false
16112 4881 Acked Read 5 Done Sent Done false
16113 4882 Acked Write 3 Done Sent Done false
16114 4883 Acked Write 1 Done Sent Done false
16115 4884 Acked Write 9 Done Sent Done false
16116 4885 Acked Flush 0 Done Sent Done false
16117 4886 Acked Write 10 Done Sent Done false
16118 4887 Acked Read 10 Done Sent Done false
16119 4888 Acked Flush 0 Done Sent Done false
16120 4889 Acked Read 8 Done Sent Done false
16121 4890 Acked Write 10 Done Sent Done false
16122 4891 Acked Flush 0 Done Sent Done false
16123 4892 Acked Read 9 Done Sent Done false
16124 4893 Acked Read 1 Done Sent Done false
16125 4894 Acked Write 6 Done Sent Done false
16126 4895 Acked Read 1 Done Sent Done false
16127 4896 Acked Write 4 Done Sent Done false
16128 4897 Acked Write 7 Done Sent Done false
16129 4898 Acked Read 2 Done Sent Done false
16130 4899 Acked Read 9 Done Sent Done false
16131 4900 Acked Read 3 Done Sent Done false
16132 4901 Acked Write 8 Done Sent Done false
16133 4902 Acked Read 9 Done Sent Done false
16134 4903 Acked Write 3 Done Sent Done false
16135 4904 Acked Flush 0 Done Sent Done false
16136 4905 Acked Write 7 Done Sent Done false
16137 4906 Acked Read 4 Done Sent Done false
16138 4907 Acked Read 5 Done Sent Done false
16139 4908 Acked Read 8 Done Sent Done false
16140 4909 Acked Read 3 Done Sent Done false
16141 4910 Acked Read 6 Done Sent Done false
16142 4911 Acked Flush 0 Done Sent Done false
16143 4912 Acked Read 10 Done Sent Done false
16144 4913 Acked Write 1 Done Sent Done false
16145 4914 Acked Flush 0 Done Sent Done false
16146 4915 Acked Write 7 Done Sent Done false
16147 4916 Acked Read 7 Done Sent Done false
16148 4917 Acked Read 1 Done Sent Done false
16149 4918 Acked Write 6 Done Sent Done false
16150 4919 Acked Read 9 Done Sent Done false
16151 4920 Acked Read 5 Done Sent Done false
16152 4921 Acked Read 6 Done Sent Done false
16153 4922 Acked Write 10 Done Sent Done false
16154 4923 Acked Read 3 Done Sent Done false
16155 4924 Acked Read 5 Done Sent Done false
16156 STATES DS:0 DS:1 DS:2 TOTAL
16157 Sent 0 301 0 301
16158 Done 301 0 301 602
16159 Skipped 0 0 0 0
16160 Error 0 0 0 0
16161 Last Flush: Some(JobId(4914)) Some(JobId(4623)) Some(JobId(4914))
16162 Downstairs last five completed: 4623 4622 4621 4620 4619
16163 Upstairs last five completed: 4924 4923 4922 4921 4920
16164 CLIENT: Up:0 ds:301 act:2
161652025-03-17T22:38:04.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161662025-03-17T22:38:05.646ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161672025-03-17T22:38:06.648ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161682025-03-17T22:38:07.649ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161692025-03-17T22:38:07.903ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161702025-03-17T22:38:07.903ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58786) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161712025-03-17T22:38:07.904ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161722025-03-17T22:38:07.905ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161732025-03-17T22:38:07.905ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161742025-03-17T22:38:07.905ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161752025-03-17T22:38:07.905ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161762025-03-17T22:38:07.905ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4623)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
161772025-03-17T22:38:07.905ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4623)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
16178 ----------------------------------------------------------------
16179 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
16180 States: Active Active Active
16181 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16182 4915 Acked Write 7 Done Done Done true
16183 4916 Acked Read 7 Done Done Done true
16184 4917 Acked Read 1 Done Done Done true
16185 4918 Acked Write 6 Done Done Done true
16186 4919 Acked Read 9 Done Done Done true
16187 4920 Acked Read 5 Done Done Done true
16188 4921 Acked Read 6 Done Done Done true
16189 4922 Acked Write 10 Done Done Done true
16190 4923 Acked Read 3 Done Done Done true
16191 4924 Acked Read 5 Done Done Done true
16192 STATES DS:0 DS:1 DS:2 TOTAL
16193 Sent 0 0 0 0
16194 Done 10 10 10 30
16195 Skipped 0 0 0 0
16196 Error 0 0 0 0
16197 Last Flush: Some(JobId(4914)) Some(JobId(4914)) Some(JobId(4914))
16198 Downstairs last five completed: 4914 4913 4912 4911 4910
16199 Upstairs last five completed: 4924 4923 4922 4921 4920
16200 CLIENT: Up:0 ds:10 act:3
16201 ----------------------------------------------------------------
16202 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16203 Downstairs last five completed: 4925 4924 4923 4922 4921
16204 Upstairs last five completed: 4925 4924 4923 4922 4921
16205 CLIENT: Up:0 ds:0 act:3
16206 Replay: All jobs finished
16207 Main task has work to do, go find it
16208 got dsc Stop(1)
16209 stop 1
16210 Mar 17 22:38:12.167 INFO request completed, latency_us: 359, response_code:[8820] Got stop action so:false kr:false
16211 204, uri: /stop/cid/1, method: POST, req_id: 72bd3cf0-f307-4810-aa17-ce1e7751b77f, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
162122025-03-17T22:38:12.168ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
162132025-03-17T22:38:12.168ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
162142025-03-17T22:38:12.168ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
16215 Mar 17 22:38:12.168 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/1, method: GET, req_id: 968a4e50-5500-44f9-917f-a4a021e31803, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
162162025-03-17T22:38:12.168ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
16217 [8820] Exited with: ExitStatus(unix_wait_status(9))
16218 [8820][1] reports Exit
16219 Mar 17 22:38:16.170 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4d144999-3f4b-4161-b058-61aa8f235b02, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
16220 001/300 Read block 4242 len 28672
16221 002/300 Write block 3541 len 24576 data: 2 2 3 4 5 6
16222 003/300 Write block 1759 len 4096 data: 2
16223 004/300 Write block 4723 len 24576 data: 2 3 4 3 3 4
16224 005/300 Write block 3455 len 8192 data: 3 2
16225 006/300 Read block 772 len 28672
16226 007/300 Read block 1372 len 36864
16227 008/300 Write block 716 len 28672 data: 4 3 3 2 2 4 4
16228 009/300 Read block 3474 len 16384
16229 010/300 Read block 2049 len 20480
16230 011/300 Read block 3775 len 20480
16231 012/300 Write block 1851 len 20480 data: 3 3 3 3 3
16232 013/300 Write block 371 len 12288 data: 3 2 2
16233 014/300 Read block 887 len 28672
16234 015/300 Read block 960 len 40960
16235 016/300 Read block 2133 len 4096
16236 017/300 Write block 2738 len 24576 data: 3 5 5 6 6 4
16237 018/300 Read block 4463 len 12288
16238 019/300 Read block 4881 len 40960
16239 020/300 Read block 1538 len 8192
16240 021/300 Read block 2324 len 24576
16241 022/300 Flush
16242 023/300 Write block 2201 len 36864 data: 4 5 5 5 7 5 4 4 3
16243 024/300 Flush
16244 025/300 Read block 764 len 28672
16245 026/300 Read block 4401 len 8192
16246 027/300 Write block 1665 len 20480 data: 3 3 3 2 2
16247 028/300 Read block 238 len 8192
16248 029/300 Read block 1681 len 40960
16249 030/300 Write block 1391 len 24576 data: 3 4 4 5 5 6
16250 031/300 Read block 4898 len 28672
16251 032/300 Write block 494 len 8192 data: 3 2
16252 033/300 Read block 3936 len 4096
16253 034/300 Write block 451 len 16384 data: 5 3 3 3
16254 035/300 Flush
16255 036/300 Flush
16256 037/300 Write block 4830 len 24576 data: 4 4 3 2 3 3
16257 038/300 Write block 4420 len 20480 data: 3 3 4 5 5
16258 039/300 Read block 1600 len 32768
16259 040/300 Flush
16260 041/300 Read block 2416 len 32768
16261 042/300 Write block 2653 len 4096 data: 2
16262 043/300 Write block 3034 len 12288 data: 3 3 3
16263 044/300 Read block 2786 len 32768
16264 045/300 Write block 1322 len 28672 data: 4 4 4 4 5 4 4
16265 046/300 Write block 3502 len 12288 data: 2 2 3
16266 047/300 Read block 4449 len 8192
16267 048/300 Write block 2524 len 20480 data: 4 4 3 3 4
16268 049/300 Read block 3694 len 32768
16269 050/300 Write block 4667 len 16384 data: 2 2 2 2
16270 051/300 Write block 3690 len 32768 data: 5 4 3 3 2 2 2 2
16271 052/300 Write block 339 len 28672 data: 4 4 5 5 4 4 2
16272 053/300 Write block 2644 len 28672 data: 2 2 2 2 2 2 2
16273 054/300 Read block 4023 len 8192
16274 055/300 Read block 2930 len 16384
16275 056/300 Write block 770 len 4096 data: 5
16276 057/300 Flush
16277 058/300 Read block 4918 len 12288
16278 059/300 Write block 0 len 8192 data: 2 2
16279 060/300 Read block 2442 len 16384
16280 061/300 Write block 1309 len 12288 data: 4 3 3
16281 062/300 Flush
16282 063/300 Read block 3271 len 8192
16283 064/300 Read block 149 len 40960
16284 065/300 Read block 818 len 8192
16285 066/300 Write block 3128 len 4096 data: 5
16286 067/300 Write block 2423 len 28672 data: 3 4 4 4 4 3 2
16287 068/300 Flush
16288 069/300 Read block 4284 len 28672
16289 070/300 Write block 4443 len 12288 data: 4 3 3
16290 071/300 Read block 2593 len 8192
16291 072/300 Read block 830 len 16384
16292 073/300 Flush
16293 074/300 Read block 1898 len 16384
16294 075/300 Flush
16295 076/300 Read block 1489 len 28672
16296 077/300 Write block 4575 len 28672 data: 5 5 3 4 3 4 3
16297 078/300 Write block 3059 len 40960 data: 3 4 4 4 4 2 2 3 3 3
16298 079/300 Read block 2283 len 24576
16299 080/300 Write block 153 len 4096 data: 3
16300 081/300 Read block 4959 len 24576
16301 082/300 Write block 3402 len 16384 data: 5 4 3 4
16302 083/300 Read block 1755 len 16384
16303 084/300 Read block 2701 len 28672
16304 085/300 Read block 415 len 20480
16305 086/300 Read block 4461 len 40960
16306 087/300 Write block 3385 len 28672 data: 6 5 5 7 5 5 5
16307 088/300 Read block 4819 len 24576
16308 089/300 Flush
16309 090/300 Write block 76 len 8192 data: 3 3
16310 091/300 Write block 3804 len 36864 data: 3 4 4 4 4 4 5 5 4
16311 092/300 Write block 2962 len 16384 data: 3 4 3 3
16312 093/300 Write block 2467 len 36864 data: 5 5 4 5 4 4 4 5 5
16313 094/300 Read block 2554 len 40960
16314 095/300 Write block 1876 len 8192 data: 3 4
16315 096/300 Read block 4002 len 20480
16316 097/300 Write block 477 len 36864 data: 2 3 3 3 3 4 3 5 4
16317 098/300 Write block 142 len 16384 data: 4 4 3 3
16318 099/300 Write block 3593 len 24576 data: 5 5 5 3 4 4
16319 100/300 Write block 841 len 8192 data: 2 3
16320 101/300 Write block 2750 len 24576 data: 7 6 7 7 6 6
16321 102/300 Flush
16322 103/300 Flush
16323 104/300 Write block 230 len 20480 data: 5 5 5 4 5
16324 105/300 Read block 2019 len 32768
16325 106/300 Write block 2507 len 24576 data: 3 3 3 3 3 3
16326 107/300 Write block 4298 len 16384 data: 5 4 2 2
16327 108/300 Write block 919 len 4096 data: 3
16328 109/300 Write block 2262 len 16384 data: 3 2 2 4
16329 110/300 Read block 2098 len 16384
16330 111/300 Write block 4799 len 32768 data: 5 4 3 3 4 4 4 4
16331 112/300 Write block 4788 len 16384 data: 3 3 3 3
16332 113/300 Write block 1515 len 24576 data: 4 4 3 2 2 2
16333 114/300 Write block 2135 len 16384 data: 3 3 3 3
16334 115/300 Write block 2127 len 40960 data: 3 3 4 4 4 4 4 3 4 4
16335 116/300 Write block 1677 len 36864 data: 2 2 2 2 3 3 4 4 4
16336 117/300 Write block 4344 len 20480 data: 6 6 7 5 5
16337 118/300 Read block 2646 len 28672
16338 119/300 Read block 3788 len 24576
16339 120/300 Write block 3361 len 12288 data: 3 4 4
16340 121/300 Read block 2531 len 24576
16341 122/300 Read block 3600 len 4096
16342 123/300 Read block 3872 len 16384
16343 124/300 Write block 1624 len 8192 data: 5 5
16344 125/300 Flush
16345 126/300 Write block 1981 len 32768 data: 3 5 6 5 5 5 5 5
16346 127/300 Read block 806 len 24576
16347 128/300 Write block 3030 len 20480 data: 5 3 3 3 4
16348 129/300 Read block 602 len 24576
16349 130/300 Flush
16350 131/300 Write block 4818 len 16384 data: 3 3 3 2
16351 132/300 Read block 4150 len 8192
16352 133/300 Read block 1837 len 16384
16353 134/300 Write block 231 len 12288 data: 6 6 5
16354 135/300 Read block 832 len 32768
16355 136/300 Read block 641 len 32768
16356 137/300 Read block 198 len 8192
16357 138/300 Read block 3070 len 36864
16358 139/300 Read block 1244 len 20480
16359 140/300 Read block 2320 len 8192
16360 141/300 Write block 4534 len 12288 data: 5 6 6
16361 142/300 Read block 3059 len 12288
16362 143/300 Read block 1425 len 32768
16363 144/300 Read block 1114 len 32768
16364 145/300 Flush
16365 146/300 Read block 4100 len 32768
16366 147/300 Write block 4700 len 36864 data: 4 4 4 3 3 4 3 2 2
16367 148/300 Write block 2430 len 16384 data: 2 2 2 2
16368 149/300 Read block 2193 len 12288
16369 150/300 Write block 3707 len 16384 data: 2 3 4 3
16370 151/300 Write block 3429 len 24576 data: 3 3 2 3 5 6
16371 152/300 Read block 755 len 28672
16372 153/300 Read block 4002 len 12288
16373 154/300 Write block 1815 len 12288 data: 3 3 4
16374 155/300 Write block 1841 len 28672 data: 3 3 3 4 5 5 5
16375 156/300 Write block 2097 len 4096 data: 4
16376 157/300 Write block 440 len 8192 data: 5 4
16377 158/300 Read block 837 len 8192
16378 159/300 Flush
16379 160/300 Write block 2300 len 4096 data: 4
16380 161/300 Read block 1132 len 32768
16381 162/300 Write block 2834 len 24576 data: 5 5 4 4 3 2
16382 163/300 Read block 636 len 20480
16383 164/300 Write block 652 len 24576 data: 3 3 4 4 4 5
16384 165/300 Write block 2535 len 4096 data: 6
16385 166/300 Read block 3518 len 4096
16386 167/300 Write block 3913 len 28672 data: 5 5 5 4 4 4 3
16387 168/300 Write block 249 len 32768 data: 3 4 4 3 5 6 5 4
16388 169/300 Read block 2269 len 8192
16389 170/300 Read block 4900 len 32768
16390 171/300 Read block 893 len 16384
16391 172/300 Flush
16392 173/300 Write block 4012 len 4096 data: 6
16393 174/300 Write block 2441 len 16384 data: 6 5 4 4
16394 175/300 Read block 2857 len 4096
16395 176/300 Flush
16396 177/300 Write block 3619 len 32768 data: 2 2 3 3 3 3 3 3
16397 178/300 Read block 1763 len 20480
16398 179/300 Read block 3477 len 20480
16399 180/300 Read block 2598 len 12288
16400 181/300 Read block 2073 len 28672
16401 182/300 Read block 3797 len 20480
16402 183/300 Write block 3720 len 20480 data: 4 5 5 5 4
16403 184/300 Read block 3003 len 28672
16404 185/300 Read block 249 len 16384
16405 186/300 Flush
16406 187/300 Read block 479 len 32768
16407 188/300 Read block 2214 len 24576
16408 189/300 Read block 2915 len 28672
16409 190/300 Read block 2836 len 32768
16410 191/300 Write block 899 len 20480 data: 3 3 4 3 5
16411 192/300 Read block 4065 len 40960
16412 193/300 Flush
16413 194/300 Write block 3516 len 8192 data: 2 2
16414 195/300 Flush
16415 196/300 Read block 861 len 36864
16416 197/300 Read block 4174 len 24576
16417 198/300 Read block 265 len 24576
16418 199/300 Write block 2832 len 36864 data: 5 5 6 6 5 5 4 3 2
16419 200/300 Write block 4041 len 28672 data: 4 4 3 3 3 3 3
16420 201/300 Write block 4694 len 8192 data: 4 4
16421 202/300 Flush
16422 203/300 Read block 4705 len 4096
16423 204/300 Write block 2987 len 8192 data: 3 3
16424 205/300 Flush
16425 206/300 Write block 3404 len 20480 data: 4 5 5 5 5
16426 207/300 Read block 4337 len 4096
16427 208/300 Read block 3051 len 40960
16428 209/300 Read block 4284 len 4096
16429 210/300 Read block 1057 len 16384
16430 211/300 Write block 349 len 8192 data: 2 2
16431 212/300 Write block 2358 len 8192 data: 2 2
16432 213/300 Write block 415 len 20480 data: 7 6 6 6 6
16433 214/300 Write block 3905 len 4096 data: 3
16434 215/300 Read block 1032 len 32768
16435 216/300 Flush
16436 217/300 Read block 1975 len 32768
16437 218/300 Read block 2369 len 32768
16438 219/300 Read block 1333 len 4096
16439 220/300 Read block 3855 len 28672
16440 221/300 Read block 1715 len 4096
16441 222/300 Read block 3420 len 8192
16442 223/300 Read block 3494 len 12288
16443 224/300 Read block 4770 len 20480
16444 225/300 Read block 1836 len 8192
16445 226/300 Write block 4095 len 28672 data: 3 5 5 4 4 4 3
16446 227/300 Read block 1077 len 16384
16447 228/300 Read block 907 len 40960
16448 229/300 Write block 3889 len 32768 data: 4 4 5 4 4 4 4 5
16449 230/300 Flush
16450 231/300 Read block 3296 len 24576
16451 232/300 Read block 4053 len 4096
16452 233/300 Read block 1873 len 36864
16453 234/300 Read block 1453 len 28672
16454 235/300 Read block 1500 len 28672
16455 236/300 Read block 3140 len 8192
16456 237/300 Write block 2068 len 32768 data: 3 4 3 5 5 5 6 6
16457 238/300 Read block 1045 len 36864
16458 239/300 Flush
16459 240/300 Read block 3301 len 40960
16460 241/300 Read block 3852 len 36864
16461 242/300 Write block 2531 len 28672 data: 6 7 6 7 7 5 5
16462 243/300 Flush
16463 244/300 Read block 4003 len 8192
16464 245/300 Read block 4844 len 32768
16465 246/300 Read block 1957 len 36864
16466 247/300 Read block 4253 len 12288
16467 248/300 Read block 3045 len 32768
16468 249/300 Write block 2043 len 4096 data: 7
16469 250/300 Write block 521 len 36864 data: 6 7 7 7 6 5 5 6 5
16470 251/300 Read block 2013 len 40960
16471 252/300 Write block 4635 len 32768 data: 6 6 5 6 6 6 6 6
16472 253/300 Read block 2651 len 40960
16473 254/300 Read block 4173 len 4096
16474 255/300 Write block 3119 len 20480 data: 3 3 2 2 3
16475 256/300 Write block 3834 len 8192 data: 3 4
16476 257/300 Read block 1649 len 12288
16477 258/300 Write block 3716 len 28672 data: 4 3 4 5 5 6 6
16478 259/300 Write block 2842 len 28672 data: 2 2 2 3 2 3 3
16479 260/300 Read block 4697 len 28672
16480 261/300 Read block 2192 len 4096
16481 262/300 Write block 1985 len 4096 data: 6
16482 263/300 Read block 1611 len 28672
16483 264/300 Read block 574 len 28672
16484 265/300 Write block 1948 len 12288 data: 3 3 3
16485 266/300 Write block 2788 len 24576 data: 5 5 5 5 6 4
16486 267/300 Read block 2296 len 16384
16487 268/300 Read block 789 len 4096
16488 269/300 Read block 4723 len 16384
16489 270/300 Read block 2115 len 36864
16490 271/300 Read block 3627 len 8192
16491 272/300 Write block 3335 len 20480 data: 3 3 3 4 4
16492 273/300 Read block 1346 len 36864
16493 274/300 Write block 1835 len 12288 data: 3 3 3
16494 275/300 Read block 3256 len 24576
16495 276/300 Read block 2686 len 16384
16496 277/300 Write block 544 len 36864 data: 3 3 3 2 2 3 4 5 5
16497 278/300 Read block 4421 len 12288
16498 279/300 Write block 3412 len 24576 data: 7 7 6 6 5 5
16499 280/300 Read block 976 len 4096
16500 281/300 Write block 2086 len 8192 data: 4 4
16501 282/300 Write block 1785 len 20480 data: 5 5 5 4 4
16502 283/300 Write block 3874 len 28672 data: 8 8 8 8 8 7 6
16503 284/300 Write block 3673 len 40960 data: 3 3 3 3 3 5 6 6 6 6
16504 285/300 Write block 1078 len 12288 data: 2 2 2
16505 286/300 Write block 1075 len 40960 data: 4 4 3 3 3 3 2 2 3 3
16506 287/300 Write block 1007 len 12288 data: 4 3 3
16507 288/300 Read block 4075 len 8192
16508 289/300 Read block 948 len 24576
16509 290/300 Write block 4106 len 36864 data: 3 3 3 3 2 2 2 4 6
16510 291/300 Read block 2841 len 12288
16511 292/300 Read block 3342 len 28672
16512 293/300 Write block 1959 len 20480 data: 4 4 3 3 3
16513 294/300 Read block 4205 len 24576
16514 295/300 Write block 1325 len 40960 data: 5 6 5 5 3 3 3 3 2 2
16515 296/300 Read block 2907 len 4096
16516 297/300 Write block 4847 len 4096 data: 5
16517 298/300 Flush
16518 299/300 Read block 1478 len 36864
16519 300/300 Write block 1424 len 4096 data: 6
16520 301/300 Write block 3031 len 16384 data: 4 4 4 5
16521 Main task has work to do, go find it
16522 got dsc Start(1)
16523 start 1
16524 Mar 17 22:38:16.405 INFO request completed, latency_us: 200, response_code: 204, uri: /start/cid/1, method: POST, req_id: f673d98b-97a9-4f6e-8ce6-35605b3250f5, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
16525 Replay: started 1, returned:Ok(())
16526 ----------------------------------------------------------------
16527 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16528 States: Active Offline Active
16529 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16530 4926 Acked Read 7 Done Sent Done false
16531 4927 Acked Write 6 Done Sent Done false
16532 4928 Acked Write 1 Done Sent Done false
16533 4929 Acked Write 6 Done Sent Done false
16534 4930 Acked Write 2 Done Sent Done false
16535 4931 Acked Read 7 Done Sent Done false
16536 4932 Acked Read 9 Done Sent Done false
16537 4933 Acked Write 7 Done Sent Done false
16538 4934 Acked Read 4 Done Sent Done false
16539 4935 Acked Read 5 Done Sent Done false
16540 4936 Acked Read 5 Done Sent Done false
16541 4937 Acked Write 5 Done Sent Done false
16542 4938 Acked Write 3 Done Sent Done false
16543 4939 Acked Read 7 Done Sent Done false
16544 4940 Acked Read 10 Done Sent Done false
16545 4941 Acked Read 1 Done Sent Done false
16546 4942 Acked Write 6 Done Sent Done false
16547 4943 Acked Read 3 Done Sent Done false
16548 4944 Acked Read 10 Done Sent Done false
16549 4945 Acked Read 2 Done Sent Done false
16550 4946 Acked Read 6 Done Sent Done false
16551 4947 Acked Flush 0 Done Sent Done false
16552 4948 Acked Write 9 Done Sent Done false
16553 4949 Acked Flush 0 Done Sent Done false
16554 4950 Acked Read 7 Done Sent Done false
16555 4951 Acked Read 2 Done Sent Done false
16556 4952 Acked Write 5 Done Sent Done false
16557 4953 Acked Read 2 Done Sent Done false
16558 4954 Acked Read 10 Done Sent Done false
16559 4955 Acked Write 6 Done Sent Done false
16560 4956 Acked Read 7 Done Sent Done false
16561 4957 Acked Write 2 Done Sent Done false
16562 4958 Acked Read 1 Done Sent Done false
16563 4959 Acked Write 4 Done Sent Done false
16564 4960 Acked Flush 0 Done Sent Done false
16565 4961 Acked Flush 0 Done Sent Done false
16566 4962 Acked Write 6 Done Sent Done false
16567 4963 Acked Write 5 Done Sent Done false
16568 4964 Acked Read 8 Done Sent Done false
16569 4965 Acked Flush 0 Done Sent Done false
16570 4966 Acked Read 8 Done Sent Done false
16571 4967 Acked Write 1 Done Sent Done false
16572 4968 Acked Write 3 Done Sent Done false
16573 4969 Acked Read 8 Done Sent Done false
16574 4970 Acked Write 7 Done Sent Done false
16575 4971 Acked Write 3 Done Sent Done false
16576 4972 Acked Read 2 Done Sent Done false
16577 4973 Acked Write 5 Done Sent Done false
16578 4974 Acked Read 8 Done Sent Done false
16579 4975 Acked Write 4 Done Sent Done false
16580 4976 Acked Write 8 Done Sent Done false
16581 4977 Acked Write 7 Done Sent Done false
16582 4978 Acked Write 7 Done Sent Done false
16583 4979 Acked Read 2 Done Sent Done false
16584 4980 Acked Read 4 Done Sent Done false
16585 4981 Acked Write 1 Done Sent Done false
16586 4982 Acked Flush 0 Done Sent Done false
16587 4983 Acked Read 3 Done Sent Done false
16588 4984 Acked Write 2 Done Sent Done false
16589 4985 Acked Read 4 Done Sent Done false
16590 4986 Acked Write 3 Done Sent Done false
16591 4987 Acked Flush 0 Done Sent Done false
16592 4988 Acked Read 2 Done Sent Done false
16593 4989 Acked Read 10 Done Sent Done false
16594 4990 Acked Read 2 Done Sent Done false
16595 4991 Acked Write 1 Done Sent Done false
16596 4992 Acked Write 7 Done Sent Done false
16597 4993 Acked Flush 0 Done Sent Done false
16598 4994 Acked Read 7 Done Sent Done false
16599 4995 Acked Write 3 Done Sent Done false
16600 4996 Acked Read 2 Done Sent Done false
16601 4997 Acked Read 4 Done Sent Done false
16602 4998 Acked Flush 0 Done Sent Done false
16603 4999 Acked Read 4 Done Sent Done false
16604 5000 Acked Flush 0 Done Sent Done false
16605 5001 Acked Read 7 Done Sent Done false
16606 5002 Acked Write 7 Done Sent Done false
16607 5003 Acked Write 10 Done Sent Done false
16608 5004 Acked Read 6 Done Sent Done false
16609 5005 Acked Write 1 Done Sent Done false
16610 5006 Acked Read 6 Done Sent Done false
16611 5007 Acked Write 4 Done Sent Done false
16612 5008 Acked Read 4 Done Sent Done false
16613 5009 Acked Read 7 Done Sent Done false
16614 5010 Acked Read 5 Done Sent Done false
16615 5011 Acked Read 10 Done Sent Done false
16616 5012 Acked Write 7 Done Sent Done false
16617 5013 Acked Read 6 Done Sent Done false
16618 5014 Acked Flush 0 Done Sent Done false
16619 5015 Acked Write 2 Done Sent Done false
16620 5016 Acked Write 9 Done Sent Done false
16621 5017 Acked Write 4 Done Sent Done false
16622 5018 Acked Write 9 Done Sent Done false
16623 5019 Acked Read 10 Done Sent Done false
16624 5020 Acked Write 2 Done Sent Done false
16625 5021 Acked Read 5 Done Sent Done false
16626 5022 Acked Write 9 Done Sent Done false
16627 5023 Acked Write 4 Done Sent Done false
16628 5024 Acked Write 6 Done Sent Done false
16629 5025 Acked Write 2 Done Sent Done false
16630 5026 Acked Write 6 Done Sent Done false
16631 5027 Acked Flush 0 Done Sent Done false
16632 5028 Acked Flush 0 Done Sent Done false
16633 5029 Acked Write 5 Done Sent Done false
16634 5030 Acked Read 8 Done Sent Done false
16635 5031 Acked Write 6 Done Sent Done false
16636 5032 Acked Write 4 Done Sent Done false
16637 5033 Acked Write 1 Done Sent Done false
16638 5034 Acked Write 4 Done Sent Done false
16639 5035 Acked Read 4 Done Sent Done false
16640 5036 Acked Write 8 Done Sent Done false
16641 5037 Acked Write 4 Done Sent Done false
16642 5038 Acked Write 6 Done Sent Done false
16643 5039 Acked Write 4 Done Sent Done false
16644 5040 Acked Write 10 Done Sent Done false
16645 5041 Acked Write 9 Done Sent Done false
16646 5042 Acked Write 5 Done Sent Done false
16647 5043 Acked Read 7 Done Sent Done false
16648 5044 Acked Read 6 Done Sent Done false
16649 5045 Acked Write 3 Done Sent Done false
16650 5046 Acked Read 6 Done Sent Done false
16651 5047 Acked Read 1 Done Sent Done false
16652 5048 Acked Read 4 Done Sent Done false
16653 5049 Acked Write 2 Done Sent Done false
16654 5050 Acked Flush 0 Done Sent Done false
16655 5051 Acked Write 8 Done Sent Done false
16656 5052 Acked Read 6 Done Sent Done false
16657 5053 Acked Write 5 Done Sent Done false
16658 5054 Acked Read 6 Done Sent Done false
16659 5055 Acked Flush 0 Done Sent Done false
16660 5056 Acked Write 4 Done Sent Done false
16661 5057 Acked Read 2 Done Sent Done false
16662 5058 Acked Read 4 Done Sent Done false
16663 5059 Acked Write 3 Done Sent Done false
16664 5060 Acked Read 8 Done Sent Done false
16665 5061 Acked Read 8 Done Sent Done false
16666 5062 Acked Read 2 Done Sent Done false
16667 5063 Acked Read 9 Done Sent Done false
16668 5064 Acked Read 5 Done Sent Done false
16669 5065 Acked Read 2 Done Sent Done false
16670 5066 Acked Write 3 Done Sent Done false
16671 5067 Acked Read 3 Done Sent Done false
16672 5068 Acked Read 8 Done Sent Done false
16673 5069 Acked Read 8 Done Sent Done false
16674 5070 Acked Flush 0 Done Sent Done false
16675 5071 Acked Read 8 Done Sent Done false
16676 5072 Acked Write 9 Done Sent Done false
16677 5073 Acked Write 4 Done Sent Done false
16678 5074 Acked Read 3 Done Sent Done false
16679 5075 Acked Write 4 Done Sent Done false
16680 5076 Acked Write 6 Done Sent Done false
16681 5077 Acked Read 7 Done Sent Done false
16682 5078 Acked Read 3 Done Sent Done false
16683 5079 Acked Write 3 Done Sent Done false
16684 5080 Acked Write 7 Done Sent Done false
16685 5081 Acked Write 1 Done Sent Done false
16686 5082 Acked Write 2 Done Sent Done false
16687 5083 Acked Read 2 Done Sent Done false
16688 5084 Acked Flush 0 Done Sent Done false
16689 5085 Acked Write 1 Done Sent Done false
16690 5086 Acked Read 8 Done Sent Done false
16691 5087 Acked Write 6 Done Sent Done false
16692 5088 Acked Read 5 Done Sent Done false
16693 5089 Acked Write 6 Done Sent Done false
16694 5090 Acked Write 1 Done Sent Done false
16695 5091 Acked Read 1 Done Sent Done false
16696 5092 Acked Write 7 Done Sent Done false
16697 5093 Acked Write 8 Done Sent Done false
16698 5094 Acked Read 2 Done Sent Done false
16699 5095 Acked Read 8 Done Sent Done false
16700 5096 Acked Read 4 Done Sent Done false
16701 5097 Acked Flush 0 Done Sent Done false
16702 5098 Acked Write 1 Done Sent Done false
16703 5099 Acked Write 4 Done Sent Done false
16704 5100 Acked Read 1 Done Sent Done false
16705 5101 Acked Flush 0 Done Sent Done false
16706 5102 Acked Write 8 Done Sent Done false
16707 5103 Acked Read 5 Done Sent Done false
16708 5104 Acked Read 5 Done Sent Done false
16709 5105 Acked Read 3 Done Sent Done false
16710 5106 Acked Read 7 Done Sent Done false
16711 5107 Acked Read 5 Done Sent Done false
16712 5108 Acked Write 5 Done Sent Done false
16713 5109 Acked Read 7 Done Sent Done false
16714 5110 Acked Read 4 Done Sent Done false
16715 5111 Acked Flush 0 Done Sent Done false
16716 5112 Acked Read 8 Done Sent Done false
16717 5113 Acked Read 6 Done Sent Done false
16718 5114 Acked Read 7 Done Sent Done false
16719 5115 Acked Read 8 Done Sent Done false
16720 5116 Acked Write 5 Done Sent Done false
16721 5117 Acked Read 10 Done Sent Done false
16722 5118 Acked Flush 0 Done Sent Done false
16723 5119 Acked Write 2 Done Sent Done false
16724 5120 Acked Flush 0 Done Sent Done false
16725 5121 Acked Read 9 Done Sent Done false
16726 5122 Acked Read 6 Done Sent Done false
16727 5123 Acked Read 6 Done Sent Done false
16728 5124 Acked Write 9 Done Sent Done false
16729 5125 Acked Write 7 Done Sent Done false
16730 5126 Acked Write 2 Done Sent Done false
16731 5127 Acked Flush 0 Done Sent Done false
16732 5128 Acked Read 1 Done Sent Done false
16733 5129 Acked Write 2 Done Sent Done false
16734 5130 Acked Flush 0 Done Sent Done false
16735 5131 Acked Write 5 Done Sent Done false
16736 5132 Acked Read 1 Done Sent Done false
16737 5133 Acked Read 10 Done Sent Done false
16738 5134 Acked Read 1 Done Sent Done false
16739 5135 Acked Read 4 Done Sent Done false
16740 5136 Acked Write 2 Done Sent Done false
16741 5137 Acked Write 2 Done Sent Done false
16742 5138 Acked Write 5 Done Sent Done false
16743 5139 Acked Write 1 Done Sent Done false
16744 5140 Acked Read 8 Done Sent Done false
16745 5141 Acked Flush 0 Done Sent Done false
16746 5142 Acked Read 8 Done Sent Done false
16747 5143 Acked Read 8 Done Sent Done false
16748 5144 Acked Read 1 Done Sent Done false
16749 5145 Acked Read 7 Done Sent Done false
16750 5146 Acked Read 1 Done Sent Done false
16751 5147 Acked Read 2 Done Sent Done false
16752 5148 Acked Read 3 Done Sent Done false
16753 5149 Acked Read 5 Done Sent Done false
16754 5150 Acked Read 2 Done Sent Done false
16755 5151 Acked Write 7 Done Sent Done false
16756 5152 Acked Read 4 Done Sent Done false
16757 5153 Acked Read 10 Done Sent Done false
16758 5154 Acked Write 8 Done Sent Done false
16759 5155 Acked Flush 0 Done Sent Done false
16760 5156 Acked Read 6 Done Sent Done false
16761 5157 Acked Read 1 Done Sent Done false
16762 5158 Acked Read 9 Done Sent Done false
16763 5159 Acked Read 7 Done Sent Done false
16764 5160 Acked Read 7 Done Sent Done false
16765 5161 Acked Read 2 Done Sent Done false
16766 5162 Acked Write 8 Done Sent Done false
16767 5163 Acked Read 9 Done Sent Done false
16768 5164 Acked Flush 0 Done Sent Done false
16769 5165 Acked Read 10 Done Sent Done false
16770 5166 Acked Read 9 Done Sent Done false
16771 5167 Acked Write 7 Done Sent Done false
16772 5168 Acked Flush 0 Done Sent Done false
16773 5169 Acked Read 2 Done Sent Done false
16774 5170 Acked Read 8 Done Sent Done false
16775 5171 Acked Read 9 Done Sent Done false
16776 5172 Acked Read 3 Done Sent Done false
16777 5173 Acked Read 8 Done Sent Done false
16778 5174 Acked Write 1 Done Sent Done false
16779 5175 Acked Write 9 Done Sent Done false
16780 5176 Acked Read 10 Done Sent Done false
16781 5177 Acked Write 8 Done Sent Done false
16782 5178 Acked Read 10 Done Sent Done false
16783 5179 Acked Read 1 Done Sent Done false
16784 5180 Acked Write 5 Done Sent Done false
16785 5181 Acked Write 2 Done Sent Done false
16786 5182 Acked Read 3 Done Sent Done false
16787 5183 Acked Write 7 Done Sent Done false
16788 5184 Acked Write 7 Done Sent Done false
16789 5185 Acked Read 7 Done Sent Done false
16790 5186 Acked Read 1 Done Sent Done false
16791 5187 Acked Write 1 Done Sent Done false
16792 5188 Acked Read 7 Done Sent Done false
16793 5189 Acked Read 7 Done Sent Done false
16794 5190 Acked Write 3 Done Sent Done false
16795 5191 Acked Write 6 Done Sent Done false
16796 5192 Acked Read 4 Done Sent Done false
16797 5193 Acked Read 1 Done Sent Done false
16798 5194 Acked Read 4 Done Sent Done false
16799 5195 Acked Read 9 Done Sent Done false
16800 5196 Acked Read 2 Done Sent Done false
16801 5197 Acked Write 5 Done Sent Done false
16802 5198 Acked Read 9 Done Sent Done false
16803 5199 Acked Write 3 Done Sent Done false
16804 5200 Acked Read 6 Done Sent Done false
16805 5201 Acked Read 4 Done Sent Done false
16806 5202 Acked Write 9 Done Sent Done false
16807 5203 Acked Read 3 Done Sent Done false
16808 5204 Acked Write 6 Done Sent Done false
16809 5205 Acked Read 1 Done Sent Done false
16810 5206 Acked Write 2 Done Sent Done false
16811 5207 Acked Write 5 Done Sent Done false
16812 5208 Acked Write 7 Done Sent Done false
16813 5209 Acked Write 10 Done Sent Done false
16814 5210 Acked Write 3 Done Sent Done false
16815 5211 Acked Write 10 Done Sent Done false
16816 5212 Acked Write 3 Done Sent Done false
16817 5213 Acked Read 2 Done Sent Done false
16818 5214 Acked Read 6 Done Sent Done false
16819 5215 Acked Write 9 Done Sent Done false
16820 5216 Acked Read 3 Done Sent Done false
16821 5217 Acked Read 7 Done Sent Done false
16822 5218 Acked Write 5 Done Sent Done false
16823 5219 Acked Read 6 Done Sent Done false
16824 5220 Acked Write 10 Done Sent Done false
16825 5221 Acked Read 1 Done Sent Done false
16826 5222 Acked Write 1 Done Sent Done false
16827 5223 Acked Flush 0 Done Sent Done false
16828 5224 Acked Read 9 Done Sent Done false
16829 5225 Acked Write 1 Done Sent Done false
16830 5226 Acked Write 4 Done Sent Done false
16831 STATES DS:0 DS:1 DS:2 TOTAL
16832 Sent 0 301 0 301
16833 Done 301 0 301 602
16834 Skipped 0 0 0 0
16835 Error 0 0 0 0
16836 Last Flush: Some(JobId(5223)) Some(JobId(4925)) Some(JobId(5223))
16837 Downstairs last five completed: 4925 4924 4923 4922 4921
16838 Upstairs last five completed: 5226 5225 5224 5223 5222
16839 CLIENT: Up:0 ds:301 act:2
168402025-03-17T22:38:16.907ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
168412025-03-17T22:38:17.663ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
168422025-03-17T22:38:18.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
16843 [8820] got start action
16844 [8820] I am going to restart
16845 Starting downstairs at port 8820
16846 Make output file at "/tmp/dsc/downstairs-8820.txt"
16847 [8820][1] reports Starting
16848 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1222)
16849 [8820][1] reports Running
168502025-03-17T22:38:18.895ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
168512025-03-17T22:38:19.667ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
16852 ----------------------------------------------------------------
16853 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16854 States: Active Offline Active
16855 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16856 4926 Acked Read 7 Done Sent Done false
16857 4927 Acked Write 6 Done Sent Done false
16858 4928 Acked Write 1 Done Sent Done false
16859 4929 Acked Write 6 Done Sent Done false
16860 4930 Acked Write 2 Done Sent Done false
16861 4931 Acked Read 7 Done Sent Done false
16862 4932 Acked Read 9 Done Sent Done false
16863 4933 Acked Write 7 Done Sent Done false
16864 4934 Acked Read 4 Done Sent Done false
16865 4935 Acked Read 5 Done Sent Done false
16866 4936 Acked Read 5 Done Sent Done false
16867 4937 Acked Write 5 Done Sent Done false
16868 4938 Acked Write 3 Done Sent Done false
16869 4939 Acked Read 7 Done Sent Done false
16870 4940 Acked Read 10 Done Sent Done false
16871 4941 Acked Read 1 Done Sent Done false
16872 4942 Acked Write 6 Done Sent Done false
16873 4943 Acked Read 3 Done Sent Done false
16874 4944 Acked Read 10 Done Sent Done false
16875 4945 Acked Read 2 Done Sent Done false
16876 4946 Acked Read 6 Done Sent Done false
16877 4947 Acked Flush 0 Done Sent Done false
16878 4948 Acked Write 9 Done Sent Done false
16879 4949 Acked Flush 0 Done Sent Done false
16880 4950 Acked Read 7 Done Sent Done false
16881 4951 Acked Read 2 Done Sent Done false
16882 4952 Acked Write 5 Done Sent Done false
16883 4953 Acked Read 2 Done Sent Done false
16884 4954 Acked Read 10 Done Sent Done false
16885 4955 Acked Write 6 Done Sent Done false
16886 4956 Acked Read 7 Done Sent Done false
16887 4957 Acked Write 2 Done Sent Done false
16888 4958 Acked Read 1 Done Sent Done false
16889 4959 Acked Write 4 Done Sent Done false
16890 4960 Acked Flush 0 Done Sent Done false
16891 4961 Acked Flush 0 Done Sent Done false
16892 4962 Acked Write 6 Done Sent Done false
16893 4963 Acked Write 5 Done Sent Done false
16894 4964 Acked Read 8 Done Sent Done false
16895 4965 Acked Flush 0 Done Sent Done false
16896 4966 Acked Read 8 Done Sent Done false
16897 4967 Acked Write 1 Done Sent Done false
16898 4968 Acked Write 3 Done Sent Done false
16899 4969 Acked Read 8 Done Sent Done false
16900 4970 Acked Write 7 Done Sent Done false
16901 4971 Acked Write 3 Done Sent Done false
16902 4972 Acked Read 2 Done Sent Done false
16903 4973 Acked Write 5 Done Sent Done false
16904 4974 Acked Read 8 Done Sent Done false
16905 4975 Acked Write 4 Done Sent Done false
16906 4976 Acked Write 8 Done Sent Done false
16907 4977 Acked Write 7 Done Sent Done false
16908 4978 Acked Write 7 Done Sent Done false
16909 4979 Acked Read 2 Done Sent Done false
16910 4980 Acked Read 4 Done Sent Done false
16911 4981 Acked Write 1 Done Sent Done false
16912 4982 Acked Flush 0 Done Sent Done false
16913 4983 Acked Read 3 Done Sent Done false
16914 4984 Acked Write 2 Done Sent Done false
16915 4985 Acked Read 4 Done Sent Done false
16916 4986 Acked Write 3 Done Sent Done false
16917 4987 Acked Flush 0 Done Sent Done false
16918 4988 Acked Read 2 Done Sent Done false
16919 4989 Acked Read 10 Done Sent Done false
16920 4990 Acked Read 2 Done Sent Done false
16921 4991 Acked Write 1 Done Sent Done false
16922 4992 Acked Write 7 Done Sent Done false
16923 4993 Acked Flush 0 Done Sent Done false
16924 4994 Acked Read 7 Done Sent Done false
16925 4995 Acked Write 3 Done Sent Done false
16926 4996 Acked Read 2 Done Sent Done false
16927 4997 Acked Read 4 Done Sent Done false
16928 4998 Acked Flush 0 Done Sent Done false
16929 4999 Acked Read 4 Done Sent Done false
16930 5000 Acked Flush 0 Done Sent Done false
16931 5001 Acked Read 7 Done Sent Done false
16932 5002 Acked Write 7 Done Sent Done false
16933 5003 Acked Write 10 Done Sent Done false
16934 5004 Acked Read 6 Done Sent Done false
16935 5005 Acked Write 1 Done Sent Done false
16936 5006 Acked Read 6 Done Sent Done false
16937 5007 Acked Write 4 Done Sent Done false
16938 5008 Acked Read 4 Done Sent Done false
16939 5009 Acked Read 7 Done Sent Done false
16940 5010 Acked Read 5 Done Sent Done false
16941 5011 Acked Read 10 Done Sent Done false
16942 5012 Acked Write 7 Done Sent Done false
16943 5013 Acked Read 6 Done Sent Done false
16944 5014 Acked Flush 0 Done Sent Done false
16945 5015 Acked Write 2 Done Sent Done false
16946 5016 Acked Write 9 Done Sent Done false
16947 5017 Acked Write 4 Done Sent Done false
16948 5018 Acked Write 9 Done Sent Done false
16949 5019 Acked Read 10 Done Sent Done false
16950 5020 Acked Write 2 Done Sent Done false
16951 5021 Acked Read 5 Done Sent Done false
16952 5022 Acked Write 9 Done Sent Done false
16953 5023 Acked Write 4 Done Sent Done false
16954 5024 Acked Write 6 Done Sent Done false
16955 5025 Acked Write 2 Done Sent Done false
16956 5026 Acked Write 6 Done Sent Done false
16957 5027 Acked Flush 0 Done Sent Done false
16958 5028 Acked Flush 0 Done Sent Done false
16959 5029 Acked Write 5 Done Sent Done false
16960 5030 Acked Read 8 Done Sent Done false
16961 5031 Acked Write 6 Done Sent Done false
16962 5032 Acked Write 4 Done Sent Done false
16963 5033 Acked Write 1 Done Sent Done false
16964 5034 Acked Write 4 Done Sent Done false
16965 5035 Acked Read 4 Done Sent Done false
16966 5036 Acked Write 8 Done Sent Done false
16967 5037 Acked Write 4 Done Sent Done false
16968 5038 Acked Write 6 Done Sent Done false
16969 5039 Acked Write 4 Done Sent Done false
16970 5040 Acked Write 10 Done Sent Done false
16971 5041 Acked Write 9 Done Sent Done false
16972 5042 Acked Write 5 Done Sent Done false
16973 5043 Acked Read 7 Done Sent Done false
16974 5044 Acked Read 6 Done Sent Done false
16975 5045 Acked Write 3 Done Sent Done false
16976 5046 Acked Read 6 Done Sent Done false
16977 5047 Acked Read 1 Done Sent Done false
16978 5048 Acked Read 4 Done Sent Done false
16979 5049 Acked Write 2 Done Sent Done false
16980 5050 Acked Flush 0 Done Sent Done false
16981 5051 Acked Write 8 Done Sent Done false
16982 5052 Acked Read 6 Done Sent Done false
16983 5053 Acked Write 5 Done Sent Done false
16984 5054 Acked Read 6 Done Sent Done false
16985 5055 Acked Flush 0 Done Sent Done false
16986 5056 Acked Write 4 Done Sent Done false
16987 5057 Acked Read 2 Done Sent Done false
16988 5058 Acked Read 4 Done Sent Done false
16989 5059 Acked Write 3 Done Sent Done false
16990 5060 Acked Read 8 Done Sent Done false
16991 5061 Acked Read 8 Done Sent Done false
16992 5062 Acked Read 2 Done Sent Done false
16993 5063 Acked Read 9 Done Sent Done false
16994 5064 Acked Read 5 Done Sent Done false
16995 5065 Acked Read 2 Done Sent Done false
16996 5066 Acked Write 3 Done Sent Done false
16997 5067 Acked Read 3 Done Sent Done false
16998 5068 Acked Read 8 Done Sent Done false
16999 5069 Acked Read 8 Done Sent Done false
17000 5070 Acked Flush 0 Done Sent Done false
17001 5071 Acked Read 8 Done Sent Done false
17002 5072 Acked Write 9 Done Sent Done false
17003 5073 Acked Write 4 Done Sent Done false
17004 5074 Acked Read 3 Done Sent Done false
17005 5075 Acked Write 4 Done Sent Done false
17006 5076 Acked Write 6 Done Sent Done false
17007 5077 Acked Read 7 Done Sent Done false
17008 5078 Acked Read 3 Done Sent Done false
17009 5079 Acked Write 3 Done Sent Done false
17010 5080 Acked Write 7 Done Sent Done false
17011 5081 Acked Write 1 Done Sent Done false
17012 5082 Acked Write 2 Done Sent Done false
17013 5083 Acked Read 2 Done Sent Done false
17014 5084 Acked Flush 0 Done Sent Done false
17015 5085 Acked Write 1 Done Sent Done false
17016 5086 Acked Read 8 Done Sent Done false
17017 5087 Acked Write 6 Done Sent Done false
17018 5088 Acked Read 5 Done Sent Done false
17019 5089 Acked Write 6 Done Sent Done false
17020 5090 Acked Write 1 Done Sent Done false
17021 5091 Acked Read 1 Done Sent Done false
17022 5092 Acked Write 7 Done Sent Done false
17023 5093 Acked Write 8 Done Sent Done false
17024 5094 Acked Read 2 Done Sent Done false
17025 5095 Acked Read 8 Done Sent Done false
17026 5096 Acked Read 4 Done Sent Done false
17027 5097 Acked Flush 0 Done Sent Done false
17028 5098 Acked Write 1 Done Sent Done false
17029 5099 Acked Write 4 Done Sent Done false
17030 5100 Acked Read 1 Done Sent Done false
17031 5101 Acked Flush 0 Done Sent Done false
17032 5102 Acked Write 8 Done Sent Done false
17033 5103 Acked Read 5 Done Sent Done false
17034 5104 Acked Read 5 Done Sent Done false
17035 5105 Acked Read 3 Done Sent Done false
17036 5106 Acked Read 7 Done Sent Done false
17037 5107 Acked Read 5 Done Sent Done false
17038 5108 Acked Write 5 Done Sent Done false
17039 5109 Acked Read 7 Done Sent Done false
17040 5110 Acked Read 4 Done Sent Done false
17041 5111 Acked Flush 0 Done Sent Done false
17042 5112 Acked Read 8 Done Sent Done false
17043 5113 Acked Read 6 Done Sent Done false
17044 5114 Acked Read 7 Done Sent Done false
17045 5115 Acked Read 8 Done Sent Done false
17046 5116 Acked Write 5 Done Sent Done false
17047 5117 Acked Read 10 Done Sent Done false
17048 5118 Acked Flush 0 Done Sent Done false
17049 5119 Acked Write 2 Done Sent Done false
17050 5120 Acked Flush 0 Done Sent Done false
17051 5121 Acked Read 9 Done Sent Done false
17052 5122 Acked Read 6 Done Sent Done false
17053 5123 Acked Read 6 Done Sent Done false
17054 5124 Acked Write 9 Done Sent Done false
17055 5125 Acked Write 7 Done Sent Done false
17056 5126 Acked Write 2 Done Sent Done false
17057 5127 Acked Flush 0 Done Sent Done false
17058 5128 Acked Read 1 Done Sent Done false
17059 5129 Acked Write 2 Done Sent Done false
17060 5130 Acked Flush 0 Done Sent Done false
17061 5131 Acked Write 5 Done Sent Done false
17062 5132 Acked Read 1 Done Sent Done false
17063 5133 Acked Read 10 Done Sent Done false
17064 5134 Acked Read 1 Done Sent Done false
17065 5135 Acked Read 4 Done Sent Done false
17066 5136 Acked Write 2 Done Sent Done false
17067 5137 Acked Write 2 Done Sent Done false
17068 5138 Acked Write 5 Done Sent Done false
17069 5139 Acked Write 1 Done Sent Done false
17070 5140 Acked Read 8 Done Sent Done false
17071 5141 Acked Flush 0 Done Sent Done false
17072 5142 Acked Read 8 Done Sent Done false
17073 5143 Acked Read 8 Done Sent Done false
17074 5144 Acked Read 1 Done Sent Done false
17075 5145 Acked Read 7 Done Sent Done false
17076 5146 Acked Read 1 Done Sent Done false
17077 5147 Acked Read 2 Done Sent Done false
17078 5148 Acked Read 3 Done Sent Done false
17079 5149 Acked Read 5 Done Sent Done false
17080 5150 Acked Read 2 Done Sent Done false
17081 5151 Acked Write 7 Done Sent Done false
17082 5152 Acked Read 4 Done Sent Done false
17083 5153 Acked Read 10 Done Sent Done false
17084 5154 Acked Write 8 Done Sent Done false
17085 5155 Acked Flush 0 Done Sent Done false
17086 5156 Acked Read 6 Done Sent Done false
17087 5157 Acked Read 1 Done Sent Done false
17088 5158 Acked Read 9 Done Sent Done false
17089 5159 Acked Read 7 Done Sent Done false
17090 5160 Acked Read 7 Done Sent Done false
17091 5161 Acked Read 2 Done Sent Done false
17092 5162 Acked Write 8 Done Sent Done false
17093 5163 Acked Read 9 Done Sent Done false
17094 5164 Acked Flush 0 Done Sent Done false
17095 5165 Acked Read 10 Done Sent Done false
17096 5166 Acked Read 9 Done Sent Done false
17097 5167 Acked Write 7 Done Sent Done false
17098 5168 Acked Flush 0 Done Sent Done false
17099 5169 Acked Read 2 Done Sent Done false
17100 5170 Acked Read 8 Done Sent Done false
17101 5171 Acked Read 9 Done Sent Done false
17102 5172 Acked Read 3 Done Sent Done false
17103 5173 Acked Read 8 Done Sent Done false
17104 5174 Acked Write 1 Done Sent Done false
17105 5175 Acked Write 9 Done Sent Done false
17106 5176 Acked Read 10 Done Sent Done false
17107 5177 Acked Write 8 Done Sent Done false
17108 5178 Acked Read 10 Done Sent Done false
17109 5179 Acked Read 1 Done Sent Done false
17110 5180 Acked Write 5 Done Sent Done false
17111 5181 Acked Write 2 Done Sent Done false
17112 5182 Acked Read 3 Done Sent Done false
17113 5183 Acked Write 7 Done Sent Done false
17114 5184 Acked Write 7 Done Sent Done false
17115 5185 Acked Read 7 Done Sent Done false
17116 5186 Acked Read 1 Done Sent Done false
17117 5187 Acked Write 1 Done Sent Done false
17118 5188 Acked Read 7 Done Sent Done false
17119 5189 Acked Read 7 Done Sent Done false
17120 5190 Acked Write 3 Done Sent Done false
17121 5191 Acked Write 6 Done Sent Done false
17122 5192 Acked Read 4 Done Sent Done false
17123 5193 Acked Read 1 Done Sent Done false
17124 5194 Acked Read 4 Done Sent Done false
17125 5195 Acked Read 9 Done Sent Done false
17126 5196 Acked Read 2 Done Sent Done false
17127 5197 Acked Write 5 Done Sent Done false
17128 5198 Acked Read 9 Done Sent Done false
17129 5199 Acked Write 3 Done Sent Done false
17130 5200 Acked Read 6 Done Sent Done false
17131 5201 Acked Read 4 Done Sent Done false
17132 5202 Acked Write 9 Done Sent Done false
17133 5203 Acked Read 3 Done Sent Done false
17134 5204 Acked Write 6 Done Sent Done false
17135 5205 Acked Read 1 Done Sent Done false
17136 5206 Acked Write 2 Done Sent Done false
17137 5207 Acked Write 5 Done Sent Done false
17138 5208 Acked Write 7 Done Sent Done false
17139 5209 Acked Write 10 Done Sent Done false
17140 5210 Acked Write 3 Done Sent Done false
17141 5211 Acked Write 10 Done Sent Done false
17142 5212 Acked Write 3 Done Sent Done false
17143 5213 Acked Read 2 Done Sent Done false
17144 5214 Acked Read 6 Done Sent Done false
17145 5215 Acked Write 9 Done Sent Done false
17146 5216 Acked Read 3 Done Sent Done false
17147 5217 Acked Read 7 Done Sent Done false
17148 5218 Acked Write 5 Done Sent Done false
17149 5219 Acked Read 6 Done Sent Done false
17150 5220 Acked Write 10 Done Sent Done false
17151 5221 Acked Read 1 Done Sent Done false
17152 5222 Acked Write 1 Done Sent Done false
17153 5223 Acked Flush 0 Done Sent Done false
17154 5224 Acked Read 9 Done Sent Done false
17155 5225 Acked Write 1 Done Sent Done false
17156 5226 Acked Write 4 Done Sent Done false
17157 STATES DS:0 DS:1 DS:2 TOTAL
17158 Sent 0 301 0 301
17159 Done 301 0 301 602
17160 Skipped 0 0 0 0
17161 Error 0 0 0 0
17162 Last Flush: Some(JobId(5223)) Some(JobId(4925)) Some(JobId(5223))
17163 Downstairs last five completed: 4925 4924 4923 4922 4921
17164 Upstairs last five completed: 5226 5225 5224 5223 5222
17165 CLIENT: Up:0 ds:301 act:2
171662025-03-17T22:38:21.157ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171672025-03-17T22:38:21.670ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171682025-03-17T22:38:22.169ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171692025-03-17T22:38:22.169ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60556) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171702025-03-17T22:38:22.170ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171712025-03-17T22:38:22.171ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171722025-03-17T22:38:22.171ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171732025-03-17T22:38:22.171ZINFOcrucible: send last flush ID to this DS: Some(JobId(4925)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171742025-03-17T22:38:22.171ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171752025-03-17T22:38:22.171ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4925)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171762025-03-17T22:38:22.171ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4925)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
17177 ----------------------------------------------------------------
17178 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17179 Downstairs last five completed: 5227 5226 5225 5224 5223
17180 Upstairs last five completed: 5227 5226 5225 5224 5223
17181 CLIENT: Up:0 ds:0 act:3
17182 Replay: All jobs finished
17183 Main task has work to do, go find it
17184 got dsc Stop(0)
17185 stop 0
17186 Mar 17 22:38:24.410 INFO request completed, [8810] Got stop action so:false kr:false
17187 latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9d764290-c275-444e-88d7-6a4b5dab2c01, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
17188 Mar 17 22:38:24.411 INFO request completed, latency_us: 153, response_code: 200, uri: /state/cid/0, method: GET, req_id: c0b9b0de-5611-42fe-9373-df00ee3f2770, remote_addr: 127.0.0.1:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:38:24.411643625Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
17189 35180, local_addr: 127.0.0.1:9998
171902025-03-17T22:38:24.411ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171912025-03-17T22:38:24.411ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
171922025-03-17T22:38:24.411ZINFOcrucible: client task is exiting = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
17193 [8810] Exited with: ExitStatus(unix_wait_status(9))
17194 [8810][0] reports Exit
17195 Mar 17 22:38:28.414 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7f605c8f-05e8-4389-ae4f-fbc4b8010a07, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
17196 001/300 Read block 424 len 20480
17197 002/300 Write block 2198 len 40960 data: 3 3 3 5 6 6 6 8 6 5
17198 003/300 Read block 3691 len 24576
17199 004/300 Write block 3824 len 12288 data: 3 3 2
17200 005/300 Read block 2898 len 12288
17201 006/300 Flush
17202 007/300 Read block 2106 len 32768
17203 008/300 Read block 3609 len 4096
17204 009/300 Read block 1739 len 32768
17205 010/300 Read block 2678 len 4096
17206 011/300 Flush
17207 012/300 Read block 348 len 32768
17208 013/300 Flush
17209 014/300 Read block 793 len 32768
17210 015/300 Read block 934 len 12288
17211 016/300 Flush
17212 017/300 Read block 4295 len 32768
17213 018/300 Read block 2744 len 28672
17214 019/300 Flush
17215 020/300 Write block 3961 len 12288 data: 4 5 5
17216 021/300 Write block 2562 len 40960 data: 4 4 4 4 4 3 4 3 3 4
17217 022/300 Read block 3867 len 4096
17218 023/300 Flush
17219 024/300 Read block 4336 len 40960
17220 025/300 Write block 3813 len 16384 data: 4 3 2 2
17221 026/300 Write block 2952 len 24576 data: 4 4 4 3 3 3
17222 027/300 Write block 4386 len 20480 data: 4 4 4 4 4
17223 028/300 Write block 4547 len 4096 data: 3
17224 029/300 Read block 653 len 32768
17225 030/300 Write block 1379 len 24576 data: 4 5 4 4 5 4
17226 031/300 Read block 4215 len 20480
17227 032/300 Read block 4393 len 36864
17228 033/300 Read block 3461 len 28672
17229 034/300 Read block 3165 len 12288
17230 035/300 Read block 2870 len 32768
17231 036/300 Read block 1819 len 28672
17232 037/300 Read block 1201 len 32768
17233 038/300 Read block 2238 len 4096
17234 039/300 Write block 4486 len 12288 data: 3 3 4
17235 040/300 Read block 4206 len 36864
17236 041/300 Read block 3013 len 40960
17237 042/300 Write block 256 len 40960 data: 5 4 4 3 3 3 4 6 7 8
17238 043/300 Read block 838 len 32768
17239 044/300 Read block 1610 len 40960
17240 045/300 Write block 4544 len 40960 data: 3 3 2 4 3 5 6 5 4 4
17241 046/300 Write block 3280 len 28672 data: 4 4 3 3 3 3 3
17242 047/300 Write block 3627 len 40960 data: 4 4 4 3 3 4 4 4 4 4
17243 048/300 Read block 3924 len 16384
17244 049/300 Write block 1210 len 40960 data: 3 3 4 3 3 3 3 4 3 3
17245 050/300 Write block 1672 len 12288 data: 2 2 2
17246 051/300 Flush
17247 052/300 Read block 2639 len 20480
17248 053/300 Write block 4434 len 4096 data: 4
17249 054/300 Flush
17250 055/300 Write block 974 len 24576 data: 5 5 4 4 3 4
17251 056/300 Read block 4903 len 4096
17252 057/300 Write block 1404 len 8192 data: 3 3
17253 058/300 Read block 1587 len 32768
17254 059/300 Flush
17255 060/300 Write block 2028 len 36864 data: 6 6 7 7 7 7 7 6 5
17256 061/300 Read block 2407 len 40960
17257 062/300 Read block 3259 len 12288
17258 063/300 Read block 633 len 8192
17259 064/300 Write block 2707 len 32768 data: 3 2 2 3 2 2 2 4
17260 065/300 Write block 2770 len 36864 data: 5 4 5 4 4 2 2 3 3
17261 066/300 Write block 3891 len 12288 data: 6 5 5
17262 067/300 Read block 865 len 36864
17263 068/300 Read block 3353 len 20480
17264 069/300 Write block 3043 len 4096 data: 3
17265 070/300 Flush
17266 071/300 Write block 4296 len 28672 data: 5 5 6 5 3 3 3
17267 072/300 Write block 3520 len 8192 data: 4 4
17268 073/300 Read block 1037 len 8192
17269 074/300 Read block 2820 len 24576
17270 075/300 Write block 2147 len 8192 data: 5 4
17271 076/300 Read block 3090 len 24576
17272 077/300 Read block 2250 len 24576
17273 078/300 Write block 2048 len 28672 data: 5 5 5 3 3 3 3
17274 079/300 Read block 2484 len 20480
17275 080/300 Write block 4898 len 4096 data: 2
17276 081/300 Read block 3167 len 4096
17277 082/300 Write block 1157 len 12288 data: 4 4 5
17278 083/300 Read block 1244 len 24576
17279 084/300 Flush
17280 085/300 Read block 3341 len 28672
17281 086/300 Write block 2952 len 36864 data: 5 5 5 4 4 4 3 2 2
17282 087/300 Read block 3845 len 12288
17283 088/300 Write block 4652 len 20480 data: 3 4 4 4 4
17284 089/300 Read block 4889 len 12288
17285 090/300 Write block 4535 len 28672 data: 7 7 6 5 5 4 5
17286 091/300 Write block 4334 len 4096 data: 4
17287 092/300 Read block 3502 len 28672
17288 093/300 Write block 4661 len 32768 data: 5 4 4 3 3 2 3 3
17289 094/300 Read block 2154 len 8192
17290 095/300 Flush
17291 096/300 Read block 3141 len 40960
17292 097/300 Flush
17293 098/300 Write block 4889 len 8192 data: 3 3
17294 099/300 Write block 3319 len 28672 data: 5 6 5 5 5 4 4
17295 100/300 Read block 4 len 36864
17296 101/300 Read block 145 len 40960
17297 102/300 Write block 2492 len 8192 data: 2 3
17298 103/300 Write block 111 len 24576 data: 5 4 3 3 3 3
17299 104/300 Read block 2409 len 24576
17300 105/300 Read block 38 len 32768
17301 106/300 Read block 2187 len 12288
17302 107/300 Write block 4930 len 32768 data: 5 5 3 4 5 5 5 5
17303 108/300 Write block 3441 len 12288 data: 3 3 3
17304 109/300 Read block 53 len 28672
17305 110/300 Flush
17306 111/300 Read block 4211 len 20480
17307 112/300 Write block 2617 len 36864 data: 4 4 3 4 4 3 3 2 2
17308 113/300 Write block 1827 len 4096 data: 3
17309 114/300 Flush
17310 115/300 Read block 2134 len 4096
17311 116/300 Flush
17312 117/300 Write block 3356 len 32768 data: 3 5 5 4 3 4 5 5
17313 118/300 Write block 1902 len 20480 data: 5 4 5 4 4
17314 119/300 Read block 2207 len 40960
17315 120/300 Read block 3223 len 8192
17316 121/300 Write block 3175 len 24576 data: 6 6 6 7 7 7
17317 122/300 Write block 1963 len 28672 data: 4 3 2 2 2 2 3
17318 123/300 Read block 573 len 12288
17319 124/300 Read block 2576 len 4096
17320 125/300 Write block 501 len 8192 data: 2 2
17321 126/300 Read block 3407 len 20480
17322 127/300 Read block 190 len 16384
17323 128/300 Write block 4691 len 12288 data: 5 4 4
17324 129/300 Write block 2515 len 20480 data: 3 4 5 5 5
17325 130/300 Read block 3191 len 32768
17326 131/300 Flush
17327 132/300 Write block 641 len 20480 data: 4 3 4 4 5
17328 133/300 Read block 1328 len 8192
17329 134/300 Read block 4425 len 4096
17330 135/300 Read block 3329 len 28672
17331 136/300 Read block 4553 len 28672
17332 137/300 Read block 3964 len 16384
17333 138/300 Write block 3847 len 16384 data: 4 3 2 2
17334 139/300 Read block 4082 len 28672
17335 140/300 Write block 1075 len 8192 data: 5 5
17336 141/300 Write block 4036 len 24576 data: 4 4 4 4 4 5
17337 142/300 Flush
17338 143/300 Read block 2697 len 20480
17339 144/300 Write block 1319 len 4096 data: 4
17340 145/300 Write block 4604 len 16384 data: 6 6 5 4
17341 146/300 Write block 4670 len 20480 data: 3 4 3 4 3
17342 147/300 Read block 3285 len 8192
17343 148/300 Read block 1096 len 4096
17344 149/300 Write block 3544 len 24576 data: 5 6 7 6 6 6
17345 150/300 Write block 2382 len 4096 data: 3
17346 151/300 Read block 2895 len 36864
17347 152/300 Read block 2400 len 20480
17348 153/300 Write block 1714 len 16384 data: 2 2 2 2
17349 154/300 Read block 3080 len 32768
17350 155/300 Read block 2938 len 24576
17351 156/300 Read block 720 len 20480
17352 157/300 Write block 4156 len 16384 data: 3 4 4 3
17353 158/300 Read block 789 len 8192
17354 159/300 Write block 3212 len 16384 data: 9 8 8 8
17355 160/300 Read block 752 len 40960
17356 161/300 Write block 230 len 32768 data: 6 7 7 6 6 5 4 5
17357 162/300 Read block 1694 len 4096
17358 163/300 Read block 837 len 28672
17359 164/300 Read block 1551 len 4096
17360 165/300 Read block 2801 len 28672
17361 166/300 Write block 3457 len 20480 data: 2 2 2 2 2
17362 167/300 Write block 2050 len 12288 data: 6 4 4
17363 168/300 Read block 609 len 40960
17364 169/300 Read block 2689 len 24576
17365 170/300 Read block 71 len 32768
17366 171/300 Write block 855 len 12288 data: 4 4 3
17367 172/300 Read block 1106 len 4096
17368 173/300 Read block 1896 len 28672
17369 174/300 Read block 3272 len 16384
17370 175/300 Read block 4935 len 36864
17371 176/300 Write block 3430 len 32768 data: 4 3 4 6 7 5 4 3
17372 177/300 Read block 1270 len 8192
17373 178/300 Write block 992 len 8192 data: 4 4
17374 179/300 Read block 2800 len 24576
17375 180/300 Write block 267 len 32768 data: 7 6 6 5 5 4 4 4
17376 181/300 Write block 1254 len 40960 data: 3 3 3 3 4 5 4 5 6 4
17377 182/300 Flush
17378 183/300 Read block 1154 len 28672
17379 184/300 Write block 597 len 40960 data: 2 2 2 2 2 3 3 3 3 3
17380 185/300 Read block 3869 len 24576
17381 186/300 Write block 961 len 28672 data: 3 3 2 2 4 4 4
17382 187/300 Write block 2485 len 16384 data: 6 6 5 5
17383 188/300 Read block 3361 len 28672
17384 189/300 Write block 1079 len 16384 data: 4 4 3 3
17385 190/300 Flush
17386 191/300 Write block 2728 len 16384 data: 4 6 6 6
17387 192/300 Write block 4969 len 24576 data: 4 4 4 4 4 3
17388 193/300 Write block 4537 len 40960 data: 7 6 6 5 6 4 3 4 4 3
17389 194/300 Read block 2531 len 12288
17390 195/300 Write block 3534 len 40960 data: 2 2 2 2 2 2 2 3 3 4
17391 196/300 Read block 1112 len 32768
17392 197/300 Read block 3053 len 12288
17393 198/300 Read block 2246 len 8192
17394 199/300 Write block 534 len 8192 data: 7 7
17395 200/300 Write block 1314 len 4096 data: 4
17396 201/300 Read block 3728 len 12288
17397 202/300 Write block 3354 len 40960 data: 3 3 4 6 6 5 4 5 6 6
17398 203/300 Write block 2397 len 16384 data: 2 2 3 5
17399 204/300 Read block 850 len 20480
17400 205/300 Read block 4483 len 24576
17401 206/300 Read block 3098 len 32768
17402 207/300 Write block 2385 len 8192 data: 2 2
17403 208/300 Write block 4672 len 24576 data: 4 5 4 3 3 3
17404 209/300 Read block 1657 len 24576
17405 210/300 Read block 2945 len 40960
17406 211/300 Read block 1339 len 32768
17407 212/300 Read block 4850 len 20480
17408 213/300 Flush
17409 214/300 Write block 1508 len 12288 data: 5 5 4
17410 215/300 Write block 371 len 20480 data: 4 3 3 2 3
17411 216/300 Read block 2680 len 24576
17412 217/300 Read block 3487 len 36864
17413 218/300 Read block 3623 len 16384
17414 219/300 Write block 782 len 20480 data: 6 6 6 5 3
17415 220/300 Read block 1789 len 12288
17416 221/300 Read block 3496 len 4096
17417 222/300 Read block 3884 len 16384
17418 223/300 Write block 723 len 36864 data: 5 5 5 6 5 6 5 4 3
17419 224/300 Read block 2221 len 12288
17420 225/300 Read block 141 len 40960
17421 226/300 Read block 4062 len 4096
17422 227/300 Write block 2578 len 16384 data: 7 7 7 6
17423 228/300 Flush
17424 229/300 Write block 3263 len 28672 data: 3 4 3 3 3 3 3
17425 230/300 Read block 4205 len 28672
17426 231/300 Read block 4469 len 36864
17427 232/300 Read block 88 len 36864
17428 233/300 Write block 3427 len 12288 data: 4 4 4
17429 234/300 Read block 3972 len 4096
17430 235/300 Write block 2901 len 8192 data: 2 5
17431 236/300 Write block 3906 len 32768 data: 3 5 5 5 6 6 6 6
17432 237/300 Write block 4381 len 8192 data: 4 2
17433 238/300 Read block 1108 len 28672
17434 239/300 Read block 3005 len 40960
17435 240/300 Write block 1743 len 16384 data: 4 5 5 5
17436 241/300 Write block 3386 len 4096 data: 6
17437 242/300 Read block 3141 len 36864
17438 243/300 Read block 3548 len 12288
17439 244/300 Read block 4114 len 4096
17440 245/300 Read block 3959 len 40960
17441 246/300 Read block 4293 len 16384
17442 247/300 Read block 4690 len 20480
17443 248/300 Write block 340 len 16384 data: 5 6 6 5
17444 249/300 Write block 3797 len 16384 data: 3 3 3 3
17445 250/300 Write block 4199 len 8192 data: 3 3
17446 251/300 Read block 4229 len 32768
17447 252/300 Read block 1568 len 16384
17448 253/300 Read block 3032 len 4096
17449 254/300 Read block 3341 len 32768
17450 255/300 Read block 438 len 4096
17451 256/300 Write block 1206 len 4096 data: 3
17452 257/300 Write block 362 len 28672 data: 3 3 3 3 3 3 3
17453 258/300 Flush
17454 259/300 Flush
17455 260/300 Read block 3006 len 12288
17456 261/300 Read block 4765 len 20480
17457 262/300 Write block 3256 len 20480 data: 3 2 2 2 2
17458 263/300 Read block 1537 len 16384
17459 264/300 Write block 3082 len 8192 data: 3 3
17460 265/300 Read block 3265 len 16384
17461 266/300 Read block 579 len 36864
17462 267/300 Write block 4339 len 36864 data: 5 6 5 5 4 7 7 8 6
17463 268/300 Write block 4933 len 12288 data: 5 6 6
17464 269/300 Read block 3820 len 28672
17465 270/300 Read block 1925 len 24576
17466 271/300 Read block 1368 len 28672
17467 272/300 Write block 3566 len 16384 data: 3 3 4 5
17468 273/300 Write block 1385 len 32768 data: 3 3 3 4 4 3 4 5
17469 274/300 Read block 3600 len 40960
17470 275/300 Write block 4263 len 40960 data: 7 7 8 8 7 5 4 3 4 3
17471 276/300 Read block 862 len 36864
17472 277/300 Write block 465 len 16384 data: 2 2 2 4
17473 278/300 Read block 2864 len 28672
17474 279/300 Read block 1239 len 20480
17475 280/300 Flush
17476 281/300 Write block 3661 len 36864 data: 3 4 3 2 3 2 2 3 4
17477 282/300 Read block 3624 len 40960
17478 283/300 Write block 2569 len 20480 data: 4 4 5 5 5
17479 284/300 Write block 1668 len 28672 data: 3 3 2 2 3 3 3
17480 285/300 Write block 4016 len 32768 data: 5 5 4 4 4 4 4 5
17481 286/300 Read block 698 len 16384
17482 287/300 Write block 4370 len 28672 data: 4 4 4 4 4 4 3
17483 288/300 Read block 4534 len 8192
17484 289/300 Write block 1991 len 32768 data: 3 3 2 2 2 2 2 4
17485 290/300 Write block 4055 len 36864 data: 3 3 3 4 4 4 3 3 3
17486 291/300 Flush
17487 292/300 Write block 4373 len 24576 data: 5 5 5 4 3 3
17488 293/300 Read block 1693 len 20480
17489 294/300 Read block 4816 len 4096
17490 295/300 Write block 3924 len 40960 data: 3 3 4 4 3 3 4 4 5 5
17491 296/300 Read block 4135 len 4096
17492 297/300 Flush
17493 298/300 Read block 1715 len 36864
17494 299/300 Read block 4159 len 20480
17495 300/300 Read block 1673 len 28672
17496 301/300 Read block 1288 len 4096
17497 Main task has work to do, go find it
17498 got dsc Start(0)
17499 start 0
17500 Mar 17 22:38:28.663 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/0, method: POST, req_id: a2aca7bb-b943-4716-b3cd-4e276a3daf52, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
17501 Replay: started 0, returned:Ok(())
17502 ----------------------------------------------------------------
17503 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17504 States: Offline Active Active
17505 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17506 5228 Acked Read 5 Sent Done Done false
17507 5229 Acked Write 10 Sent Done Done false
17508 5230 Acked Read 6 Sent Done Done false
17509 5231 Acked Write 3 Sent Done Done false
17510 5232 Acked Read 3 Sent Done Done false
17511 5233 Acked Flush 0 Sent Done Done false
17512 5234 Acked Read 8 Sent Done Done false
17513 5235 Acked Read 1 Sent Done Done false
17514 5236 Acked Read 8 Sent Done Done false
17515 5237 Acked Read 1 Sent Done Done false
17516 5238 Acked Flush 0 Sent Done Done false
17517 5239 Acked Read 8 Sent Done Done false
17518 5240 Acked Flush 0 Sent Done Done false
17519 5241 Acked Read 8 Sent Done Done false
17520 5242 Acked Read 3 Sent Done Done false
17521 5243 Acked Flush 0 Sent Done Done false
17522 5244 Acked Read 8 Sent Done Done false
17523 5245 Acked Read 7 Sent Done Done false
17524 5246 Acked Flush 0 Sent Done Done false
17525 5247 Acked Write 3 Sent Done Done false
17526 5248 Acked Write 10 Sent Done Done false
17527 5249 Acked Read 1 Sent Done Done false
17528 5250 Acked Flush 0 Sent Done Done false
17529 5251 Acked Read 10 Sent Done Done false
17530 5252 Acked Write 4 Sent Done Done false
17531 5253 Acked Write 6 Sent Done Done false
17532 5254 Acked Write 5 Sent Done Done false
17533 5255 Acked Write 1 Sent Done Done false
17534 5256 Acked Read 8 Sent Done Done false
17535 5257 Acked Write 6 Sent Done Done false
17536 5258 Acked Read 5 Sent Done Done false
17537 5259 Acked Read 9 Sent Done Done false
17538 5260 Acked Read 7 Sent Done Done false
17539 5261 Acked Read 3 Sent Done Done false
17540 5262 Acked Read 8 Sent Done Done false
17541 5263 Acked Read 7 Sent Done Done false
17542 5264 Acked Read 8 Sent Done Done false
17543 5265 Acked Read 1 Sent Done Done false
17544 5266 Acked Write 3 Sent Done Done false
17545 5267 Acked Read 9 Sent Done Done false
17546 5268 Acked Read 10 Sent Done Done false
17547 5269 Acked Write 10 Sent Done Done false
17548 5270 Acked Read 8 Sent Done Done false
17549 5271 Acked Read 10 Sent Done Done false
17550 5272 Acked Write 10 Sent Done Done false
17551 5273 Acked Write 7 Sent Done Done false
17552 5274 Acked Write 10 Sent Done Done false
17553 5275 Acked Read 4 Sent Done Done false
17554 5276 Acked Write 10 Sent Done Done false
17555 5277 Acked Write 3 Sent Done Done false
17556 5278 Acked Flush 0 Sent Done Done false
17557 5279 Acked Read 5 Sent Done Done false
17558 5280 Acked Write 1 Sent Done Done false
17559 5281 Acked Flush 0 Sent Done Done false
17560 5282 Acked Write 6 Sent Done Done false
17561 5283 Acked Read 1 Sent Done Done false
17562 5284 Acked Write 2 Sent Done Done false
17563 5285 Acked Read 8 Sent Done Done false
17564 5286 Acked Flush 0 Sent Done Done false
17565 5287 Acked Write 9 Sent Done Done false
17566 5288 Acked Read 10 Sent Done Done false
17567 5289 Acked Read 3 Sent Done Done false
17568 5290 Acked Read 2 Sent Done Done false
17569 5291 Acked Write 8 Sent Done Done false
17570 5292 Acked Write 9 Sent Done Done false
17571 5293 Acked Write 3 Sent Done Done false
17572 5294 Acked Read 9 Sent Done Done false
17573 5295 Acked Read 5 Sent Done Done false
17574 5296 Acked Write 1 Sent Done Done false
17575 5297 Acked Flush 0 Sent Done Done false
17576 5298 Acked Write 7 Sent Done Done false
17577 5299 Acked Write 2 Sent Done Done false
17578 5300 Acked Read 2 Sent Done Done false
17579 5301 Acked Read 6 Sent Done Done false
17580 5302 Acked Write 2 Sent Done Done false
17581 5303 Acked Read 6 Sent Done Done false
17582 5304 Acked Read 6 Sent Done Done false
17583 5305 Acked Write 7 Sent Done Done false
17584 5306 Acked Read 5 Sent Done Done false
17585 5307 Acked Write 1 Sent Done Done false
17586 5308 Acked Read 1 Sent Done Done false
17587 5309 Acked Write 3 Sent Done Done false
17588 5310 Acked Read 6 Sent Done Done false
17589 5311 Acked Flush 0 Sent Done Done false
17590 5312 Acked Read 7 Sent Done Done false
17591 5313 Acked Write 9 Sent Done Done false
17592 5314 Acked Read 3 Sent Done Done false
17593 5315 Acked Write 5 Sent Done Done false
17594 5316 Acked Read 3 Sent Done Done false
17595 5317 Acked Write 7 Sent Done Done false
17596 5318 Acked Write 1 Sent Done Done false
17597 5319 Acked Read 7 Sent Done Done false
17598 5320 Acked Write 8 Sent Done Done false
17599 5321 Acked Read 2 Sent Done Done false
17600 5322 Acked Flush 0 Sent Done Done false
17601 5323 Acked Read 10 Sent Done Done false
17602 5324 Acked Flush 0 Sent Done Done false
17603 5325 Acked Write 2 Sent Done Done false
17604 5326 Acked Write 7 Sent Done Done false
17605 5327 Acked Read 9 Sent Done Done false
17606 5328 Acked Read 10 Sent Done Done false
17607 5329 Acked Write 2 Sent Done Done false
17608 5330 Acked Write 6 Sent Done Done false
17609 5331 Acked Read 6 Sent Done Done false
17610 5332 Acked Read 8 Sent Done Done false
17611 5333 Acked Read 3 Sent Done Done false
17612 5334 Acked Write 8 Sent Done Done false
17613 5335 Acked Write 3 Sent Done Done false
17614 5336 Acked Read 7 Sent Done Done false
17615 5337 Acked Flush 0 Sent Done Done false
17616 5338 Acked Read 5 Sent Done Done false
17617 5339 Acked Write 9 Sent Done Done false
17618 5340 Acked Write 1 Sent Done Done false
17619 5341 Acked Flush 0 Sent Done Done false
17620 5342 Acked Read 1 Sent Done Done false
17621 5343 Acked Flush 0 Sent Done Done false
17622 5344 Acked Write 8 Sent Done Done false
17623 5345 Acked Write 5 Sent Done Done false
17624 5346 Acked Read 10 Sent Done Done false
17625 5347 Acked Read 2 Sent Done Done false
17626 5348 Acked Write 6 Sent Done Done false
17627 5349 Acked Write 7 Sent Done Done false
17628 5350 Acked Read 3 Sent Done Done false
17629 5351 Acked Read 1 Sent Done Done false
17630 5352 Acked Write 2 Sent Done Done false
17631 5353 Acked Read 5 Sent Done Done false
17632 5354 Acked Read 4 Sent Done Done false
17633 5355 Acked Write 3 Sent Done Done false
17634 5356 Acked Write 5 Sent Done Done false
17635 5357 Acked Read 8 Sent Done Done false
17636 5358 Acked Flush 0 Sent Done Done false
17637 5359 Acked Write 5 Sent Done Done false
17638 5360 Acked Read 2 Sent Done Done false
17639 5361 Acked Read 1 Sent Done Done false
17640 5362 Acked Read 7 Sent Done Done false
17641 5363 Acked Read 7 Sent Done Done false
17642 5364 Acked Read 4 Sent Done Done false
17643 5365 Acked Write 4 Sent Done Done false
17644 5366 Acked Read 7 Sent Done Done false
17645 5367 Acked Write 2 Sent Done Done false
17646 5368 Acked Write 6 Sent Done Done false
17647 5369 Acked Flush 0 Sent Done Done false
17648 5370 Acked Read 5 Sent Done Done false
17649 5371 Acked Write 1 Sent Done Done false
17650 5372 Acked Write 4 Sent Done Done false
17651 5373 Acked Write 5 Sent Done Done false
17652 5374 Acked Read 2 Sent Done Done false
17653 5375 Acked Read 1 Sent Done Done false
17654 5376 Acked Write 6 Sent Done Done false
17655 5377 Acked Write 1 Sent Done Done false
17656 5378 Acked Read 9 Sent Done Done false
17657 5379 Acked Read 5 Sent Done Done false
17658 5380 Acked Write 4 Sent Done Done false
17659 5381 Acked Read 8 Sent Done Done false
17660 5382 Acked Read 6 Sent Done Done false
17661 5383 Acked Read 5 Sent Done Done false
17662 5384 Acked Write 4 Sent Done Done false
17663 5385 Acked Read 2 Sent Done Done false
17664 5386 Acked Write 4 Sent Done Done false
17665 5387 Acked Read 10 Sent Done Done false
17666 5388 Acked Write 8 Sent Done Done false
17667 5389 Acked Read 1 Sent Done Done false
17668 5390 Acked Read 7 Sent Done Done false
17669 5391 Acked Read 1 Sent Done Done false
17670 5392 Acked Read 7 Sent Done Done false
17671 5393 Acked Write 5 Sent Done Done false
17672 5394 Acked Write 3 Sent Done Done false
17673 5395 Acked Read 10 Sent Done Done false
17674 5396 Acked Read 6 Sent Done Done false
17675 5397 Acked Read 8 Sent Done Done false
17676 5398 Acked Write 3 Sent Done Done false
17677 5399 Acked Read 1 Sent Done Done false
17678 5400 Acked Read 7 Sent Done Done false
17679 5401 Acked Read 4 Sent Done Done false
17680 5402 Acked Read 9 Sent Done Done false
17681 5403 Acked Write 8 Sent Done Done false
17682 5404 Acked Read 2 Sent Done Done false
17683 5405 Acked Write 2 Sent Done Done false
17684 5406 Acked Read 6 Sent Done Done false
17685 5407 Acked Write 8 Sent Done Done false
17686 5408 Acked Write 10 Sent Done Done false
17687 5409 Acked Flush 0 Sent Done Done false
17688 5410 Acked Read 7 Sent Done Done false
17689 5411 Acked Write 10 Sent Done Done false
17690 5412 Acked Read 6 Sent Done Done false
17691 5413 Acked Write 7 Sent Done Done false
17692 5414 Acked Write 4 Sent Done Done false
17693 5415 Acked Read 7 Sent Done Done false
17694 5416 Acked Write 4 Sent Done Done false
17695 5417 Acked Flush 0 Sent Done Done false
17696 5418 Acked Write 4 Sent Done Done false
17697 5419 Acked Write 6 Sent Done Done false
17698 5420 Acked Write 10 Sent Done Done false
17699 5421 Acked Read 3 Sent Done Done false
17700 5422 Acked Write 10 Sent Done Done false
17701 5423 Acked Read 8 Sent Done Done false
17702 5424 Acked Read 3 Sent Done Done false
17703 5425 Acked Read 2 Sent Done Done false
17704 5426 Acked Write 2 Sent Done Done false
17705 5427 Acked Write 1 Sent Done Done false
17706 5428 Acked Read 3 Sent Done Done false
17707 5429 Acked Write 10 Sent Done Done false
17708 5430 Acked Write 4 Sent Done Done false
17709 5431 Acked Read 5 Sent Done Done false
17710 5432 Acked Read 6 Sent Done Done false
17711 5433 Acked Read 8 Sent Done Done false
17712 5434 Acked Write 2 Sent Done Done false
17713 5435 Acked Write 6 Sent Done Done false
17714 5436 Acked Read 6 Sent Done Done false
17715 5437 Acked Read 10 Sent Done Done false
17716 5438 Acked Read 8 Sent Done Done false
17717 5439 Acked Read 5 Sent Done Done false
17718 5440 Acked Flush 0 Sent Done Done false
17719 5441 Acked Write 3 Sent Done Done false
17720 5442 Acked Write 5 Sent Done Done false
17721 5443 Acked Read 6 Sent Done Done false
17722 5444 Acked Read 9 Sent Done Done false
17723 5445 Acked Read 4 Sent Done Done false
17724 5446 Acked Write 5 Sent Done Done false
17725 5447 Acked Read 3 Sent Done Done false
17726 5448 Acked Read 1 Sent Done Done false
17727 5449 Acked Read 4 Sent Done Done false
17728 5450 Acked Write 9 Sent Done Done false
17729 5451 Acked Read 3 Sent Done Done false
17730 5452 Acked Read 10 Sent Done Done false
17731 5453 Acked Read 1 Sent Done Done false
17732 5454 Acked Write 4 Sent Done Done false
17733 5455 Acked Flush 0 Sent Done Done false
17734 5456 Acked Write 7 Sent Done Done false
17735 5457 Acked Read 7 Sent Done Done false
17736 5458 Acked Read 9 Sent Done Done false
17737 5459 Acked Read 9 Sent Done Done false
17738 5460 Acked Write 3 Sent Done Done false
17739 5461 Acked Read 1 Sent Done Done false
17740 5462 Acked Write 2 Sent Done Done false
17741 5463 Acked Write 8 Sent Done Done false
17742 5464 Acked Write 2 Sent Done Done false
17743 5465 Acked Read 7 Sent Done Done false
17744 5466 Acked Read 10 Sent Done Done false
17745 5467 Acked Write 4 Sent Done Done false
17746 5468 Acked Write 1 Sent Done Done false
17747 5469 Acked Read 9 Sent Done Done false
17748 5470 Acked Read 3 Sent Done Done false
17749 5471 Acked Read 1 Sent Done Done false
17750 5472 Acked Read 10 Sent Done Done false
17751 5473 Acked Read 4 Sent Done Done false
17752 5474 Acked Read 5 Sent Done Done false
17753 5475 Acked Write 4 Sent Done Done false
17754 5476 Acked Write 4 Sent Done Done false
17755 5477 Acked Write 2 Sent Done Done false
17756 5478 Acked Read 8 Sent Done Done false
17757 5479 Acked Read 4 Sent Done Done false
17758 5480 Acked Read 1 Sent Done Done false
17759 5481 Acked Read 8 Sent Done Done false
17760 5482 Acked Read 1 Sent Done Done false
17761 5483 Acked Write 1 Sent Done Done false
17762 5484 Acked Write 7 Sent Done Done false
17763 5485 Acked Flush 0 Sent Done Done false
17764 5486 Acked Flush 0 Sent Done Done false
17765 5487 Acked Read 3 Sent Done Done false
17766 5488 Acked Read 5 Sent Done Done false
17767 5489 Acked Write 5 Sent Done Done false
17768 5490 Acked Read 4 Sent Done Done false
17769 5491 Acked Write 2 Sent Done Done false
17770 5492 Acked Read 4 Sent Done Done false
17771 5493 Acked Read 9 Sent Done Done false
17772 5494 Acked Write 9 Sent Done Done false
17773 5495 Acked Write 3 Sent Done Done false
17774 5496 Acked Read 7 Sent Done Done false
17775 5497 Acked Read 6 Sent Done Done false
17776 5498 Acked Read 7 Sent Done Done false
17777 5499 Acked Write 4 Sent Done Done false
17778 5500 Acked Write 8 Sent Done Done false
17779 5501 Acked Read 10 Sent Done Done false
17780 5502 Acked Write 10 Sent Done Done false
17781 5503 Acked Read 9 Sent Done Done false
17782 5504 Acked Write 4 Sent Done Done false
17783 5505 Acked Read 7 Sent Done Done false
17784 5506 Acked Read 5 Sent Done Done false
17785 5507 Acked Flush 0 Sent Done Done false
17786 5508 Acked Write 9 Sent Done Done false
17787 5509 Acked Read 10 Sent Done Done false
17788 5510 Acked Write 5 Sent Done Done false
17789 5511 Acked Write 7 Sent Done Done false
17790 5512 Acked Write 8 Sent Done Done false
17791 5513 Acked Read 4 Sent Done Done false
17792 5514 Acked Write 7 Sent Done Done false
17793 5515 Acked Read 2 Sent Done Done false
17794 5516 Acked Write 8 Sent Done Done false
17795 5517 Acked Write 9 Sent Done Done false
17796 5518 Acked Flush 0 Sent Done Done false
17797 5519 Acked Write 6 Sent Done Done false
17798 5520 Acked Read 5 Sent Done Done false
17799 5521 Acked Read 1 Sent Done Done false
17800 5522 Acked Write 10 Sent Done Done false
17801 5523 Acked Read 1 Sent Done Done false
17802 5524 Acked Flush 0 Sent Done Done false
17803 5525 Acked Read 9 Sent Done Done false
17804 5526 Acked Read 5 Sent Done Done false
17805 5527 Acked Read 7 Sent Done Done false
17806 5528 Acked Read 1 Sent Done Done false
17807 STATES DS:0 DS:1 DS:2 TOTAL
17808 Sent 301 0 0 301
17809 Done 0 301 301 602
17810 Skipped 0 0 0 0
17811 Error 0 0 0 0
17812 Last Flush: Some(JobId(5227)) Some(JobId(5524)) Some(JobId(5524))
17813 Downstairs last five completed: 5227 5226 5225 5224 5223
17814 Upstairs last five completed: 5528 5527 5526 5525 5524
17815 CLIENT: Up:0 ds:301 act:2
178162025-03-17T22:38:29.165ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
178172025-03-17T22:38:29.682ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
17818 [8810] got start action
17819 [8810] I am going to restart
17820 Starting downstairs at port 8810
17821 Make output file at "/tmp/dsc/downstairs-8810.txt"
17822 [8810][0] reports Starting
17823 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1223)
17824 [8810][0] reports Running
178252025-03-17T22:38:30.684ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
178262025-03-17T22:38:31.685ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
17827 ----------------------------------------------------------------
17828 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17829 States: Offline Active Active
17830 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17831 5228 Acked Read 5 Sent Done Done false
17832 5229 Acked Write 10 Sent Done Done false
17833 5230 Acked Read 6 Sent Done Done false
17834 5231 Acked Write 3 Sent Done Done false
17835 5232 Acked Read 3 Sent Done Done false
17836 5233 Acked Flush 0 Sent Done Done false
17837 5234 Acked Read 8 Sent Done Done false
17838 5235 Acked Read 1 Sent Done Done false
17839 5236 Acked Read 8 Sent Done Done false
17840 5237 Acked Read 1 Sent Done Done false
17841 5238 Acked Flush 0 Sent Done Done false
17842 5239 Acked Read 8 Sent Done Done false
17843 5240 Acked Flush 0 Sent Done Done false
17844 5241 Acked Read 8 Sent Done Done false
17845 5242 Acked Read 3 Sent Done Done false
17846 5243 Acked Flush 0 Sent Done Done false
17847 5244 Acked Read 8 Sent Done Done false
17848 5245 Acked Read 7 Sent Done Done false
17849 5246 Acked Flush 0 Sent Done Done false
17850 5247 Acked Write 3 Sent Done Done false
17851 5248 Acked Write 10 Sent Done Done false
17852 5249 Acked Read 1 Sent Done Done false
17853 5250 Acked Flush 0 Sent Done Done false
17854 5251 Acked Read 10 Sent Done Done false
17855 5252 Acked Write 4 Sent Done Done false
17856 5253 Acked Write 6 Sent Done Done false
17857 5254 Acked Write 5 Sent Done Done false
17858 5255 Acked Write 1 Sent Done Done false
17859 5256 Acked Read 8 Sent Done Done false
17860 5257 Acked Write 6 Sent Done Done false
17861 5258 Acked Read 5 Sent Done Done false
17862 5259 Acked Read 9 Sent Done Done false
17863 5260 Acked Read 7 Sent Done Done false
17864 5261 Acked Read 3 Sent Done Done false
17865 5262 Acked Read 8 Sent Done Done false
17866 5263 Acked Read 7 Sent Done Done false
17867 5264 Acked Read 8 Sent Done Done false
17868 5265 Acked Read 1 Sent Done Done false
17869 5266 Acked Write 3 Sent Done Done false
17870 5267 Acked Read 9 Sent Done Done false
17871 5268 Acked Read 10 Sent Done Done false
17872 5269 Acked Write 10 Sent Done Done false
17873 5270 Acked Read 8 Sent Done Done false
17874 5271 Acked Read 10 Sent Done Done false
17875 5272 Acked Write 10 Sent Done Done false
17876 5273 Acked Write 7 Sent Done Done false
17877 5274 Acked Write 10 Sent Done Done false
17878 5275 Acked Read 4 Sent Done Done false
17879 5276 Acked Write 10 Sent Done Done false
17880 5277 Acked Write 3 Sent Done Done false
17881 5278 Acked Flush 0 Sent Done Done false
17882 5279 Acked Read 5 Sent Done Done false
17883 5280 Acked Write 1 Sent Done Done false
17884 5281 Acked Flush 0 Sent Done Done false
17885 5282 Acked Write 6 Sent Done Done false
17886 5283 Acked Read 1 Sent Done Done false
17887 5284 Acked Write 2 Sent Done Done false
17888 5285 Acked Read 8 Sent Done Done false
17889 5286 Acked Flush 0 Sent Done Done false
17890 5287 Acked Write 9 Sent Done Done false
17891 5288 Acked Read 10 Sent Done Done false
17892 5289 Acked Read 3 Sent Done Done false
17893 5290 Acked Read 2 Sent Done Done false
17894 5291 Acked Write 8 Sent Done Done false
17895 5292 Acked Write 9 Sent Done Done false
17896 5293 Acked Write 3 Sent Done Done false
17897 5294 Acked Read 9 Sent Done Done false
17898 5295 Acked Read 5 Sent Done Done false
17899 5296 Acked Write 1 Sent Done Done false
17900 5297 Acked Flush 0 Sent Done Done false
17901 5298 Acked Write 7 Sent Done Done false
17902 5299 Acked Write 2 Sent Done Done false
17903 5300 Acked Read 2 Sent Done Done false
17904 5301 Acked Read 6 Sent Done Done false
17905 5302 Acked Write 2 Sent Done Done false
17906 5303 Acked Read 6 Sent Done Done false
17907 5304 Acked Read 6 Sent Done Done false
17908 5305 Acked Write 7 Sent Done Done false
17909 5306 Acked Read 5 Sent Done Done false
17910 5307 Acked Write 1 Sent Done Done false
17911 5308 Acked Read 1 Sent Done Done false
17912 5309 Acked Write 3 Sent Done Done false
17913 5310 Acked Read 6 Sent Done Done false
17914 5311 Acked Flush 0 Sent Done Done false
17915 5312 Acked Read 7 Sent Done Done false
17916 5313 Acked Write 9 Sent Done Done false
17917 5314 Acked Read 3 Sent Done Done false
17918 5315 Acked Write 5 Sent Done Done false
17919 5316 Acked Read 3 Sent Done Done false
17920 5317 Acked Write 7 Sent Done Done false
17921 5318 Acked Write 1 Sent Done Done false
17922 5319 Acked Read 7 Sent Done Done false
17923 5320 Acked Write 8 Sent Done Done false
17924 5321 Acked Read 2 Sent Done Done false
17925 5322 Acked Flush 0 Sent Done Done false
17926 5323 Acked Read 10 Sent Done Done false
17927 5324 Acked Flush 0 Sent Done Done false
17928 5325 Acked Write 2 Sent Done Done false
17929 5326 Acked Write 7 Sent Done Done false
17930 5327 Acked Read 9 Sent Done Done false
17931 5328 Acked Read 10 Sent Done Done false
17932 5329 Acked Write 2 Sent Done Done false
17933 5330 Acked Write 6 Sent Done Done false
17934 5331 Acked Read 6 Sent Done Done false
17935 5332 Acked Read 8 Sent Done Done false
17936 5333 Acked Read 3 Sent Done Done false
17937 5334 Acked Write 8 Sent Done Done false
17938 5335 Acked Write 3 Sent Done Done false
17939 5336 Acked Read 7 Sent Done Done false
17940 5337 Acked Flush 0 Sent Done Done false
17941 5338 Acked Read 5 Sent Done Done false
17942 5339 Acked Write 9 Sent Done Done false
17943 5340 Acked Write 1 Sent Done Done false
17944 5341 Acked Flush 0 Sent Done Done false
17945 5342 Acked Read 1 Sent Done Done false
17946 5343 Acked Flush 0 Sent Done Done false
17947 5344 Acked Write 8 Sent Done Done false
17948 5345 Acked Write 5 Sent Done Done false
17949 5346 Acked Read 10 Sent Done Done false
17950 5347 Acked Read 2 Sent Done Done false
17951 5348 Acked Write 6 Sent Done Done false
17952 5349 Acked Write 7 Sent Done Done false
17953 5350 Acked Read 3 Sent Done Done false
17954 5351 Acked Read 1 Sent Done Done false
17955 5352 Acked Write 2 Sent Done Done false
17956 5353 Acked Read 5 Sent Done Done false
17957 5354 Acked Read 4 Sent Done Done false
17958 5355 Acked Write 3 Sent Done Done false
17959 5356 Acked Write 5 Sent Done Done false
17960 5357 Acked Read 8 Sent Done Done false
17961 5358 Acked Flush 0 Sent Done Done false
17962 5359 Acked Write 5 Sent Done Done false
17963 5360 Acked Read 2 Sent Done Done false
17964 5361 Acked Read 1 Sent Done Done false
17965 5362 Acked Read 7 Sent Done Done false
17966 5363 Acked Read 7 Sent Done Done false
17967 5364 Acked Read 4 Sent Done Done false
17968 5365 Acked Write 4 Sent Done Done false
17969 5366 Acked Read 7 Sent Done Done false
17970 5367 Acked Write 2 Sent Done Done false
17971 5368 Acked Write 6 Sent Done Done false
17972 5369 Acked Flush 0 Sent Done Done false
17973 5370 Acked Read 5 Sent Done Done false
17974 5371 Acked Write 1 Sent Done Done false
17975 5372 Acked Write 4 Sent Done Done false
17976 5373 Acked Write 5 Sent Done Done false
17977 5374 Acked Read 2 Sent Done Done false
17978 5375 Acked Read 1 Sent Done Done false
17979 5376 Acked Write 6 Sent Done Done false
17980 5377 Acked Write 1 Sent Done Done false
17981 5378 Acked Read 9 Sent Done Done false
17982 5379 Acked Read 5 Sent Done Done false
17983 5380 Acked Write 4 Sent Done Done false
17984 5381 Acked Read 8 Sent Done Done false
17985 5382 Acked Read 6 Sent Done Done false
17986 5383 Acked Read 5 Sent Done Done false
17987 5384 Acked Write 4 Sent Done Done false
17988 5385 Acked Read 2 Sent Done Done false
17989 5386 Acked Write 4 Sent Done Done false
17990 5387 Acked Read 10 Sent Done Done false
17991 5388 Acked Write 8 Sent Done Done false
17992 5389 Acked Read 1 Sent Done Done false
17993 5390 Acked Read 7 Sent Done Done false
17994 5391 Acked Read 1 Sent Done Done false
17995 5392 Acked Read 7 Sent Done Done false
17996 5393 Acked Write 5 Sent Done Done false
17997 5394 Acked Write 3 Sent Done Done false
17998 5395 Acked Read 10 Sent Done Done false
17999 5396 Acked Read 6 Sent Done Done false
18000 5397 Acked Read 8 Sent Done Done false
18001 5398 Acked Write 3 Sent Done Done false
18002 5399 Acked Read 1 Sent Done Done false
18003 5400 Acked Read 7 Sent Done Done false
18004 5401 Acked Read 4 Sent Done Done false
18005 5402 Acked Read 9 Sent Done Done false
18006 5403 Acked Write 8 Sent Done Done false
18007 5404 Acked Read 2 Sent Done Done false
18008 5405 Acked Write 2 Sent Done Done false
18009 5406 Acked Read 6 Sent Done Done false
18010 5407 Acked Write 8 Sent Done Done false
18011 5408 Acked Write 10 Sent Done Done false
18012 5409 Acked Flush 0 Sent Done Done false
18013 5410 Acked Read 7 Sent Done Done false
18014 5411 Acked Write 10 Sent Done Done false
18015 5412 Acked Read 6 Sent Done Done false
18016 5413 Acked Write 7 Sent Done Done false
18017 5414 Acked Write 4 Sent Done Done false
18018 5415 Acked Read 7 Sent Done Done false
18019 5416 Acked Write 4 Sent Done Done false
18020 5417 Acked Flush 0 Sent Done Done false
18021 5418 Acked Write 4 Sent Done Done false
18022 5419 Acked Write 6 Sent Done Done false
18023 5420 Acked Write 10 Sent Done Done false
18024 5421 Acked Read 3 Sent Done Done false
18025 5422 Acked Write 10 Sent Done Done false
18026 5423 Acked Read 8 Sent Done Done false
18027 5424 Acked Read 3 Sent Done Done false
18028 5425 Acked Read 2 Sent Done Done false
18029 5426 Acked Write 2 Sent Done Done false
18030 5427 Acked Write 1 Sent Done Done false
18031 5428 Acked Read 3 Sent Done Done false
18032 5429 Acked Write 10 Sent Done Done false
18033 5430 Acked Write 4 Sent Done Done false
18034 5431 Acked Read 5 Sent Done Done false
18035 5432 Acked Read 6 Sent Done Done false
18036 5433 Acked Read 8 Sent Done Done false
18037 5434 Acked Write 2 Sent Done Done false
18038 5435 Acked Write 6 Sent Done Done false
18039 5436 Acked Read 6 Sent Done Done false
18040 5437 Acked Read 10 Sent Done Done false
18041 5438 Acked Read 8 Sent Done Done false
18042 5439 Acked Read 5 Sent Done Done false
18043 5440 Acked Flush 0 Sent Done Done false
18044 5441 Acked Write 3 Sent Done Done false
18045 5442 Acked Write 5 Sent Done Done false
18046 5443 Acked Read 6 Sent Done Done false
18047 5444 Acked Read 9 Sent Done Done false
18048 5445 Acked Read 4 Sent Done Done false
18049 5446 Acked Write 5 Sent Done Done false
18050 5447 Acked Read 3 Sent Done Done false
18051 5448 Acked Read 1 Sent Done Done false
18052 5449 Acked Read 4 Sent Done Done false
18053 5450 Acked Write 9 Sent Done Done false
18054 5451 Acked Read 3 Sent Done Done false
18055 5452 Acked Read 10 Sent Done Done false
18056 5453 Acked Read 1 Sent Done Done false
18057 5454 Acked Write 4 Sent Done Done false
18058 5455 Acked Flush 0 Sent Done Done false
18059 5456 Acked Write 7 Sent Done Done false
18060 5457 Acked Read 7 Sent Done Done false
18061 5458 Acked Read 9 Sent Done Done false
18062 5459 Acked Read 9 Sent Done Done false
18063 5460 Acked Write 3 Sent Done Done false
18064 5461 Acked Read 1 Sent Done Done false
18065 5462 Acked Write 2 Sent Done Done false
18066 5463 Acked Write 8 Sent Done Done false
18067 5464 Acked Write 2 Sent Done Done false
18068 5465 Acked Read 7 Sent Done Done false
18069 5466 Acked Read 10 Sent Done Done false
18070 5467 Acked Write 4 Sent Done Done false
18071 5468 Acked Write 1 Sent Done Done false
18072 5469 Acked Read 9 Sent Done Done false
18073 5470 Acked Read 3 Sent Done Done false
18074 5471 Acked Read 1 Sent Done Done false
18075 5472 Acked Read 10 Sent Done Done false
18076 5473 Acked Read 4 Sent Done Done false
18077 5474 Acked Read 5 Sent Done Done false
18078 5475 Acked Write 4 Sent Done Done false
18079 5476 Acked Write 4 Sent Done Done false
18080 5477 Acked Write 2 Sent Done Done false
18081 5478 Acked Read 8 Sent Done Done false
18082 5479 Acked Read 4 Sent Done Done false
18083 5480 Acked Read 1 Sent Done Done false
18084 5481 Acked Read 8 Sent Done Done false
18085 5482 Acked Read 1 Sent Done Done false
18086 5483 Acked Write 1 Sent Done Done false
18087 5484 Acked Write 7 Sent Done Done false
18088 5485 Acked Flush 0 Sent Done Done false
18089 5486 Acked Flush 0 Sent Done Done false
18090 5487 Acked Read 3 Sent Done Done false
18091 5488 Acked Read 5 Sent Done Done false
18092 5489 Acked Write 5 Sent Done Done false
18093 5490 Acked Read 4 Sent Done Done false
18094 5491 Acked Write 2 Sent Done Done false
18095 5492 Acked Read 4 Sent Done Done false
18096 5493 Acked Read 9 Sent Done Done false
18097 5494 Acked Write 9 Sent Done Done false
18098 5495 Acked Write 3 Sent Done Done false
18099 5496 Acked Read 7 Sent Done Done false
18100 5497 Acked Read 6 Sent Done Done false
18101 5498 Acked Read 7 Sent Done Done false
18102 5499 Acked Write 4 Sent Done Done false
18103 5500 Acked Write 8 Sent Done Done false
18104 5501 Acked Read 10 Sent Done Done false
18105 5502 Acked Write 10 Sent Done Done false
18106 5503 Acked Read 9 Sent Done Done false
18107 5504 Acked Write 4 Sent Done Done false
18108 5505 Acked Read 7 Sent Done Done false
18109 5506 Acked Read 5 Sent Done Done false
18110 5507 Acked Flush 0 Sent Done Done false
18111 5508 Acked Write 9 Sent Done Done false
18112 5509 Acked Read 10 Sent Done Done false
18113 5510 Acked Write 5 Sent Done Done false
18114 5511 Acked Write 7 Sent Done Done false
18115 5512 Acked Write 8 Sent Done Done false
18116 5513 Acked Read 4 Sent Done Done false
18117 5514 Acked Write 7 Sent Done Done false
18118 5515 Acked Read 2 Sent Done Done false
18119 5516 Acked Write 8 Sent Done Done false
18120 5517 Acked Write 9 Sent Done Done false
18121 5518 Acked Flush 0 Sent Done Done false
18122 5519 Acked Write 6 Sent Done Done false
18123 5520 Acked Read 5 Sent Done Done false
18124 5521 Acked Read 1 Sent Done Done false
18125 5522 Acked Write 10 Sent Done Done false
18126 5523 Acked Read 1 Sent Done Done false
18127 5524 Acked Flush 0 Sent Done Done false
18128 5525 Acked Read 9 Sent Done Done false
18129 5526 Acked Read 5 Sent Done Done false
18130 5527 Acked Read 7 Sent Done Done false
18131 5528 Acked Read 1 Sent Done Done false
18132 STATES DS:0 DS:1 DS:2 TOTAL
18133 Sent 301 0 0 301
18134 Done 0 301 301 602
18135 Skipped 0 0 0 0
18136 Error 0 0 0 0
18137 Last Flush: Some(JobId(5227)) Some(JobId(5524)) Some(JobId(5524))
18138 Downstairs last five completed: 5227 5226 5225 5224 5223
18139 Upstairs last five completed: 5528 5527 5526 5525 5524
18140 CLIENT: Up:0 ds:301 act:2
181412025-03-17T22:38:33.168ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181422025-03-17T22:38:33.898ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181432025-03-17T22:38:34.413ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181442025-03-17T22:38:34.413ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61347) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181452025-03-17T22:38:34.414ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181462025-03-17T22:38:34.415ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181472025-03-17T22:38:34.415ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181482025-03-17T22:38:34.415ZINFOcrucible: send last flush ID to this DS: Some(JobId(5227)) = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181492025-03-17T22:38:34.415ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181502025-03-17T22:38:34.415ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5227)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181512025-03-17T22:38:34.415ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5227)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18152 ----------------------------------------------------------------
18153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18154 Downstairs last five completed: 5529 5528 5527 5526 5525
18155 Upstairs last five completed: 5529 5528 5527 5526 5525
18156 CLIENT: Up:0 ds:0 act:3
18157 Replay: All jobs finished
18158 Main task has work to do, go find it
18159 got dsc Stop(0)
18160 stop 0
18161 Mar 17 22:38:36.669 INFO request completed, latency_us: 366, response_code: 204, uri: /stop/cid/0, method: POST, req_id: feff599b-3ee2-41b0-8033-acba38368229, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
18162 [8810] Got stop action so:false kr:false
18163 Mar 17 22:38:36.669 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/0, method: GET, req_id: ba157492-8171-415e-b76d-0fdd4a17fab5, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
181642025-03-17T22:38:36.670ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181652025-03-17T22:38:36.670ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181662025-03-17T22:38:36.670ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
181672025-03-17T22:38:36.670ZINFOcrucible: client task is exiting = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18168 [8810] Exited with: ExitStatus(unix_wait_status(9))
18169 [8810][0] reports Exit
18170 Mar 17 22:38:40.672 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/0, method: GET, req_id: 02ab108e-ab3f-4c46-a348-6ea6258a8c2f, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
18171 001/300 Read block 610 len 24576
18172 002/300 Write block 4670 len 16384 data: 4 5 5 6
18173 003/300 Read block 93 len 36864
18174 004/300 Read block 4374 len 28672
18175 005/300 Read block 4807 len 32768
18176 006/300 Write block 3309 len 8192 data: 3 3
18177 007/300 Write block 1593 len 24576 data: 6 7 8 8 7 6
18178 008/300 Read block 2399 len 12288
18179 009/300 Write block 384 len 40960 data: 5 6 5 5 5 4 4 3 3 3
18180 010/300 Read block 2785 len 28672
18181 011/300 Read block 1115 len 40960
18182 012/300 Read block 4399 len 32768
18183 013/300 Read block 4084 len 32768
18184 014/300 Write block 3400 len 16384 data: 6 5 6 5
18185 015/300 Write block 4159 len 8192 data: 4 3
18186 016/300 Read block 3889 len 4096
18187 017/300 Read block 4961 len 24576
18188 018/300 Read block 2904 len 4096
18189 019/300 Read block 3617 len 12288
18190 020/300 Write block 4818 len 20480 data: 4 4 4 3 4
18191 021/300 Read block 1616 len 4096
18192 022/300 Read block 2606 len 24576
18193 023/300 Flush
18194 024/300 Read block 4518 len 8192
18195 025/300 Write block 4300 len 4096 data: 4
18196 026/300 Read block 4684 len 40960
18197 027/300 Flush
18198 028/300 Write block 4903 len 12288 data: 4 5 4
18199 029/300 Write block 1070 len 4096 data: 4
18200 030/300 Read block 2972 len 36864
18201 031/300 Write block 2865 len 8192 data: 2 2
18202 032/300 Read block 3655 len 16384
18203 033/300 Write block 4959 len 40960 data: 3 3 3 3 3 3 3 4 4 4
18204 034/300 Write block 3203 len 24576 data: 3 4 4 3 4 4
18205 035/300 Write block 3188 len 8192 data: 6 6
18206 036/300 Read block 2105 len 12288
18207 037/300 Write block 1248 len 16384 data: 2 2 2 2
18208 038/300 Read block 1813 len 24576
18209 039/300 Flush
18210 040/300 Read block 3691 len 24576
18211 041/300 Read block 3022 len 20480
18212 042/300 Flush
18213 043/300 Read block 724 len 20480
18214 044/300 Write block 1090 len 32768 data: 6 5 5 4 4 2 3 3
18215 045/300 Read block 3979 len 16384
18216 046/300 Read block 1843 len 12288
18217 047/300 Read block 3545 len 24576
18218 048/300 Read block 363 len 32768
18219 049/300 Read block 1824 len 40960
18220 050/300 Read block 3582 len 16384
18221 051/300 Read block 651 len 24576
18222 052/300 Read block 3642 len 20480
18223 053/300 Write block 23 len 28672 data: 4 4 4 4 4 4 3
18224 054/300 Read block 2852 len 20480
18225 055/300 Write block 4338 len 36864 data: 4 6 7 6 6 5 8 8 9
18226 056/300 Write block 612 len 36864 data: 3 4 4 5 6 5 4 3 3
18227 057/300 Read block 2182 len 28672
18228 058/300 Read block 2235 len 16384
18229 059/300 Read block 486 len 4096
18230 060/300 Read block 451 len 20480
18231 061/300 Read block 3508 len 16384
18232 062/300 Write block 387 len 36864 data: 6 6 5 5 4 4 4 3 2
18233 063/300 Read block 819 len 32768
18234 064/300 Flush
18235 065/300 Read block 359 len 4096
18236 066/300 Write block 1301 len 8192 data: 3 3
18237 067/300 Read block 2 len 8192
18238 068/300 Write block 2410 len 24576 data: 3 3 3 2 2 2
18239 069/300 Write block 284 len 24576 data: 3 2 3 3 3 4
18240 070/300 Read block 2759 len 36864
18241 071/300 Write block 3141 len 4096 data: 3
18242 072/300 Read block 2773 len 36864
18243 073/300 Read block 1809 len 32768
18244 074/300 Read block 2627 len 36864
18245 075/300 Read block 4914 len 4096
18246 076/300 Write block 946 len 12288 data: 6 5 4
18247 077/300 Flush
18248 078/300 Write block 4307 len 8192 data: 5 7
18249 079/300 Write block 1169 len 36864 data: 3 2 2 2 2 2 3 4 4
18250 080/300 Read block 1836 len 20480
18251 081/300 Read block 37 len 16384
18252 082/300 Read block 166 len 32768
18253 083/300 Flush
18254 084/300 Read block 1804 len 16384
18255 085/300 Write block 3207 len 32768 data: 5 5 6 8 8 10 9 9
18256 086/300 Write block 3834 len 32768 data: 4 5 4 5 4 6 5 4
18257 087/300 Read block 1046 len 12288
18258 088/300 Flush
18259 089/300 Read block 3419 len 40960
18260 090/300 Write block 4789 len 12288 data: 4 4 4
18261 091/300 Read block 3201 len 28672
18262 092/300 Write block 4493 len 16384 data: 5 5 4 4
18263 093/300 Write block 559 len 8192 data: 4 4
18264 094/300 Read block 445 len 16384
18265 095/300 Read block 4905 len 40960
18266 096/300 Write block 99 len 4096 data: 4
18267 097/300 Write block 136 len 24576 data: 5 4 5 4 4 4
18268 098/300 Read block 3525 len 8192
18269 099/300 Read block 2953 len 32768
18270 100/300 Write block 4110 len 28672 data: 3 3 3 5 7 6 6
18271 101/300 Read block 3021 len 36864
18272 102/300 Write block 1048 len 20480 data: 2 2 2 2 3
18273 103/300 Write block 3675 len 40960 data: 4 4 4 6 7 7 7 7 4 3
18274 104/300 Read block 2358 len 32768
18275 105/300 Read block 2500 len 32768
18276 106/300 Write block 3417 len 20480 data: 6 6 5 6 7
18277 107/300 Write block 3081 len 20480 data: 4 4 4 3 3
18278 108/300 Write block 67 len 16384 data: 5 4 4 3
18279 109/300 Write block 3860 len 40960 data: 3 2 2 2 2 2 2 2 2 2
18280 110/300 Write block 1987 len 4096 data: 6
18281 111/300 Write block 3940 len 36864 data: 4 5 5 6 6 7 7 6 3
18282 112/300 Write block 3145 len 32768 data: 4 4 4 4 5 5 5 6
18283 113/300 Write block 800 len 32768 data: 4 4 4 3 4 4 4 4
18284 114/300 Write block 10 len 4096 data: 4
18285 115/300 Write block 4390 len 40960 data: 5 5 5 4 4 4 4 4 4 4
18286 116/300 Read block 222 len 4096
18287 117/300 Write block 1293 len 24576 data: 6 6 5 5 5 3
18288 118/300 Write block 3623 len 4096 data: 4
18289 119/300 Write block 3626 len 36864 data: 4 5 5 5 4 4 5 5 5
18290 120/300 Read block 4856 len 16384
18291 121/300 Read block 1889 len 20480
18292 122/300 Write block 4100 len 4096 data: 5
18293 123/300 Read block 3361 len 4096
18294 124/300 Flush
18295 125/300 Write block 1983 len 12288 data: 7 6 7
18296 126/300 Write block 4542 len 4096 data: 5
18297 127/300 Write block 2302 len 4096 data: 3
18298 128/300 Read block 3889 len 28672
18299 129/300 Flush
18300 130/300 Read block 446 len 20480
18301 131/300 Read block 3836 len 36864
18302 132/300 Write block 4765 len 36864 data: 2 2 2 2 2 2 2 2 5
18303 133/300 Read block 2217 len 28672
18304 134/300 Read block 4801 len 16384
18305 135/300 Write block 4948 len 24576 data: 3 3 3 3 4 5
18306 136/300 Write block 840 len 36864 data: 2 3 4 3 3 5 5 4 3
18307 137/300 Write block 190 len 24576 data: 3 2 3 3 3 4
18308 138/300 Read block 3682 len 16384
18309 139/300 Write block 3038 len 12288 data: 2 2 2
18310 140/300 Write block 901 len 28672 data: 5 4 6 5 5 5 5
18311 141/300 Read block 3805 len 40960
18312 142/300 Flush
18313 143/300 Write block 1982 len 32768 data: 6 8 7 8 6 7 6 4
18314 144/300 Flush
18315 145/300 Write block 4374 len 12288 data: 6 6 5
18316 146/300 Write block 2790 len 8192 data: 6 6
18317 147/300 Flush
18318 148/300 Write block 4088 len 4096 data: 4
18319 149/300 Write block 83 len 12288 data: 5 3 3
18320 150/300 Read block 448 len 40960
18321 151/300 Write block 2206 len 8192 data: 7 6
18322 152/300 Read block 1435 len 36864
18323 153/300 Read block 83 len 28672
18324 154/300 Write block 696 len 8192 data: 2 2
18325 155/300 Write block 3940 len 4096 data: 5
18326 156/300 Write block 2550 len 4096 data: 6
18327 157/300 Read block 2986 len 4096
18328 158/300 Read block 1054 len 20480
18329 159/300 Read block 4845 len 32768
18330 160/300 Read block 1250 len 32768
18331 161/300 Write block 207 len 8192 data: 3 3
18332 162/300 Read block 2751 len 24576
18333 163/300 Write block 3812 len 12288 data: 5 5 4
18334 164/300 Read block 4768 len 28672
18335 165/300 Read block 1133 len 12288
18336 166/300 Write block 4568 len 20480 data: 3 3 2 2 2
18337 167/300 Read block 438 len 4096
18338 168/300 Read block 2037 len 12288
18339 169/300 Read block 2744 len 4096
18340 170/300 Read block 4371 len 12288
18341 171/300 Flush
18342 172/300 Write block 3342 len 12288 data: 5 6 6
18343 173/300 Read block 434 len 24576
18344 174/300 Write block 4516 len 40960 data: 4 4 5 5 5 5 3 2 2 3
18345 175/300 Read block 2447 len 24576
18346 176/300 Write block 4119 len 12288 data: 6 5 3
18347 177/300 Write block 1668 len 8192 data: 4 4
18348 178/300 Write block 2546 len 8192 data: 5 5
18349 179/300 Read block 3852 len 40960
18350 180/300 Read block 2754 len 4096
18351 181/300 Write block 2372 len 16384 data: 3 4 4 4
18352 182/300 Read block 2812 len 16384
18353 183/300 Write block 1273 len 32768 data: 4 3 3 3 3 3 4 5
18354 184/300 Write block 4735 len 16384 data: 3 3 3 3
18355 185/300 Read block 4136 len 16384
18356 186/300 Read block 2189 len 24576
18357 187/300 Read block 1452 len 24576
18358 188/300 Write block 3694 len 24576 data: 3 3 3 3 2 2
18359 189/300 Read block 2485 len 20480
18360 190/300 Read block 4248 len 24576
18361 191/300 Read block 2842 len 20480
18362 192/300 Write block 1909 len 36864 data: 3 3 3 4 4 4 3 3 3
18363 193/300 Read block 553 len 4096
18364 194/300 Read block 326 len 24576
18365 195/300 Read block 4678 len 16384
18366 196/300 Read block 2433 len 16384
18367 197/300 Write block 858 len 4096 data: 3
18368 198/300 Flush
18369 199/300 Write block 4291 len 16384 data: 5 6 7 7
18370 200/300 Write block 384 len 24576 data: 6 7 6 7 7 6
18371 201/300 Flush
18372 202/300 Write block 4177 len 16384 data: 3 3 4 4
18373 203/300 Write block 2874 len 16384 data: 3 3 4 5
18374 204/300 Read block 2261 len 36864
18375 205/300 Write block 509 len 24576 data: 3 3 3 3 4 3
18376 206/300 Write block 3624 len 24576 data: 4 4 5 6 6 6
18377 207/300 Read block 2768 len 16384
18378 208/300 Write block 4825 len 16384 data: 3 3 4 5
18379 209/300 Write block 1894 len 16384 data: 3 4 5 5
18380 210/300 Read block 3323 len 36864
18381 211/300 Read block 4989 len 20480
18382 212/300 Write block 732 len 36864 data: 3 4 4 4 4 3 3 3 3
18383 213/300 Read block 3403 len 36864
18384 214/300 Read block 3314 len 32768
18385 215/300 Read block 4926 len 16384
18386 216/300 Flush
18387 217/300 Read block 3927 len 36864
18388 218/300 Write block 1844 len 40960 data: 5 6 6 6 4 3 3 4 4 4
18389 219/300 Write block 3621 len 36864 data: 4 4 5 5 5 6 7 7 7
18390 220/300 Read block 3174 len 16384
18391 221/300 Read block 3837 len 40960
18392 222/300 Read block 3536 len 16384
18393 223/300 Flush
18394 224/300 Read block 4837 len 36864
18395 225/300 Write block 1620 len 8192 data: 3 4
18396 226/300 Write block 2160 len 12288 data: 3 4 4
18397 227/300 Write block 2134 len 16384 data: 4 5 5 4
18398 228/300 Read block 4143 len 28672
18399 229/300 Read block 270 len 24576
18400 230/300 Read block 1511 len 12288
18401 231/300 Write block 2209 len 24576 data: 4 4 3 3 3 3
18402 232/300 Read block 2285 len 8192
18403 233/300 Write block 571 len 4096 data: 2
18404 234/300 Write block 1567 len 16384 data: 3 3 3 3
18405 235/300 Write block 2412 len 36864 data: 4 3 3 3 2 2 2 2 2
18406 236/300 Read block 3212 len 40960
18407 237/300 Write block 3704 len 8192 data: 2 2
18408 238/300 Read block 3889 len 4096
18409 239/300 Read block 2438 len 4096
18410 240/300 Write block 3562 len 8192 data: 4 4
18411 241/300 Flush
18412 242/300 Read block 3996 len 12288
18413 243/300 Flush
18414 244/300 Flush
18415 245/300 Write block 4892 len 20480 data: 3 3 3 3 2
18416 246/300 Read block 369 len 4096
18417 247/300 Read block 2262 len 20480
18418 248/300 Read block 4857 len 36864
18419 249/300 Read block 3784 len 40960
18420 250/300 Read block 779 len 36864
18421 251/300 Read block 2002 len 16384
18422 252/300 Write block 4185 len 8192 data: 5 4
18423 253/300 Read block 2913 len 32768
18424 254/300 Read block 3776 len 32768
18425 255/300 Write block 1460 len 36864 data: 3 3 3 3 3 3 3 3 3
18426 256/300 Read block 2082 len 4096
18427 257/300 Write block 1647 len 40960 data: 3 4 3 3 3 3 3 3 5 6
18428 258/300 Write block 1479 len 20480 data: 8 8 8 8 6
18429 259/300 Flush
18430 260/300 Write block 362 len 24576 data: 4 4 4 4 4 4
18431 261/300 Read block 4654 len 12288
18432 262/300 Read block 3102 len 16384
18433 263/300 Read block 1099 len 16384
18434 264/300 Read block 971 len 36864
18435 265/300 Read block 2387 len 28672
18436 266/300 Read block 1592 len 36864
18437 267/300 Read block 4072 len 40960
18438 268/300 Write block 1361 len 28672 data: 5 5 5 5 4 3 3
18439 269/300 Read block 2440 len 16384
18440 270/300 Flush
18441 271/300 Write block 3896 len 24576 data: 6 4 3 3 5 6
18442 272/300 Write block 3508 len 36864 data: 4 4 3 2 2 2 2 2 3
18443 273/300 Read block 1802 len 4096
18444 274/300 Write block 2214 len 36864 data: 4 4 4 3 2 2 2 2 2
18445 275/300 Write block 4700 len 28672 data: 5 5 5 4 4 5 4
18446 276/300 Write block 3563 len 40960 data: 5 3 4 4 4 5 6 7 6 6
18447 277/300 Write block 4561 len 36864 data: 2 3 3 4 3 3 3 4 4
18448 278/300 Write block 3839 len 36864 data: 7 6 5 4 4 4 5 4 5
18449 279/300 Read block 2780 len 12288
18450 280/300 Read block 4863 len 40960
18451 281/300 Write block 4095 len 24576 data: 4 6 6 5 5 6
18452 282/300 Read block 2901 len 32768
18453 283/300 Read block 1969 len 8192
18454 284/300 Write block 2994 len 36864 data: 5 4 3 4 4 4 4 4 3
18455 285/300 Read block 1998 len 20480
18456 286/300 Write block 1162 len 12288 data: 7 5 5
18457 287/300 Flush
18458 288/300 Write block 2053 len 36864 data: 4 4 3 2 2 3 2 2 2
18459 289/300 Write block 576 len 28672 data: 2 2 2 3 3 3 3
18460 290/300 Read block 3419 len 16384
18461 291/300 Read block 1543 len 28672
18462 292/300 Write block 3887 len 4096 data: 5
18463 293/300 Flush
18464 294/300 Write block 2849 len 36864 data: 2 2 2 2 2 2 2 2 2
18465 295/300 Write block 209 len 8192 data: 3 3
18466 296/300 Read block 2665 len 20480
18467 297/300 Write block 1754 len 24576 data: 2 2 2 2 2 3
18468 298/300 Read block 3463 len 36864
18469 299/300 Write block 2328 len 16384 data: 2 4 4 4
18470 300/300 Flush
18471 301/300 Flush
18472 Main task has work to do, go find it
18473 got dsc Start(0)
18474 start 0
18475 Mar 17 22:38:40.916 INFO request completed, latency_us: 333, response_code: 204, uri: /start/cid/0, method: POST, req_id: ae875e36-31c2-4c54-a7cb-e65a4ecd77a7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
18476 Replay: started 0, returned:Ok(())
18477 ----------------------------------------------------------------
18478 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18479 States: Offline Active Active
18480 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18481 5530 Acked Read 6 Sent Done Done false
18482 5531 Acked Write 4 Sent Done Done false
18483 5532 Acked Read 9 Sent Done Done false
18484 5533 Acked Read 7 Sent Done Done false
18485 5534 Acked Read 8 Sent Done Done false
18486 5535 Acked Write 2 Sent Done Done false
18487 5536 Acked Write 6 Sent Done Done false
18488 5537 Acked Read 3 Sent Done Done false
18489 5538 Acked Write 10 Sent Done Done false
18490 5539 Acked Read 7 Sent Done Done false
18491 5540 Acked Read 10 Sent Done Done false
18492 5541 Acked Read 8 Sent Done Done false
18493 5542 Acked Read 8 Sent Done Done false
18494 5543 Acked Write 4 Sent Done Done false
18495 5544 Acked Write 2 Sent Done Done false
18496 5545 Acked Read 1 Sent Done Done false
18497 5546 Acked Read 6 Sent Done Done false
18498 5547 Acked Read 1 Sent Done Done false
18499 5548 Acked Read 3 Sent Done Done false
18500 5549 Acked Write 5 Sent Done Done false
18501 5550 Acked Read 1 Sent Done Done false
18502 5551 Acked Read 6 Sent Done Done false
18503 5552 Acked Flush 0 Sent Done Done false
18504 5553 Acked Read 2 Sent Done Done false
18505 5554 Acked Write 1 Sent Done Done false
18506 5555 Acked Read 10 Sent Done Done false
18507 5556 Acked Flush 0 Sent Done Done false
18508 5557 Acked Write 3 Sent Done Done false
18509 5558 Acked Write 1 Sent Done Done false
18510 5559 Acked Read 9 Sent Done Done false
18511 5560 Acked Write 2 Sent Done Done false
18512 5561 Acked Read 4 Sent Done Done false
18513 5562 Acked Write 10 Sent Done Done false
18514 5563 Acked Write 6 Sent Done Done false
18515 5564 Acked Write 2 Sent Done Done false
18516 5565 Acked Read 3 Sent Done Done false
18517 5566 Acked Write 4 Sent Done Done false
18518 5567 Acked Read 6 Sent Done Done false
18519 5568 Acked Flush 0 Sent Done Done false
18520 5569 Acked Read 6 Sent Done Done false
18521 5570 Acked Read 5 Sent Done Done false
18522 5571 Acked Flush 0 Sent Done Done false
18523 5572 Acked Read 5 Sent Done Done false
18524 5573 Acked Write 8 Sent Done Done false
18525 5574 Acked Read 4 Sent Done Done false
18526 5575 Acked Read 3 Sent Done Done false
18527 5576 Acked Read 6 Sent Done Done false
18528 5577 Acked Read 8 Sent Done Done false
18529 5578 Acked Read 10 Sent Done Done false
18530 5579 Acked Read 4 Sent Done Done false
18531 5580 Acked Read 6 Sent Done Done false
18532 5581 Acked Read 5 Sent Done Done false
18533 5582 Acked Write 7 Sent Done Done false
18534 5583 Acked Read 5 Sent Done Done false
18535 5584 Acked Write 9 Sent Done Done false
18536 5585 Acked Write 9 Sent Done Done false
18537 5586 Acked Read 7 Sent Done Done false
18538 5587 Acked Read 4 Sent Done Done false
18539 5588 Acked Read 1 Sent Done Done false
18540 5589 Acked Read 5 Sent Done Done false
18541 5590 Acked Read 4 Sent Done Done false
18542 5591 Acked Write 9 Sent Done Done false
18543 5592 Acked Read 8 Sent Done Done false
18544 5593 Acked Flush 0 Sent Done Done false
18545 5594 Acked Read 1 Sent Done Done false
18546 5595 Acked Write 2 Sent Done Done false
18547 5596 Acked Read 2 Sent Done Done false
18548 5597 Acked Write 6 Sent Done Done false
18549 5598 Acked Write 6 Sent Done Done false
18550 5599 Acked Read 9 Sent Done Done false
18551 5600 Acked Write 1 Sent Done Done false
18552 5601 Acked Read 9 Sent Done Done false
18553 5602 Acked Read 8 Sent Done Done false
18554 5603 Acked Read 9 Sent Done Done false
18555 5604 Acked Read 1 Sent Done Done false
18556 5605 Acked Write 3 Sent Done Done false
18557 5606 Acked Flush 0 Sent Done Done false
18558 5607 Acked Write 2 Sent Done Done false
18559 5608 Acked Write 9 Sent Done Done false
18560 5609 Acked Read 5 Sent Done Done false
18561 5610 Acked Read 4 Sent Done Done false
18562 5611 Acked Read 8 Sent Done Done false
18563 5612 Acked Flush 0 Sent Done Done false
18564 5613 Acked Read 4 Sent Done Done false
18565 5614 Acked Write 8 Sent Done Done false
18566 5615 Acked Write 8 Sent Done Done false
18567 5616 Acked Read 3 Sent Done Done false
18568 5617 Acked Flush 0 Sent Done Done false
18569 5618 Acked Read 10 Sent Done Done false
18570 5619 Acked Write 3 Sent Done Done false
18571 5620 Acked Read 7 Sent Done Done false
18572 5621 Acked Write 4 Sent Done Done false
18573 5622 Acked Write 2 Sent Done Done false
18574 5623 Acked Read 4 Sent Done Done false
18575 5624 Acked Read 10 Sent Done Done false
18576 5625 Acked Write 1 Sent Done Done false
18577 5626 Acked Write 6 Sent Done Done false
18578 5627 Acked Read 2 Sent Done Done false
18579 5628 Acked Read 8 Sent Done Done false
18580 5629 Acked Write 7 Sent Done Done false
18581 5630 Acked Read 9 Sent Done Done false
18582 5631 Acked Write 5 Sent Done Done false
18583 5632 Acked Write 10 Sent Done Done false
18584 5633 Acked Read 8 Sent Done Done false
18585 5634 Acked Read 8 Sent Done Done false
18586 5635 Acked Write 5 Sent Done Done false
18587 5636 Acked Write 5 Sent Done Done false
18588 5637 Acked Write 4 Sent Done Done false
18589 5638 Acked Write 10 Sent Done Done false
18590 5639 Acked Write 1 Sent Done Done false
18591 5640 Acked Write 9 Sent Done Done false
18592 5641 Acked Write 8 Sent Done Done false
18593 5642 Acked Write 8 Sent Done Done false
18594 5643 Acked Write 1 Sent Done Done false
18595 5644 Acked Write 10 Sent Done Done false
18596 5645 Acked Read 1 Sent Done Done false
18597 5646 Acked Write 6 Sent Done Done false
18598 5647 Acked Write 1 Sent Done Done false
18599 5648 Acked Write 9 Sent Done Done false
18600 5649 Acked Read 4 Sent Done Done false
18601 5650 Acked Read 5 Sent Done Done false
18602 5651 Acked Write 1 Sent Done Done false
18603 5652 Acked Read 1 Sent Done Done false
18604 5653 Acked Flush 0 Sent Done Done false
18605 5654 Acked Write 3 Sent Done Done false
18606 5655 Acked Write 1 Sent Done Done false
18607 5656 Acked Write 1 Sent Done Done false
18608 5657 Acked Read 7 Sent Done Done false
18609 5658 Acked Flush 0 Sent Done Done false
18610 5659 Acked Read 5 Sent Done Done false
18611 5660 Acked Read 9 Sent Done Done false
18612 5661 Acked Write 9 Sent Done Done false
18613 5662 Acked Read 7 Sent Done Done false
18614 5663 Acked Read 4 Sent Done Done false
18615 5664 Acked Write 6 Sent Done Done false
18616 5665 Acked Write 9 Sent Done Done false
18617 5666 Acked Write 6 Sent Done Done false
18618 5667 Acked Read 4 Sent Done Done false
18619 5668 Acked Write 3 Sent Done Done false
18620 5669 Acked Write 7 Sent Done Done false
18621 5670 Acked Read 10 Sent Done Done false
18622 5671 Acked Flush 0 Sent Done Done false
18623 5672 Acked Write 8 Sent Done Done false
18624 5673 Acked Flush 0 Sent Done Done false
18625 5674 Acked Write 3 Sent Done Done false
18626 5675 Acked Write 2 Sent Done Done false
18627 5676 Acked Flush 0 Sent Done Done false
18628 5677 Acked Write 1 Sent Done Done false
18629 5678 Acked Write 3 Sent Done Done false
18630 5679 Acked Read 10 Sent Done Done false
18631 5680 Acked Write 2 Sent Done Done false
18632 5681 Acked Read 9 Sent Done Done false
18633 5682 Acked Read 7 Sent Done Done false
18634 5683 Acked Write 2 Sent Done Done false
18635 5684 Acked Write 1 Sent Done Done false
18636 5685 Acked Write 1 Sent Done Done false
18637 5686 Acked Read 1 Sent Done Done false
18638 5687 Acked Read 5 Sent Done Done false
18639 5688 Acked Read 8 Sent Done Done false
18640 5689 Acked Read 8 Sent Done Done false
18641 5690 Acked Write 2 Sent Done Done false
18642 5691 Acked Read 6 Sent Done Done false
18643 5692 Acked Write 3 Sent Done Done false
18644 5693 Acked Read 7 Sent Done Done false
18645 5694 Acked Read 3 Sent Done Done false
18646 5695 Acked Write 5 Sent Done Done false
18647 5696 Acked Read 1 Sent Done Done false
18648 5697 Acked Read 3 Sent Done Done false
18649 5698 Acked Read 1 Sent Done Done false
18650 5699 Acked Read 3 Sent Done Done false
18651 5700 Acked Flush 0 Sent Done Done false
18652 5701 Acked Write 3 Sent Done Done false
18653 5702 Acked Read 6 Sent Done Done false
18654 5703 Acked Write 10 Sent Done Done false
18655 5704 Acked Read 6 Sent Done Done false
18656 5705 Acked Write 3 Sent Done Done false
18657 5706 Acked Write 2 Sent Done Done false
18658 5707 Acked Write 2 Sent Done Done false
18659 5708 Acked Read 10 Sent Done Done false
18660 5709 Acked Read 1 Sent Done Done false
18661 5710 Acked Write 4 Sent Done Done false
18662 5711 Acked Read 4 Sent Done Done false
18663 5712 Acked Write 8 Sent Done Done false
18664 5713 Acked Write 4 Sent Done Done false
18665 5714 Acked Read 4 Sent Done Done false
18666 5715 Acked Read 6 Sent Done Done false
18667 5716 Acked Read 6 Sent Done Done false
18668 5717 Acked Write 6 Sent Done Done false
18669 5718 Acked Read 5 Sent Done Done false
18670 5719 Acked Read 6 Sent Done Done false
18671 5720 Acked Read 5 Sent Done Done false
18672 5721 Acked Write 9 Sent Done Done false
18673 5722 Acked Read 1 Sent Done Done false
18674 5723 Acked Read 6 Sent Done Done false
18675 5724 Acked Read 4 Sent Done Done false
18676 5725 Acked Read 4 Sent Done Done false
18677 5726 Acked Write 1 Sent Done Done false
18678 5727 Acked Flush 0 Sent Done Done false
18679 5728 Acked Write 4 Sent Done Done false
18680 5729 Acked Write 6 Sent Done Done false
18681 5730 Acked Flush 0 Sent Done Done false
18682 5731 Acked Write 4 Sent Done Done false
18683 5732 Acked Write 4 Sent Done Done false
18684 5733 Acked Read 9 Sent Done Done false
18685 5734 Acked Write 6 Sent Done Done false
18686 5735 Acked Write 6 Sent Done Done false
18687 5736 Acked Read 4 Sent Done Done false
18688 5737 Acked Write 4 Sent Done Done false
18689 5738 Acked Write 4 Sent Done Done false
18690 5739 Acked Read 9 Sent Done Done false
18691 5740 Acked Read 5 Sent Done Done false
18692 5741 Acked Write 9 Sent Done Done false
18693 5742 Acked Read 9 Sent Done Done false
18694 5743 Acked Read 8 Sent Done Done false
18695 5744 Acked Read 4 Sent Done Done false
18696 5745 Acked Flush 0 Sent Done Done false
18697 5746 Acked Read 9 Sent Done Done false
18698 5747 Acked Write 10 Sent Done Done false
18699 5748 Acked Write 9 Sent Done Done false
18700 5749 Acked Read 4 Sent Done Done false
18701 5750 Acked Read 10 Sent Done Done false
18702 5751 Acked Read 4 Sent Done Done false
18703 5752 Acked Flush 0 Sent Done Done false
18704 5753 Acked Read 9 Sent Done Done false
18705 5754 Acked Write 2 Sent Done Done false
18706 5755 Acked Write 3 Sent Done Done false
18707 5756 Acked Write 4 Sent Done Done false
18708 5757 Acked Read 7 Sent Done Done false
18709 5758 Acked Read 6 Sent Done Done false
18710 5759 Acked Read 3 Sent Done Done false
18711 5760 Acked Write 6 Sent Done Done false
18712 5761 Acked Read 2 Sent Done Done false
18713 5762 Acked Write 1 Sent Done Done false
18714 5763 Acked Write 4 Sent Done Done false
18715 5764 Acked Write 9 Sent Done Done false
18716 5765 Acked Read 10 Sent Done Done false
18717 5766 Acked Write 2 Sent Done Done false
18718 5767 Acked Read 1 Sent Done Done false
18719 5768 Acked Read 1 Sent Done Done false
18720 5769 Acked Write 2 Sent Done Done false
18721 5770 Acked Flush 0 Sent Done Done false
18722 5771 Acked Read 3 Sent Done Done false
18723 5772 Acked Flush 0 Sent Done Done false
18724 5773 Acked Flush 0 Sent Done Done false
18725 5774 Acked Write 5 Sent Done Done false
18726 5775 Acked Read 1 Sent Done Done false
18727 5776 Acked Read 5 Sent Done Done false
18728 5777 Acked Read 9 Sent Done Done false
18729 5778 Acked Read 10 Sent Done Done false
18730 5779 Acked Read 9 Sent Done Done false
18731 5780 Acked Read 4 Sent Done Done false
18732 5781 Acked Write 2 Sent Done Done false
18733 5782 Acked Read 8 Sent Done Done false
18734 5783 Acked Read 8 Sent Done Done false
18735 5784 Acked Write 9 Sent Done Done false
18736 5785 Acked Read 1 Sent Done Done false
18737 5786 Acked Write 10 Sent Done Done false
18738 5787 Acked Write 5 Sent Done Done false
18739 5788 Acked Flush 0 Sent Done Done false
18740 5789 Acked Write 6 Sent Done Done false
18741 5790 Acked Read 3 Sent Done Done false
18742 5791 Acked Read 4 Sent Done Done false
18743 5792 Acked Read 4 Sent Done Done false
18744 5793 Acked Read 9 Sent Done Done false
18745 5794 Acked Read 7 Sent Done Done false
18746 5795 Acked Read 9 Sent Done Done false
18747 5796 Acked Read 10 Sent Done Done false
18748 5797 Acked Write 7 Sent Done Done false
18749 5798 Acked Read 4 Sent Done Done false
18750 5799 Acked Flush 0 Sent Done Done false
18751 5800 Acked Write 6 Sent Done Done false
18752 5801 Acked Write 9 Sent Done Done false
18753 5802 Acked Read 1 Sent Done Done false
18754 5803 Acked Write 9 Sent Done Done false
18755 5804 Acked Write 7 Sent Done Done false
18756 5805 Acked Write 10 Sent Done Done false
18757 5806 Acked Write 9 Sent Done Done false
18758 5807 Acked Write 9 Sent Done Done false
18759 5808 Acked Read 3 Sent Done Done false
18760 5809 Acked Read 10 Sent Done Done false
18761 5810 Acked Write 6 Sent Done Done false
18762 5811 Acked Read 8 Sent Done Done false
18763 5812 Acked Read 2 Sent Done Done false
18764 5813 Acked Write 9 Sent Done Done false
18765 5814 Acked Read 5 Sent Done Done false
18766 5815 Acked Write 3 Sent Done Done false
18767 5816 Acked Flush 0 Sent Done Done false
18768 5817 Acked Write 9 Sent Done Done false
18769 5818 Acked Write 7 Sent Done Done false
18770 5819 Acked Read 4 Sent Done Done false
18771 5820 Acked Read 7 Sent Done Done false
18772 5821 Acked Write 1 Sent Done Done false
18773 5822 Acked Flush 0 Sent Done Done false
18774 5823 Acked Write 9 Sent Done Done false
18775 5824 Acked Write 2 Sent Done Done false
18776 5825 Acked Read 5 Sent Done Done false
18777 5826 Acked Write 6 Sent Done Done false
18778 5827 Acked Read 9 Sent Done Done false
18779 5828 Acked Write 4 Sent Done Done false
18780 5829 Acked Flush 0 Sent Done Done false
18781 5830 Acked Flush 0 Sent Done Done false
18782 STATES DS:0 DS:1 DS:2 TOTAL
18783 Sent 301 0 0 301
18784 Done 0 301 301 602
18785 Skipped 0 0 0 0
18786 Error 0 0 0 0
18787 Last Flush: Some(JobId(5529)) Some(JobId(5830)) Some(JobId(5830))
18788 Downstairs last five completed: 5529 5528 5527 5526 5525
18789 Upstairs last five completed: 5830 5829 5828 5827 5826
18790 CLIENT: Up:0 ds:301 act:2
187912025-03-17T22:38:41.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
187922025-03-17T22:38:42.701ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
187932025-03-17T22:38:43.901ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18794 [8810] got start action
18795 [8810] I am going to restart
18796 Starting downstairs at port 8810
18797 Make output file at "/tmp/dsc/downstairs-8810.txt"
18798 [8810][0] reports Starting
18799 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
18800 [8810][0] reports Running
188012025-03-17T22:38:44.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
18802 ----------------------------------------------------------------
18803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18804 States: Offline Active Active
18805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18806 5530 Acked Read 6 Sent Done Done false
18807 5531 Acked Write 4 Sent Done Done false
18808 5532 Acked Read 9 Sent Done Done false
18809 5533 Acked Read 7 Sent Done Done false
18810 5534 Acked Read 8 Sent Done Done false
18811 5535 Acked Write 2 Sent Done Done false
18812 5536 Acked Write 6 Sent Done Done false
18813 5537 Acked Read 3 Sent Done Done false
18814 5538 Acked Write 10 Sent Done Done false
18815 5539 Acked Read 7 Sent Done Done false
18816 5540 Acked Read 10 Sent Done Done false
18817 5541 Acked Read 8 Sent Done Done false
18818 5542 Acked Read 8 Sent Done Done false
18819 5543 Acked Write 4 Sent Done Done false
18820 5544 Acked Write 2 Sent Done Done false
18821 5545 Acked Read 1 Sent Done Done false
18822 5546 Acked Read 6 Sent Done Done false
18823 5547 Acked Read 1 Sent Done Done false
18824 5548 Acked Read 3 Sent Done Done false
18825 5549 Acked Write 5 Sent Done Done false
18826 5550 Acked Read 1 Sent Done Done false
18827 5551 Acked Read 6 Sent Done Done false
18828 5552 Acked Flush 0 Sent Done Done false
18829 5553 Acked Read 2 Sent Done Done false
18830 5554 Acked Write 1 Sent Done Done false
18831 5555 Acked Read 10 Sent Done Done false
18832 5556 Acked Flush 0 Sent Done Done false
18833 5557 Acked Write 3 Sent Done Done false
18834 5558 Acked Write 1 Sent Done Done false
18835 5559 Acked Read 9 Sent Done Done false
18836 5560 Acked Write 2 Sent Done Done false
18837 5561 Acked Read 4 Sent Done Done false
18838 5562 Acked Write 10 Sent Done Done false
18839 5563 Acked Write 6 Sent Done Done false
18840 5564 Acked Write 2 Sent Done Done false
18841 5565 Acked Read 3 Sent Done Done false
18842 5566 Acked Write 4 Sent Done Done false
18843 5567 Acked Read 6 Sent Done Done false
18844 5568 Acked Flush 0 Sent Done Done false
18845 5569 Acked Read 6 Sent Done Done false
18846 5570 Acked Read 5 Sent Done Done false
18847 5571 Acked Flush 0 Sent Done Done false
18848 5572 Acked Read 5 Sent Done Done false
18849 5573 Acked Write 8 Sent Done Done false
18850 5574 Acked Read 4 Sent Done Done false
18851 5575 Acked Read 3 Sent Done Done false
18852 5576 Acked Read 6 Sent Done Done false
18853 5577 Acked Read 8 Sent Done Done false
18854 5578 Acked Read 10 Sent Done Done false
18855 5579 Acked Read 4 Sent Done Done false
18856 5580 Acked Read 6 Sent Done Done false
18857 5581 Acked Read 5 Sent Done Done false
18858 5582 Acked Write 7 Sent Done Done false
18859 5583 Acked Read 5 Sent Done Done false
18860 5584 Acked Write 9 Sent Done Done false
18861 5585 Acked Write 9 Sent Done Done false
18862 5586 Acked Read 7 Sent Done Done false
18863 5587 Acked Read 4 Sent Done Done false
18864 5588 Acked Read 1 Sent Done Done false
18865 5589 Acked Read 5 Sent Done Done false
18866 5590 Acked Read 4 Sent Done Done false
18867 5591 Acked Write 9 Sent Done Done false
18868 5592 Acked Read 8 Sent Done Done false
18869 5593 Acked Flush 0 Sent Done Done false
18870 5594 Acked Read 1 Sent Done Done false
18871 5595 Acked Write 2 Sent Done Done false
18872 5596 Acked Read 2 Sent Done Done false
18873 5597 Acked Write 6 Sent Done Done false
18874 5598 Acked Write 6 Sent Done Done false
18875 5599 Acked Read 9 Sent Done Done false
18876 5600 Acked Write 1 Sent Done Done false
18877 5601 Acked Read 9 Sent Done Done false
18878 5602 Acked Read 8 Sent Done Done false
18879 5603 Acked Read 9 Sent Done Done false
18880 5604 Acked Read 1 Sent Done Done false
18881 5605 Acked Write 3 Sent Done Done false
18882 5606 Acked Flush 0 Sent Done Done false
18883 5607 Acked Write 2 Sent Done Done false
18884 5608 Acked Write 9 Sent Done Done false
18885 5609 Acked Read 5 Sent Done Done false
18886 5610 Acked Read 4 Sent Done Done false
18887 5611 Acked Read 8 Sent Done Done false
18888 5612 Acked Flush 0 Sent Done Done false
18889 5613 Acked Read 4 Sent Done Done false
18890 5614 Acked Write 8 Sent Done Done false
18891 5615 Acked Write 8 Sent Done Done false
18892 5616 Acked Read 3 Sent Done Done false
18893 5617 Acked Flush 0 Sent Done Done false
18894 5618 Acked Read 10 Sent Done Done false
18895 5619 Acked Write 3 Sent Done Done false
18896 5620 Acked Read 7 Sent Done Done false
18897 5621 Acked Write 4 Sent Done Done false
18898 5622 Acked Write 2 Sent Done Done false
18899 5623 Acked Read 4 Sent Done Done false
18900 5624 Acked Read 10 Sent Done Done false
18901 5625 Acked Write 1 Sent Done Done false
18902 5626 Acked Write 6 Sent Done Done false
18903 5627 Acked Read 2 Sent Done Done false
18904 5628 Acked Read 8 Sent Done Done false
18905 5629 Acked Write 7 Sent Done Done false
18906 5630 Acked Read 9 Sent Done Done false
18907 5631 Acked Write 5 Sent Done Done false
18908 5632 Acked Write 10 Sent Done Done false
18909 5633 Acked Read 8 Sent Done Done false
18910 5634 Acked Read 8 Sent Done Done false
18911 5635 Acked Write 5 Sent Done Done false
18912 5636 Acked Write 5 Sent Done Done false
18913 5637 Acked Write 4 Sent Done Done false
18914 5638 Acked Write 10 Sent Done Done false
18915 5639 Acked Write 1 Sent Done Done false
18916 5640 Acked Write 9 Sent Done Done false
18917 5641 Acked Write 8 Sent Done Done false
18918 5642 Acked Write 8 Sent Done Done false
18919 5643 Acked Write 1 Sent Done Done false
18920 5644 Acked Write 10 Sent Done Done false
18921 5645 Acked Read 1 Sent Done Done false
18922 5646 Acked Write 6 Sent Done Done false
18923 5647 Acked Write 1 Sent Done Done false
18924 5648 Acked Write 9 Sent Done Done false
18925 5649 Acked Read 4 Sent Done Done false
18926 5650 Acked Read 5 Sent Done Done false
18927 5651 Acked Write 1 Sent Done Done false
18928 5652 Acked Read 1 Sent Done Done false
18929 5653 Acked Flush 0 Sent Done Done false
18930 5654 Acked Write 3 Sent Done Done false
18931 5655 Acked Write 1 Sent Done Done false
18932 5656 Acked Write 1 Sent Done Done false
18933 5657 Acked Read 7 Sent Done Done false
18934 5658 Acked Flush 0 Sent Done Done false
18935 5659 Acked Read 5 Sent Done Done false
18936 5660 Acked Read 9 Sent Done Done false
18937 5661 Acked Write 9 Sent Done Done false
18938 5662 Acked Read 7 Sent Done Done false
18939 5663 Acked Read 4 Sent Done Done false
18940 5664 Acked Write 6 Sent Done Done false
18941 5665 Acked Write 9 Sent Done Done false
18942 5666 Acked Write 6 Sent Done Done false
18943 5667 Acked Read 4 Sent Done Done false
18944 5668 Acked Write 3 Sent Done Done false
18945 5669 Acked Write 7 Sent Done Done false
18946 5670 Acked Read 10 Sent Done Done false
18947 5671 Acked Flush 0 Sent Done Done false
18948 5672 Acked Write 8 Sent Done Done false
18949 5673 Acked Flush 0 Sent Done Done false
18950 5674 Acked Write 3 Sent Done Done false
18951 5675 Acked Write 2 Sent Done Done false
18952 5676 Acked Flush 0 Sent Done Done false
18953 5677 Acked Write 1 Sent Done Done false
18954 5678 Acked Write 3 Sent Done Done false
18955 5679 Acked Read 10 Sent Done Done false
18956 5680 Acked Write 2 Sent Done Done false
18957 5681 Acked Read 9 Sent Done Done false
18958 5682 Acked Read 7 Sent Done Done false
18959 5683 Acked Write 2 Sent Done Done false
18960 5684 Acked Write 1 Sent Done Done false
18961 5685 Acked Write 1 Sent Done Done false
18962 5686 Acked Read 1 Sent Done Done false
18963 5687 Acked Read 5 Sent Done Done false
18964 5688 Acked Read 8 Sent Done Done false
18965 5689 Acked Read 8 Sent Done Done false
18966 5690 Acked Write 2 Sent Done Done false
18967 5691 Acked Read 6 Sent Done Done false
18968 5692 Acked Write 3 Sent Done Done false
18969 5693 Acked Read 7 Sent Done Done false
18970 5694 Acked Read 3 Sent Done Done false
18971 5695 Acked Write 5 Sent Done Done false
18972 5696 Acked Read 1 Sent Done Done false
18973 5697 Acked Read 3 Sent Done Done false
18974 5698 Acked Read 1 Sent Done Done false
18975 5699 Acked Read 3 Sent Done Done false
18976 5700 Acked Flush 0 Sent Done Done false
18977 5701 Acked Write 3 Sent Done Done false
18978 5702 Acked Read 6 Sent Done Done false
18979 5703 Acked Write 10 Sent Done Done false
18980 5704 Acked Read 6 Sent Done Done false
18981 5705 Acked Write 3 Sent Done Done false
18982 5706 Acked Write 2 Sent Done Done false
18983 5707 Acked Write 2 Sent Done Done false
18984 5708 Acked Read 10 Sent Done Done false
18985 5709 Acked Read 1 Sent Done Done false
18986 5710 Acked Write 4 Sent Done Done false
18987 5711 Acked Read 4 Sent Done Done false
18988 5712 Acked Write 8 Sent Done Done false
18989 5713 Acked Write 4 Sent Done Done false
18990 5714 Acked Read 4 Sent Done Done false
18991 5715 Acked Read 6 Sent Done Done false
18992 5716 Acked Read 6 Sent Done Done false
18993 5717 Acked Write 6 Sent Done Done false
18994 5718 Acked Read 5 Sent Done Done false
18995 5719 Acked Read 6 Sent Done Done false
18996 5720 Acked Read 5 Sent Done Done false
18997 5721 Acked Write 9 Sent Done Done false
18998 5722 Acked Read 1 Sent Done Done false
18999 5723 Acked Read 6 Sent Done Done false
19000 5724 Acked Read 4 Sent Done Done false
19001 5725 Acked Read 4 Sent Done Done false
19002 5726 Acked Write 1 Sent Done Done false
19003 5727 Acked Flush 0 Sent Done Done false
19004 5728 Acked Write 4 Sent Done Done false
19005 5729 Acked Write 6 Sent Done Done false
19006 5730 Acked Flush 0 Sent Done Done false
19007 5731 Acked Write 4 Sent Done Done false
19008 5732 Acked Write 4 Sent Done Done false
19009 5733 Acked Read 9 Sent Done Done false
19010 5734 Acked Write 6 Sent Done Done false
19011 5735 Acked Write 6 Sent Done Done false
19012 5736 Acked Read 4 Sent Done Done false
19013 5737 Acked Write 4 Sent Done Done false
19014 5738 Acked Write 4 Sent Done Done false
19015 5739 Acked Read 9 Sent Done Done false
19016 5740 Acked Read 5 Sent Done Done false
19017 5741 Acked Write 9 Sent Done Done false
19018 5742 Acked Read 9 Sent Done Done false
19019 5743 Acked Read 8 Sent Done Done false
19020 5744 Acked Read 4 Sent Done Done false
19021 5745 Acked Flush 0 Sent Done Done false
19022 5746 Acked Read 9 Sent Done Done false
19023 5747 Acked Write 10 Sent Done Done false
19024 5748 Acked Write 9 Sent Done Done false
19025 5749 Acked Read 4 Sent Done Done false
19026 5750 Acked Read 10 Sent Done Done false
19027 5751 Acked Read 4 Sent Done Done false
19028 5752 Acked Flush 0 Sent Done Done false
19029 5753 Acked Read 9 Sent Done Done false
19030 5754 Acked Write 2 Sent Done Done false
19031 5755 Acked Write 3 Sent Done Done false
19032 5756 Acked Write 4 Sent Done Done false
19033 5757 Acked Read 7 Sent Done Done false
19034 5758 Acked Read 6 Sent Done Done false
19035 5759 Acked Read 3 Sent Done Done false
19036 5760 Acked Write 6 Sent Done Done false
19037 5761 Acked Read 2 Sent Done Done false
19038 5762 Acked Write 1 Sent Done Done false
19039 5763 Acked Write 4 Sent Done Done false
19040 5764 Acked Write 9 Sent Done Done false
19041 5765 Acked Read 10 Sent Done Done false
19042 5766 Acked Write 2 Sent Done Done false
19043 5767 Acked Read 1 Sent Done Done false
19044 5768 Acked Read 1 Sent Done Done false
19045 5769 Acked Write 2 Sent Done Done false
19046 5770 Acked Flush 0 Sent Done Done false
19047 5771 Acked Read 3 Sent Done Done false
19048 5772 Acked Flush 0 Sent Done Done false
19049 5773 Acked Flush 0 Sent Done Done false
19050 5774 Acked Write 5 Sent Done Done false
19051 5775 Acked Read 1 Sent Done Done false
19052 5776 Acked Read 5 Sent Done Done false
19053 5777 Acked Read 9 Sent Done Done false
19054 5778 Acked Read 10 Sent Done Done false
19055 5779 Acked Read 9 Sent Done Done false
19056 5780 Acked Read 4 Sent Done Done false
19057 5781 Acked Write 2 Sent Done Done false
19058 5782 Acked Read 8 Sent Done Done false
19059 5783 Acked Read 8 Sent Done Done false
19060 5784 Acked Write 9 Sent Done Done false
19061 5785 Acked Read 1 Sent Done Done false
19062 5786 Acked Write 10 Sent Done Done false
19063 5787 Acked Write 5 Sent Done Done false
19064 5788 Acked Flush 0 Sent Done Done false
19065 5789 Acked Write 6 Sent Done Done false
19066 5790 Acked Read 3 Sent Done Done false
19067 5791 Acked Read 4 Sent Done Done false
19068 5792 Acked Read 4 Sent Done Done false
19069 5793 Acked Read 9 Sent Done Done false
19070 5794 Acked Read 7 Sent Done Done false
19071 5795 Acked Read 9 Sent Done Done false
19072 5796 Acked Read 10 Sent Done Done false
19073 5797 Acked Write 7 Sent Done Done false
19074 5798 Acked Read 4 Sent Done Done false
19075 5799 Acked Flush 0 Sent Done Done false
19076 5800 Acked Write 6 Sent Done Done false
19077 5801 Acked Write 9 Sent Done Done false
19078 5802 Acked Read 1 Sent Done Done false
19079 5803 Acked Write 9 Sent Done Done false
19080 5804 Acked Write 7 Sent Done Done false
19081 5805 Acked Write 10 Sent Done Done false
19082 5806 Acked Write 9 Sent Done Done false
19083 5807 Acked Write 9 Sent Done Done false
19084 5808 Acked Read 3 Sent Done Done false
19085 5809 Acked Read 10 Sent Done Done false
19086 5810 Acked Write 6 Sent Done Done false
19087 5811 Acked Read 8 Sent Done Done false
19088 5812 Acked Read 2 Sent Done Done false
19089 5813 Acked Write 9 Sent Done Done false
19090 5814 Acked Read 5 Sent Done Done false
19091 5815 Acked Write 3 Sent Done Done false
19092 5816 Acked Flush 0 Sent Done Done false
19093 5817 Acked Write 9 Sent Done Done false
19094 5818 Acked Write 7 Sent Done Done false
19095 5819 Acked Read 4 Sent Done Done false
19096 5820 Acked Read 7 Sent Done Done false
19097 5821 Acked Write 1 Sent Done Done false
19098 5822 Acked Flush 0 Sent Done Done false
19099 5823 Acked Write 9 Sent Done Done false
19100 5824 Acked Write 2 Sent Done Done false
19101 5825 Acked Read 5 Sent Done Done false
19102 5826 Acked Write 6 Sent Done Done false
19103 5827 Acked Read 9 Sent Done Done false
19104 5828 Acked Write 4 Sent Done Done false
19105 5829 Acked Flush 0 Sent Done Done false
19106 5830 Acked Flush 0 Sent Done Done false
19107 STATES DS:0 DS:1 DS:2 TOTAL
19108 Sent 301 0 0 301
19109 Done 0 301 301 602
19110 Skipped 0 0 0 0
19111 Error 0 0 0 0
19112 Last Flush: Some(JobId(5529)) Some(JobId(5830)) Some(JobId(5830))
19113 Downstairs last five completed: 5529 5528 5527 5526 5525
19114 Upstairs last five completed: 5830 5829 5828 5827 5826
19115 CLIENT: Up:0 ds:301 act:2
191162025-03-17T22:38:45.706ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191172025-03-17T22:38:46.671ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191182025-03-17T22:38:46.671ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53316) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191192025-03-17T22:38:46.672ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191202025-03-17T22:38:46.672ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191212025-03-17T22:38:46.672ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191222025-03-17T22:38:46.672ZINFOcrucible: send last flush ID to this DS: Some(JobId(5529)) = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191232025-03-17T22:38:46.672ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191242025-03-17T22:38:46.672ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5529)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191252025-03-17T22:38:46.672ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5529)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
19126 ----------------------------------------------------------------
19127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19128 Downstairs last five completed: 5830 5829 5828 5827 5826
19129 Upstairs last five completed: 5830 5829 5828 5827 5826
19130 CLIENT: Up:0 ds:0 act:3
19131 Replay: All jobs finished
19132 Main task has work to do, go find it
19133 got dsc Stop(1)
19134 stop 1
19135 Mar 17 22:38:48.921 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7b8bf427-4d50-4b4c-bf6d-c1ed2c8634f0, remote_addr: 127.0.0.[8820] Got stop action so:false kr:false
19136 1:35180, local_addr: 127.0.0.1:9998
19137 Mar 17 22:38:48.922 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/1, method: GET, req_id: 29a9e931-a65c-47ea-9a1d-b38ef97ef9bc, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
191382025-03-17T22:38:48.922ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191392025-03-17T22:38:48.922ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191402025-03-17T22:38:48.922ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
191412025-03-17T22:38:48.922ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
19142 [8820] Exited with: ExitStatus(unix_wait_status(9))
19143 [8820][1] reports Exit
19144 Mar 17 22:38:52.924 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 87b85e69-a800-4621-a67f-e177f9652d02, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
19145 001/300 Write block 3471 len 20480 data: 6 5 3 3 3
19146 002/300 Read block 238 len 4096
19147 003/300 Read block 4269 len 16384
19148 004/300 Write block 2573 len 4096 data: 6
19149 005/300 Write block 4424 len 8192 data: 6 7
19150 006/300 Read block 4427 len 8192
19151 007/300 Read block 2137 len 8192
19152 008/300 Write block 3290 len 32768 data: 4 4 4 4 4 4 3 3
19153 009/300 Read block 70 len 24576
19154 010/300 Read block 86 len 8192
19155 011/300 Read block 2751 len 20480
19156 012/300 Read block 2676 len 4096
19157 013/300 Read block 1253 len 32768
19158 014/300 Read block 2707 len 20480
19159 015/300 Write block 4832 len 8192 data: 4 3
19160 016/300 Write block 926 len 12288 data: 4 4 4
19161 017/300 Write block 4434 len 40960 data: 5 5 5 5 4 3 3 3 4 5
19162 018/300 Read block 165 len 36864
19163 019/300 Read block 4689 len 4096
19164 020/300 Write block 1507 len 32768 data: 5 6 6 5 4 6 6 4
19165 021/300 Write block 1754 len 32768 data: 3 3 3 3 3 4 2 3
19166 022/300 Read block 753 len 36864
19167 023/300 Flush
19168 024/300 Read block 1433 len 28672
19169 025/300 Read block 2333 len 32768
19170 026/300 Write block 3416 len 4096 data: 6
19171 027/300 Write block 451 len 8192 data: 6 4
19172 028/300 Read block 1968 len 24576
19173 029/300 Read block 834 len 32768
19174 030/300 Read block 4237 len 8192
19175 031/300 Flush
19176 032/300 Write block 2939 len 24576 data: 3 3 3 4 4 5
19177 033/300 Write block 4757 len 32768 data: 2 2 2 2 2 2 2 2
19178 034/300 Write block 2356 len 4096 data: 2
19179 035/300 Read block 554 len 40960
19180 036/300 Read block 2633 len 40960
19181 037/300 Write block 414 len 4096 data: 5
19182 038/300 Flush
19183 039/300 Read block 2847 len 16384
19184 040/300 Read block 3645 len 24576
19185 041/300 Write block 286 len 32768 data: 4 4 4 5 5 6 6 8
19186 042/300 Write block 1265 len 12288 data: 4 4 5
19187 043/300 Read block 4710 len 8192
19188 044/300 Flush
19189 045/300 Read block 4177 len 8192
19190 046/300 Read block 494 len 20480
19191 047/300 Write block 1426 len 4096 data: 5
19192 048/300 Read block 1336 len 12288
19193 049/300 Flush
19194 050/300 Write block 2353 len 40960 data: 3 2 2 3 2 3 3 2 2 2
19195 051/300 Read block 1691 len 40960
19196 052/300 Read block 3490 len 40960
19197 053/300 Flush
19198 054/300 Read block 606 len 8192
19199 055/300 Write block 4709 len 8192 data: 4 4
19200 056/300 Write block 2014 len 32768 data: 4 4 2 3 3 3 4 3
19201 057/300 Write block 3053 len 32768 data: 4 4 4 4 4 3 4 5
19202 058/300 Read block 296 len 20480
19203 059/300 Read block 2752 len 4096
19204 060/300 Write block 2066 len 28672 data: 3 3 4 5 4 6 6
19205 061/300 Read block 4670 len 24576
19206 062/300 Read block 575 len 24576
19207 063/300 Write block 745 len 28672 data: 5 5 6 8 6 6 3
19208 064/300 Write block 662 len 28672 data: 5 6 6 5 5 4 4
19209 065/300 Read block 3596 len 16384
19210 066/300 Write block 1729 len 40960 data: 4 4 5 5 5 5 4 5 4 4
19211 067/300 Read block 3801 len 32768
19212 068/300 Write block 2471 len 36864 data: 5 5 5 6 6 4 6 6 6
19213 069/300 Write block 4965 len 32768 data: 4 5 5 5 5 5 5 5
19214 070/300 Read block 4251 len 20480
19215 071/300 Flush
19216 072/300 Read block 954 len 8192
19217 073/300 Write block 4354 len 4096 data: 8
19218 074/300 Flush
19219 075/300 Read block 2101 len 20480
19220 076/300 Read block 1959 len 4096
19221 077/300 Write block 3154 len 20480 data: 5 3 2 2 3
19222 078/300 Read block 950 len 8192
19223 079/300 Write block 611 len 20480 data: 3 4 5 5 6
19224 080/300 Flush
19225 081/300 Read block 1029 len 40960
19226 082/300 Read block 1721 len 4096
19227 083/300 Read block 2346 len 16384
19228 084/300 Flush
19229 085/300 Write block 2157 len 28672 data: 3 4 3 4 5 5 4
19230 086/300 Read block 3102 len 36864
19231 087/300 Write block 3287 len 40960 data: 3 4 4 5 5 5 5 5 5 4
19232 088/300 Read block 2105 len 32768
19233 089/300 Read block 2633 len 12288
19234 090/300 Write block 2088 len 36864 data: 4 4 4 3 2 2 2 3 4
19235 091/300 Write block 1711 len 8192 data: 3 2
19236 092/300 Read block 3894 len 4096
19237 093/300 Read block 1147 len 4096
19238 094/300 Read block 3655 len 20480
19239 095/300 Read block 788 len 8192
19240 096/300 Write block 2891 len 16384 data: 3 3 3 2
19241 097/300 Flush
19242 098/300 Read block 2803 len 20480
19243 099/300 Write block 932 len 16384 data: 3 3 4 5
19244 100/300 Write block 2919 len 20480 data: 3 4 5 5 5
19245 101/300 Read block 1151 len 24576
19246 102/300 Read block 4671 len 32768
19247 103/300 Write block 275 len 28672 data: 5 5 4 4 3 3 3
19248 104/300 Read block 1183 len 12288
19249 105/300 Read block 2724 len 20480
19250 106/300 Read block 4122 len 24576
19251 107/300 Write block 4344 len 8192 data: 9 9
19252 108/300 Read block 3300 len 8192
19253 109/300 Flush
19254 110/300 Flush
19255 111/300 Read block 231 len 12288
19256 112/300 Write block 588 len 8192 data: 3 2
19257 113/300 Write block 1644 len 12288 data: 4 4 5
19258 114/300 Flush
19259 115/300 Read block 3641 len 12288
19260 116/300 Write block 1700 len 16384 data: 2 2 2 2
19261 117/300 Read block 3510 len 20480
19262 118/300 Read block 2117 len 16384
19263 119/300 Write block 4744 len 32768 data: 6 6 5 5 5 4 4 3
19264 120/300 Read block 4156 len 28672
19265 121/300 Flush
19266 122/300 Read block 3009 len 40960
19267 123/300 Write block 2459 len 24576 data: 2 3 3 3 4 4
19268 124/300 Write block 4879 len 36864 data: 3 3 4 4 4 4 4 3 3
19269 125/300 Write block 4560 len 12288 data: 2 3 4
19270 126/300 Write block 189 len 24576 data: 3 4 3 4 4 4
19271 127/300 Write block 4138 len 32768 data: 4 4 3 3 5 5 5 5
19272 128/300 Write block 533 len 24576 data: 5 8 8 3 3 3
19273 129/300 Read block 1502 len 8192
19274 130/300 Write block 1217 len 24576 data: 5 4 4 3 2 2
19275 131/300 Flush
19276 132/300 Read block 1004 len 32768
19277 133/300 Read block 2197 len 36864
19278 134/300 Read block 3311 len 24576
19279 135/300 Read block 2591 len 28672
19280 136/300 Flush
19281 137/300 Read block 3915 len 4096
19282 138/300 Read block 1812 len 36864
19283 139/300 Write block 3707 len 40960 data: 3 4 5 4 3 3 4 4 4 5
19284 140/300 Read block 736 len 28672
19285 141/300 Write block 753 len 4096 data: 4
19286 142/300 Read block 1206 len 32768
19287 143/300 Write block 44 len 20480 data: 3 3 3 3 3
19288 144/300 Write block 4151 len 36864 data: 2 3 3 4 3 4 5 5 5
19289 145/300 Write block 1990 len 28672 data: 4 4 4 3 3 3 3
19290 146/300 Read block 2943 len 36864
19291 147/300 Flush
19292 148/300 Write block 3465 len 36864 data: 3 3 4 4 4 5 7 6 4
19293 149/300 Read block 2698 len 28672
19294 150/300 Write block 1720 len 20480 data: 4 6 5 5 5
19295 151/300 Write block 3770 len 8192 data: 2 3
19296 152/300 Read block 1299 len 32768
19297 153/300 Read block 1511 len 4096
19298 154/300 Read block 2969 len 16384
19299 155/300 Write block 4134 len 12288 data: 4 5 4
19300 156/300 Read block 3971 len 40960
19301 157/300 Read block 3169 len 16384
19302 158/300 Read block 717 len 8192
19303 159/300 Read block 1873 len 12288
19304 160/300 Read block 298 len 12288
19305 161/300 Write block 2906 len 16384 data: 3 2 3 5
19306 162/300 Write block 2657 len 32768 data: 4 3 4 4 4 5 3 3
19307 163/300 Read block 1474 len 20480
19308 164/300 Flush
19309 165/300 Write block 1677 len 4096 data: 3
19310 166/300 Read block 4617 len 32768
19311 167/300 Write block 985 len 4096 data: 4
19312 168/300 Flush
19313 169/300 Read block 254 len 20480
19314 170/300 Read block 3070 len 8192
19315 171/300 Write block 1484 len 24576 data: 4 4 4 2 2 2
19316 172/300 Read block 3935 len 20480
19317 173/300 Read block 1062 len 12288
19318 174/300 Write block 3153 len 16384 data: 5 6 4 3
19319 175/300 Write block 3334 len 32768 data: 3 4 4 4 5 5 5 5
19320 176/300 Write block 346 len 32768 data: 2 2 2 3 3 2 2 3
19321 177/300 Write block 2083 len 16384 data: 3 3 3 5
19322 178/300 Write block 2283 len 12288 data: 3 3 3
19323 179/300 Read block 651 len 8192
19324 180/300 Write block 2976 len 16384 data: 5 5 7 8
19325 181/300 Write block 4200 len 24576 data: 4 2 2 3 3 3
19326 182/300 Read block 3079 len 20480
19327 183/300 Read block 2529 len 28672
19328 184/300 Write block 3870 len 24576 data: 3 4 4 6 9 9
19329 185/300 Write block 18 len 32768 data: 2 2 2 2 3 5 5 5
19330 186/300 Read block 4862 len 28672
19331 187/300 Read block 300 len 16384
19332 188/300 Read block 2397 len 20480
19333 189/300 Read block 3361 len 20480
19334 190/300 Read block 555 len 16384
19335 191/300 Read block 2758 len 32768
19336 192/300 Read block 3129 len 40960
19337 193/300 Read block 2281 len 32768
19338 194/300 Flush
19339 195/300 Read block 2047 len 8192
19340 196/300 Flush
19341 197/300 Write block 2684 len 20480 data: 3 3 3 4 4
19342 198/300 Read block 3868 len 20480
19343 199/300 Write block 1002 len 32768 data: 2 3 3 3 3 5 4 4
19344 200/300 Write block 4623 len 40960 data: 2 3 2 2 3 3 3 5 5 5
19345 201/300 Write block 2449 len 4096 data: 2
19346 202/300 Read block 2019 len 8192
19347 203/300 Write block 4680 len 12288 data: 3 3 3
19348 204/300 Write block 3808 len 16384 data: 5 5 6 6
19349 205/300 Read block 3970 len 28672
19350 206/300 Read block 1725 len 8192
19351 207/300 Read block 2256 len 24576
19352 208/300 Read block 705 len 32768
19353 209/300 Write block 895 len 28672 data: 3 2 3 3 4 4 6
19354 210/300 Write block 4867 len 36864 data: 6 5 6 5 6 6 5 5 4
19355 211/300 Read block 4277 len 20480
19356 212/300 Write block 183 len 12288 data: 3 3 3
19357 213/300 Write block 2156 len 16384 data: 3 4 5 4
19358 214/300 Flush
19359 215/300 Read block 669 len 4096
19360 216/300 Read block 2529 len 20480
19361 217/300 Write block 2768 len 8192 data: 5 5
19362 218/300 Read block 2741 len 20480
19363 219/300 Read block 2135 len 32768
19364 220/300 Write block 3385 len 8192 data: 7 7
19365 221/300 Read block 434 len 12288
19366 222/300 Read block 3679 len 32768
19367 223/300 Read block 3255 len 12288
19368 224/300 Write block 1716 len 32768 data: 3 3 2 3 5 7 6 6
19369 225/300 Read block 1315 len 24576
19370 226/300 Flush
19371 227/300 Flush
19372 228/300 Write block 1397 len 8192 data: 6 5
19373 229/300 Read block 1923 len 36864
19374 230/300 Flush
19375 231/300 Write block 481 len 40960 data: 4 5 4 6 5 5 5 5 4 4
19376 232/300 Read block 1826 len 28672
19377 233/300 Read block 1364 len 8192
19378 234/300 Read block 4712 len 16384
19379 235/300 Write block 1987 len 12288 data: 8 7 5
19380 236/300 Write block 9 len 8192 data: 4 5
19381 237/300 Write block 3666 len 4096 data: 3
19382 238/300 Read block 2400 len 8192
19383 239/300 Read block 2376 len 12288
19384 240/300 Write block 567 len 28672 data: 3 3 4 3 3 3 3
19385 241/300 Read block 2820 len 12288
19386 242/300 Read block 2218 len 32768
19387 243/300 Write block 2538 len 40960 data: 5 4 4 5 4 4 3 3 6 6
19388 244/300 Flush
19389 245/300 Write block 211 len 32768 data: 4 3 3 3 3 2 2 2
19390 246/300 Write block 4840 len 20480 data: 4 3 4 5 6
19391 247/300 Read block 2990 len 20480
19392 248/300 Read block 218 len 8192
19393 249/300 Write block 1405 len 28672 data: 4 2 2 2 3 3 4
19394 250/300 Read block 1873 len 24576
19395 251/300 Write block 3630 len 16384 data: 5 5 6 6
19396 252/300 Read block 3376 len 28672
19397 253/300 Read block 2923 len 16384
19398 254/300 Read block 4607 len 20480
19399 255/300 Write block 4666 len 20480 data: 3 4 4 3 5
19400 256/300 Read block 4701 len 8192
19401 257/300 Read block 4840 len 4096
19402 258/300 Read block 4526 len 36864
19403 259/300 Write block 1936 len 24576 data: 2 2 2 3 3 4
19404 260/300 Flush
19405 261/300 Read block 1773 len 32768
19406 262/300 Read block 4236 len 32768
19407 263/300 Read block 4375 len 16384
19408 264/300 Read block 4057 len 36864
19409 265/300 Write block 3768 len 4096 data: 2
19410 266/300 Write block 607 len 4096 data: 3
19411 267/300 Read block 4816 len 20480
19412 268/300 Read block 2330 len 28672
19413 269/300 Write block 2318 len 28672 data: 2 3 3 3 3 3 3
19414 270/300 Write block 4222 len 12288 data: 7 5 5
19415 271/300 Read block 226 len 20480
19416 272/300 Read block 4256 len 12288
19417 273/300 Read block 2219 len 40960
19418 274/300 Read block 3125 len 24576
19419 275/300 Write block 1622 len 32768 data: 5 5 6 6 5 5 4 4
19420 276/300 Read block 75 len 20480
19421 277/300 Read block 3596 len 12288
19422 278/300 Write block 3198 len 32768 data: 3 3 3 2 3 4 5 5
19423 279/300 Read block 2638 len 32768
19424 280/300 Read block 2838 len 32768
19425 281/300 Read block 3871 len 8192
19426 282/300 Flush
19427 283/300 Write block 1994 len 20480 data: 4 4 4 3 5
19428 284/300 Read block 4667 len 4096
19429 285/300 Read block 3968 len 20480
19430 286/300 Write block 2101 len 24576 data: 4 4 3 3 3 3
19431 287/300 Flush
19432 288/300 Write block 1646 len 20480 data: 6 4 5 4 4
19433 289/300 Read block 4393 len 12288
19434 290/300 Write block 4980 len 20480 data: 2 3 3 3 4
19435 291/300 Write block 3370 len 20480 data: 5 4 4 4 3
19436 292/300 Read block 76 len 24576
19437 293/300 Read block 70 len 8192
19438 294/300 Write block 487 len 12288 data: 6 6 5
19439 295/300 Write block 193 len 20480 data: 5 5 5 4 4
19440 296/300 Write block 3315 len 8192 data: 3 3
19441 297/300 Flush
19442 298/300 Read block 2932 len 4096
19443 299/300 Read block 2869 len 20480
19444 300/300 Read block 2541 len 16384
19445 301/300 Write block 3779 len 8192 data: 5 6
19446 Main task has work to do, go find it
19447 got dsc Start(1)
19448 start 1
19449 Mar 17 22:38:53.163 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3d1c30c5-c101-4f0f-89c1-15ed6eb3001a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
19450 Replay: started 1, returned:Ok(())
19451 ----------------------------------------------------------------
19452 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19453 States: Active Offline Active
19454 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19455 5831 Acked Write 5 Done Sent Done false
19456 5832 Acked Read 1 Done Sent Done false
19457 5833 Acked Read 4 Done Sent Done false
19458 5834 Acked Write 1 Done Sent Done false
19459 5835 Acked Write 2 Done Sent Done false
19460 5836 Acked Read 2 Done Sent Done false
19461 5837 Acked Read 2 Done Sent Done false
19462 5838 Acked Write 8 Done Sent Done false
19463 5839 Acked Read 6 Done Sent Done false
19464 5840 Acked Read 2 Done Sent Done false
19465 5841 Acked Read 5 Done Sent Done false
19466 5842 Acked Read 1 Done Sent Done false
19467 5843 Acked Read 8 Done Sent Done false
19468 5844 Acked Read 5 Done Sent Done false
19469 5845 Acked Write 2 Done Sent Done false
19470 5846 Acked Write 3 Done Sent Done false
19471 5847 Acked Write 10 Done Sent Done false
19472 5848 Acked Read 9 Done Sent Done false
19473 5849 Acked Read 1 Done Sent Done false
19474 5850 Acked Write 8 Done Sent Done false
19475 5851 Acked Write 8 Done Sent Done false
19476 5852 Acked Read 9 Done Sent Done false
19477 5853 Acked Flush 0 Done Sent Done false
19478 5854 Acked Read 7 Done Sent Done false
19479 5855 Acked Read 8 Done Sent Done false
19480 5856 Acked Write 1 Done Sent Done false
19481 5857 Acked Write 2 Done Sent Done false
19482 5858 Acked Read 6 Done Sent Done false
19483 5859 Acked Read 8 Done Sent Done false
19484 5860 Acked Read 2 Done Sent Done false
19485 5861 Acked Flush 0 Done Sent Done false
19486 5862 Acked Write 6 Done Sent Done false
19487 5863 Acked Write 8 Done Sent Done false
19488 5864 Acked Write 1 Done Sent Done false
19489 5865 Acked Read 10 Done Sent Done false
19490 5866 Acked Read 10 Done Sent Done false
19491 5867 Acked Write 1 Done Sent Done false
19492 5868 Acked Flush 0 Done Sent Done false
19493 5869 Acked Read 4 Done Sent Done false
19494 5870 Acked Read 6 Done Sent Done false
19495 5871 Acked Write 8 Done Sent Done false
19496 5872 Acked Write 3 Done Sent Done false
19497 5873 Acked Read 2 Done Sent Done false
19498 5874 Acked Flush 0 Done Sent Done false
19499 5875 Acked Read 2 Done Sent Done false
19500 5876 Acked Read 5 Done Sent Done false
19501 5877 Acked Write 1 Done Sent Done false
19502 5878 Acked Read 3 Done Sent Done false
19503 5879 Acked Flush 0 Done Sent Done false
19504 5880 Acked Write 10 Done Sent Done false
19505 5881 Acked Read 10 Done Sent Done false
19506 5882 Acked Read 10 Done Sent Done false
19507 5883 Acked Flush 0 Done Sent Done false
19508 5884 Acked Read 2 Done Sent Done false
19509 5885 Acked Write 2 Done Sent Done false
19510 5886 Acked Write 8 Done Sent Done false
19511 5887 Acked Write 8 Done Sent Done false
19512 5888 Acked Read 5 Done Sent Done false
19513 5889 Acked Read 1 Done Sent Done false
19514 5890 Acked Write 7 Done Sent Done false
19515 5891 Acked Read 6 Done Sent Done false
19516 5892 Acked Read 6 Done Sent Done false
19517 5893 Acked Write 7 Done Sent Done false
19518 5894 Acked Write 7 Done Sent Done false
19519 5895 Acked Read 4 Done Sent Done false
19520 5896 Acked Write 10 Done Sent Done false
19521 5897 Acked Read 8 Done Sent Done false
19522 5898 Acked Write 9 Done Sent Done false
19523 5899 Acked Write 8 Done Sent Done false
19524 5900 Acked Read 5 Done Sent Done false
19525 5901 Acked Flush 0 Done Sent Done false
19526 5902 Acked Read 2 Done Sent Done false
19527 5903 Acked Write 1 Done Sent Done false
19528 5904 Acked Flush 0 Done Sent Done false
19529 5905 Acked Read 5 Done Sent Done false
19530 5906 Acked Read 1 Done Sent Done false
19531 5907 Acked Write 5 Done Sent Done false
19532 5908 Acked Read 2 Done Sent Done false
19533 5909 Acked Write 5 Done Sent Done false
19534 5910 Acked Flush 0 Done Sent Done false
19535 5911 Acked Read 10 Done Sent Done false
19536 5912 Acked Read 1 Done Sent Done false
19537 5913 Acked Read 4 Done Sent Done false
19538 5914 Acked Flush 0 Done Sent Done false
19539 5915 Acked Write 7 Done Sent Done false
19540 5916 Acked Read 9 Done Sent Done false
19541 5917 Acked Write 10 Done Sent Done false
19542 5918 Acked Read 8 Done Sent Done false
19543 5919 Acked Read 3 Done Sent Done false
19544 5920 Acked Write 9 Done Sent Done false
19545 5921 Acked Write 2 Done Sent Done false
19546 5922 Acked Read 1 Done Sent Done false
19547 5923 Acked Read 1 Done Sent Done false
19548 5924 Acked Read 5 Done Sent Done false
19549 5925 Acked Read 2 Done Sent Done false
19550 5926 Acked Write 4 Done Sent Done false
19551 5927 Acked Flush 0 Done Sent Done false
19552 5928 Acked Read 5 Done Sent Done false
19553 5929 Acked Write 4 Done Sent Done false
19554 5930 Acked Write 5 Done Sent Done false
19555 5931 Acked Read 6 Done Sent Done false
19556 5932 Acked Read 8 Done Sent Done false
19557 5933 Acked Write 7 Done Sent Done false
19558 5934 Acked Read 3 Done Sent Done false
19559 5935 Acked Read 5 Done Sent Done false
19560 5936 Acked Read 6 Done Sent Done false
19561 5937 Acked Write 2 Done Sent Done false
19562 5938 Acked Read 2 Done Sent Done false
19563 5939 Acked Flush 0 Done Sent Done false
19564 5940 Acked Flush 0 Done Sent Done false
19565 5941 Acked Read 3 Done Sent Done false
19566 5942 Acked Write 2 Done Sent Done false
19567 5943 Acked Write 3 Done Sent Done false
19568 5944 Acked Flush 0 Done Sent Done false
19569 5945 Acked Read 3 Done Sent Done false
19570 5946 Acked Write 4 Done Sent Done false
19571 5947 Acked Read 5 Done Sent Done false
19572 5948 Acked Read 4 Done Sent Done false
19573 5949 Acked Write 8 Done Sent Done false
19574 5950 Acked Read 7 Done Sent Done false
19575 5951 Acked Flush 0 Done Sent Done false
19576 5952 Acked Read 10 Done Sent Done false
19577 5953 Acked Write 6 Done Sent Done false
19578 5954 Acked Write 9 Done Sent Done false
19579 5955 Acked Write 3 Done Sent Done false
19580 5956 Acked Write 6 Done Sent Done false
19581 5957 Acked Write 8 Done Sent Done false
19582 5958 Acked Write 6 Done Sent Done false
19583 5959 Acked Read 2 Done Sent Done false
19584 5960 Acked Write 6 Done Sent Done false
19585 5961 Acked Flush 0 Done Sent Done false
19586 5962 Acked Read 8 Done Sent Done false
19587 5963 Acked Read 9 Done Sent Done false
19588 5964 Acked Read 6 Done Sent Done false
19589 5965 Acked Read 7 Done Sent Done false
19590 5966 Acked Flush 0 Done Sent Done false
19591 5967 Acked Read 1 Done Sent Done false
19592 5968 Acked Read 9 Done Sent Done false
19593 5969 Acked Write 10 Done Sent Done false
19594 5970 Acked Read 7 Done Sent Done false
19595 5971 Acked Write 1 Done Sent Done false
19596 5972 Acked Read 8 Done Sent Done false
19597 5973 Acked Write 5 Done Sent Done false
19598 5974 Acked Write 9 Done Sent Done false
19599 5975 Acked Write 7 Done Sent Done false
19600 5976 Acked Read 9 Done Sent Done false
19601 5977 Acked Flush 0 Done Sent Done false
19602 5978 Acked Write 9 Done Sent Done false
19603 5979 Acked Read 7 Done Sent Done false
19604 5980 Acked Write 5 Done Sent Done false
19605 5981 Acked Write 2 Done Sent Done false
19606 5982 Acked Read 8 Done Sent Done false
19607 5983 Acked Read 1 Done Sent Done false
19608 5984 Acked Read 4 Done Sent Done false
19609 5985 Acked Write 3 Done Sent Done false
19610 5986 Acked Read 10 Done Sent Done false
19611 5987 Acked Read 4 Done Sent Done false
19612 5988 Acked Read 2 Done Sent Done false
19613 5989 Acked Read 3 Done Sent Done false
19614 5990 Acked Read 3 Done Sent Done false
19615 5991 Acked Write 4 Done Sent Done false
19616 5992 Acked Write 8 Done Sent Done false
19617 5993 Acked Read 5 Done Sent Done false
19618 5994 Acked Flush 0 Done Sent Done false
19619 5995 Acked Write 1 Done Sent Done false
19620 5996 Acked Read 8 Done Sent Done false
19621 5997 Acked Write 1 Done Sent Done false
19622 5998 Acked Flush 0 Done Sent Done false
19623 5999 Acked Read 5 Done Sent Done false
19624 6000 Acked Read 2 Done Sent Done false
19625 6001 Acked Write 6 Done Sent Done false
19626 6002 Acked Read 5 Done Sent Done false
19627 6003 Acked Read 3 Done Sent Done false
19628 6004 Acked Write 4 Done Sent Done false
19629 6005 Acked Write 8 Done Sent Done false
19630 6006 Acked Write 8 Done Sent Done false
19631 6007 Acked Write 4 Done Sent Done false
19632 6008 Acked Write 3 Done Sent Done false
19633 6009 Acked Read 2 Done Sent Done false
19634 6010 Acked Write 4 Done Sent Done false
19635 6011 Acked Write 6 Done Sent Done false
19636 6012 Acked Read 5 Done Sent Done false
19637 6013 Acked Read 7 Done Sent Done false
19638 6014 Acked Write 6 Done Sent Done false
19639 6015 Acked Write 8 Done Sent Done false
19640 6016 Acked Read 7 Done Sent Done false
19641 6017 Acked Read 4 Done Sent Done false
19642 6018 Acked Read 5 Done Sent Done false
19643 6019 Acked Read 5 Done Sent Done false
19644 6020 Acked Read 4 Done Sent Done false
19645 6021 Acked Read 8 Done Sent Done false
19646 6022 Acked Read 10 Done Sent Done false
19647 6023 Acked Read 8 Done Sent Done false
19648 6024 Acked Flush 0 Done Sent Done false
19649 6025 Acked Read 2 Done Sent Done false
19650 6026 Acked Flush 0 Done Sent Done false
19651 6027 Acked Write 5 Done Sent Done false
19652 6028 Acked Read 5 Done Sent Done false
19653 6029 Acked Write 8 Done Sent Done false
19654 6030 Acked Write 10 Done Sent Done false
19655 6031 Acked Write 1 Done Sent Done false
19656 6032 Acked Read 2 Done Sent Done false
19657 6033 Acked Write 3 Done Sent Done false
19658 6034 Acked Write 4 Done Sent Done false
19659 6035 Acked Read 7 Done Sent Done false
19660 6036 Acked Read 2 Done Sent Done false
19661 6037 Acked Read 6 Done Sent Done false
19662 6038 Acked Read 8 Done Sent Done false
19663 6039 Acked Write 7 Done Sent Done false
19664 6040 Acked Write 9 Done Sent Done false
19665 6041 Acked Read 5 Done Sent Done false
19666 6042 Acked Write 3 Done Sent Done false
19667 6043 Acked Write 4 Done Sent Done false
19668 6044 Acked Flush 0 Done Sent Done false
19669 6045 Acked Read 1 Done Sent Done false
19670 6046 Acked Read 5 Done Sent Done false
19671 6047 Acked Write 2 Done Sent Done false
19672 6048 Acked Read 5 Done Sent Done false
19673 6049 Acked Read 8 Done Sent Done false
19674 6050 Acked Write 2 Done Sent Done false
19675 6051 Acked Read 3 Done Sent Done false
19676 6052 Acked Read 8 Done Sent Done false
19677 6053 Acked Read 3 Done Sent Done false
19678 6054 Acked Write 8 Done Sent Done false
19679 6055 Acked Read 6 Done Sent Done false
19680 6056 Acked Flush 0 Done Sent Done false
19681 6057 Acked Flush 0 Done Sent Done false
19682 6058 Acked Write 2 Done Sent Done false
19683 6059 Acked Read 9 Done Sent Done false
19684 6060 Acked Flush 0 Done Sent Done false
19685 6061 Acked Write 10 Done Sent Done false
19686 6062 Acked Read 7 Done Sent Done false
19687 6063 Acked Read 2 Done Sent Done false
19688 6064 Acked Read 4 Done Sent Done false
19689 6065 Acked Write 3 Done Sent Done false
19690 6066 Acked Write 2 Done Sent Done false
19691 6067 Acked Write 1 Done Sent Done false
19692 6068 Acked Read 2 Done Sent Done false
19693 6069 Acked Read 3 Done Sent Done false
19694 6070 Acked Write 7 Done Sent Done false
19695 6071 Acked Read 3 Done Sent Done false
19696 6072 Acked Read 8 Done Sent Done false
19697 6073 Acked Write 10 Done Sent Done false
19698 6074 Acked Flush 0 Done Sent Done false
19699 6075 Acked Write 8 Done Sent Done false
19700 6076 Acked Write 5 Done Sent Done false
19701 6077 Acked Read 5 Done Sent Done false
19702 6078 Acked Read 2 Done Sent Done false
19703 6079 Acked Write 7 Done Sent Done false
19704 6080 Acked Read 6 Done Sent Done false
19705 6081 Acked Write 4 Done Sent Done false
19706 6082 Acked Read 7 Done Sent Done false
19707 6083 Acked Read 4 Done Sent Done false
19708 6084 Acked Read 5 Done Sent Done false
19709 6085 Acked Write 5 Done Sent Done false
19710 6086 Acked Read 2 Done Sent Done false
19711 6087 Acked Read 1 Done Sent Done false
19712 6088 Acked Read 9 Done Sent Done false
19713 6089 Acked Write 6 Done Sent Done false
19714 6090 Acked Flush 0 Done Sent Done false
19715 6091 Acked Read 8 Done Sent Done false
19716 6092 Acked Read 8 Done Sent Done false
19717 6093 Acked Read 4 Done Sent Done false
19718 6094 Acked Read 9 Done Sent Done false
19719 6095 Acked Write 1 Done Sent Done false
19720 6096 Acked Write 1 Done Sent Done false
19721 6097 Acked Read 5 Done Sent Done false
19722 6098 Acked Read 7 Done Sent Done false
19723 6099 Acked Write 7 Done Sent Done false
19724 6100 Acked Write 3 Done Sent Done false
19725 6101 Acked Read 5 Done Sent Done false
19726 6102 Acked Read 3 Done Sent Done false
19727 6103 Acked Read 10 Done Sent Done false
19728 6104 Acked Read 6 Done Sent Done false
19729 6105 Acked Write 8 Done Sent Done false
19730 6106 Acked Read 5 Done Sent Done false
19731 6107 Acked Read 3 Done Sent Done false
19732 6108 Acked Write 8 Done Sent Done false
19733 6109 Acked Read 8 Done Sent Done false
19734 6110 Acked Read 8 Done Sent Done false
19735 6111 Acked Read 2 Done Sent Done false
19736 6112 Acked Flush 0 Done Sent Done false
19737 6113 Acked Write 5 Done Sent Done false
19738 6114 Acked Read 1 Done Sent Done false
19739 6115 Acked Read 5 Done Sent Done false
19740 6116 Acked Write 6 Done Sent Done false
19741 6117 Acked Flush 0 Done Sent Done false
19742 6118 Acked Write 5 Done Sent Done false
19743 6119 Acked Read 3 Done Sent Done false
19744 6120 Acked Write 5 Done Sent Done false
19745 6121 Acked Write 5 Done Sent Done false
19746 6122 Acked Read 6 Done Sent Done false
19747 6123 Acked Read 2 Done Sent Done false
19748 6124 Acked Write 3 Done Sent Done false
19749 6125 Acked Write 5 Done Sent Done false
19750 6126 Acked Write 2 Done Sent Done false
19751 6127 Acked Flush 0 Done Sent Done false
19752 6128 Acked Read 1 Done Sent Done false
19753 6129 Acked Read 5 Done Sent Done false
19754 6130 Acked Read 4 Done Sent Done false
19755 6131 Acked Write 2 Done Sent Done false
19756 STATES DS:0 DS:1 DS:2 TOTAL
19757 Sent 0 301 0 301
19758 Done 301 0 301 602
19759 Skipped 0 0 0 0
19760 Error 0 0 0 0
19761 Last Flush: Some(JobId(6127)) Some(JobId(5830)) Some(JobId(6127))
19762 Downstairs last five completed: 5830 5829 5828 5827 5826
19763 Upstairs last five completed: 6131 6130 6129 6128 6127
19764 CLIENT: Up:0 ds:301 act:2
197652025-03-17T22:38:53.903ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
197662025-03-17T22:38:54.720ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
197672025-03-17T22:38:55.722ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
19768 ----------------------------------------------------------------
19769 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19770 States: Active Offline Active
19771 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19772 5831 Acked Write 5 Done Sent Done false
19773 5832 Acked Read 1 Done Sent Done false
19774 5833 Acked Read 4 Done Sent Done false
19775 5834 Acked Write 1 Done Sent Done false
19776 5835 Acked Write 2 Done Sent Done false
19777 5836 Acked Read 2 Done Sent Done false
19778 5837 Acked Read 2 Done Sent Done false
19779 5838 Acked Write 8 Done Sent Done false
19780 5839 Acked Read 6 Done Sent Done false
19781 5840 Acked Read 2 Done Sent Done false
19782 5841 Acked Read 5 Done Sent Done false
19783 5842 Acked Read 1 Done Sent Done false
19784 5843 Acked Read 8 Done Sent Done false
19785 5844 Acked Read 5 Done Sent Done false
19786 5845 Acked Write 2 Done Sent Done false
19787 5846 Acked Write 3 Done Sent Done false
19788 5847 Acked Write 10 Done Sent Done false
19789 5848 Acked Read 9 Done Sent Done false
19790 5849 Acked Read 1 Done Sent Done false
19791 5850 Acked Write 8 Done Sent Done false
19792 5851 Acked Write 8 Done Sent Done false
19793 5852 Acked Read 9 Done Sent Done false
19794 5853 Acked Flush 0 Done Sent Done false
19795 5854 Acked Read 7 Done Sent Done false
19796 5855 Acked Read 8 Done Sent Done false
19797 5856 Acked Write 1 Done Sent Done false
19798 5857 Acked Write 2 Done Sent Done false
19799 5858 Acked Read 6 Done Sent Done false
19800 5859 Acked Read 8 Done Sent Done false
19801 5860 Acked Read 2 Done Sent Done false
19802 5861 Acked Flush 0 Done Sent Done false
19803 5862 Acked Write 6 Done Sent Done false
19804 5863 Acked Write 8 Done Sent Done false
19805 5864 Acked Write 1 Done Sent Done false
19806 5865 Acked Read 10 Done Sent Done false
19807 5866 Acked Read 10 Done Sent Done false
19808 5867 Acked Write 1 Done Sent Done false
19809 5868 Acked Flush 0 Done Sent Done false
19810 5869 Acked Read 4 Done Sent Done false
19811 5870 Acked Read 6 Done Sent Done false
19812 5871 Acked Write 8 Done Sent Done false
19813 5872 Acked Write 3 Done Sent Done false
19814 5873 Acked Read 2 Done Sent Done false
19815 5874 Acked Flush 0 Done Sent Done false
19816 5875 Acked Read 2 Done Sent Done false
19817 5876 Acked Read 5 Done Sent Done false
19818 5877 Acked Write 1 Done Sent Done false
19819 5878 Acked Read 3 Done Sent Done false
19820 5879 Acked Flush 0 Done Sent Done false
19821 5880 Acked Write 10 Done Sent Done false
19822 5881 Acked Read 10 Done Sent Done false
19823 5882 Acked Read 10 Done Sent Done false
19824 5883 Acked Flush 0 Done Sent Done false
19825 5884 Acked Read 2 Done Sent Done false
19826 5885 Acked Write 2 Done Sent Done false
19827 5886 Acked Write 8 Done Sent Done false
19828 5887 Acked Write 8 Done Sent Done false
19829 5888 Acked Read 5 Done Sent Done false
19830 5889 Acked Read 1 Done Sent Done false
19831 5890 Acked Write 7 Done Sent Done false
19832 5891 Acked Read 6 Done Sent Done false
19833 5892 Acked Read 6 Done Sent Done false
19834 5893 Acked Write 7 Done Sent Done false
19835 5894 Acked Write 7 Done Sent Done false
19836 5895 Acked Read 4 Done Sent Done false
19837 5896 Acked Write 10 Done Sent Done false
19838 5897 Acked Read 8 Done Sent Done false
19839 5898 Acked Write 9 Done Sent Done false
19840 5899 Acked Write 8 Done Sent Done false
19841 5900 Acked Read 5 Done Sent Done false
19842 5901 Acked Flush 0 Done Sent Done false
19843 5902 Acked Read 2 Done Sent Done false
19844 5903 Acked Write 1 Done Sent Done false
19845 5904 Acked Flush 0 Done Sent Done false
19846 5905 Acked Read 5 Done Sent Done false
19847 5906 Acked Read 1 Done Sent Done false
19848 5907 Acked Write 5 Done Sent Done false
19849 5908 Acked Read 2 Done Sent Done false
19850 5909 Acked Write 5 Done Sent Done false
19851 5910 Acked Flush 0 Done Sent Done false
19852 5911 Acked Read 10 Done Sent Done false
19853 5912 Acked Read 1 Done Sent Done false
19854 5913 Acked Read 4 Done Sent Done false
19855 5914 Acked Flush 0 Done Sent Done false
19856 5915 Acked Write 7 Done Sent Done false
19857 5916 Acked Read 9 Done Sent Done false
19858 5917 Acked Write 10 Done Sent Done false
19859 5918 Acked Read 8 Done Sent Done false
19860 5919 Acked Read 3 Done Sent Done false
19861 5920 Acked Write 9 Done Sent Done false
19862 5921 Acked Write 2 Done Sent Done false
19863 5922 Acked Read 1 Done Sent Done false
19864 5923 Acked Read 1 Done Sent Done false
19865 5924 Acked Read 5 Done Sent Done false
19866 5925 Acked Read 2 Done Sent Done false
19867 5926 Acked Write 4 Done Sent Done false
19868 5927 Acked Flush 0 Done Sent Done false
19869 5928 Acked Read 5 Done Sent Done false
19870 5929 Acked Write 4 Done Sent Done false
19871 5930 Acked Write 5 Done Sent Done false
19872 5931 Acked Read 6 Done Sent Done false
19873 5932 Acked Read 8 Done Sent Done false
19874 5933 Acked Write 7 Done Sent Done false
19875 5934 Acked Read 3 Done Sent Done false
19876 5935 Acked Read 5 Done Sent Done false
19877 5936 Acked Read 6 Done Sent Done false
19878 5937 Acked Write 2 Done Sent Done false
19879 5938 Acked Read 2 Done Sent Done false
19880 5939 Acked Flush 0 Done Sent Done false
19881 5940 Acked Flush 0 Done Sent Done false
19882 5941 Acked Read 3 Done Sent Done false
19883 5942 Acked Write 2 Done Sent Done false
19884 5943 Acked Write 3 Done Sent Done false
19885 5944 Acked Flush 0 Done Sent Done false
19886 5945 Acked Read 3 Done Sent Done false
19887 5946 Acked Write 4 Done Sent Done false
19888 5947 Acked Read 5 Done Sent Done false
19889 5948 Acked Read 4 Done Sent Done false
19890 5949 Acked Write 8 Done Sent Done false
19891 5950 Acked Read 7 Done Sent Done false
19892 5951 Acked Flush 0 Done Sent Done false
19893 5952 Acked Read 10 Done Sent Done false
19894 5953 Acked Write 6 Done Sent Done false
19895 5954 Acked Write 9 Done Sent Done false
19896 5955 Acked Write 3 Done Sent Done false
19897 5956 Acked Write 6 Done Sent Done false
19898 5957 Acked Write 8 Done Sent Done false
19899 5958 Acked Write 6 Done Sent Done false
19900 5959 Acked Read 2 Done Sent Done false
19901 5960 Acked Write 6 Done Sent Done false
19902 5961 Acked Flush 0 Done Sent Done false
19903 5962 Acked Read 8 Done Sent Done false
19904 5963 Acked Read 9 Done Sent Done false
19905 5964 Acked Read 6 Done Sent Done false
19906 5965 Acked Read 7 Done Sent Done false
19907 5966 Acked Flush 0 Done Sent Done false
19908 5967 Acked Read 1 Done Sent Done false
19909 5968 Acked Read 9 Done Sent Done false
19910 5969 Acked Write 10 Done Sent Done false
19911 5970 Acked Read 7 Done Sent Done false
19912 5971 Acked Write 1 Done Sent Done false
19913 5972 Acked Read 8 Done Sent Done false
19914 5973 Acked Write 5 Done Sent Done false
19915 5974 Acked Write 9 Done Sent Done false
19916 5975 Acked Write 7 Done Sent Done false
19917 5976 Acked Read 9 Done Sent Done false
19918 5977 Acked Flush 0 Done Sent Done false
19919 5978 Acked Write 9 Done Sent Done false
19920 5979 Acked Read 7 Done Sent Done false
19921 5980 Acked Write 5 Done Sent Done false
19922 5981 Acked Write 2 Done Sent Done false
19923 5982 Acked Read 8 Done Sent Done false
19924 5983 Acked Read 1 Done Sent Done false
19925 5984 Acked Read 4 Done Sent Done false
19926 5985 Acked Write 3 Done Sent Done false
19927 5986 Acked Read 10 Done Sent Done false
19928 5987 Acked Read 4 Done Sent Done false
19929 5988 Acked Read 2 Done Sent Done false
19930 5989 Acked Read 3 Done Sent Done false
19931 5990 Acked Read 3 Done Sent Done false
19932 5991 Acked Write 4 Done Sent Done false
19933 5992 Acked Write 8 Done Sent Done false
19934 5993 Acked Read 5 Done Sent Done false
19935 5994 Acked Flush 0 Done Sent Done false
19936 5995 Acked Write 1 Done Sent Done false
19937 5996 Acked Read 8 Done Sent Done false
19938 5997 Acked Write 1 Done Sent Done false
19939 5998 Acked Flush 0 Done Sent Done false
19940 5999 Acked Read 5 Done Sent Done false
19941 6000 Acked Read 2 Done Sent Done false
19942 6001 Acked Write 6 Done Sent Done false
19943 6002 Acked Read 5 Done Sent Done false
19944 6003 Acked Read 3 Done Sent Done false
19945 6004 Acked Write 4 Done Sent Done false
19946 6005 Acked Write 8 Done Sent Done false
19947 6006 Acked Write 8 Done Sent Done false
19948 6007 Acked Write 4 Done Sent Done false
19949 6008 Acked Write 3 Done Sent Done false
19950 6009 Acked Read 2 Done Sent Done false
19951 6010 Acked Write 4 Done Sent Done false
19952 6011 Acked Write 6 Done Sent Done false
19953 6012 Acked Read 5 Done Sent Done false
19954 6013 Acked Read 7 Done Sent Done false
19955 6014 Acked Write 6 Done Sent Done false
19956 6015 Acked Write 8 Done Sent Done false
19957 6016 Acked Read 7 Done Sent Done false
19958 6017 Acked Read 4 Done Sent Done false
19959 6018 Acked Read 5 Done Sent Done false
19960 6019 Acked Read 5 Done Sent Done false
19961 6020 Acked Read 4 Done Sent Done false
19962 6021 Acked Read 8 Done Sent Done false
19963 6022 Acked Read 10 Done Sent Done false
19964 6023 Acked Read 8 Done Sent Done false
19965 6024 Acked Flush 0 Done Sent Done false
19966 6025 Acked Read 2 Done Sent Done false
19967 6026 Acked Flush 0 Done Sent Done false
19968 6027 Acked Write 5 Done Sent Done false
19969 6028 Acked Read 5 Done Sent Done false
19970 6029 Acked Write 8 Done Sent Done false
19971 6030 Acked Write 10 Done Sent Done false
19972 6031 Acked Write 1 Done Sent Done false
19973 6032 Acked Read 2 Done Sent Done false
19974 6033 Acked Write 3 Done Sent Done false
19975 6034 Acked Write 4 Done Sent Done false
19976 6035 Acked Read 7 Done Sent Done false
19977 6036 Acked Read 2 Done Sent Done false
19978 6037 Acked Read 6 Done Sent Done false
19979 6038 Acked Read 8 Done Sent Done false
19980 6039 Acked Write 7 Done Sent Done false
19981 6040 Acked Write 9 Done Sent Done false
19982 6041 Acked Read 5 Done Sent Done false
19983 6042 Acked Write 3 Done Sent Done false
19984 6043 Acked Write 4 Done Sent Done false
19985 6044 Acked Flush 0 Done Sent Done false
19986 6045 Acked Read 1 Done Sent Done false
19987 6046 Acked Read 5 Done Sent Done false
19988 6047 Acked Write 2 Done Sent Done false
19989 6048 Acked Read 5 Done Sent Done false
19990 6049 Acked Read 8 Done Sent Done false
19991 6050 Acked Write 2 Done Sent Done false
19992 6051 Acked Read 3 Done Sent Done false
19993 6052 Acked Read 8 Done Sent Done false
19994 6053 Acked Read 3 Done Sent Done false
19995 6054 Acked Write 8 Done Sent Done false
19996 6055 Acked Read 6 Done Sent Done false
19997 6056 Acked Flush 0 Done Sent Done false
19998 6057 Acked Flush 0 Done Sent Done false
19999 6058 Acked Write 2 Done Sent Done false
20000 6059 Acked Read 9 Done Sent Done false
20001 6060 Acked Flush 0 Done Sent Done false
20002 6061 Acked Write 10 Done Sent Done false
20003 6062 Acked Read 7 Done Sent Done false
20004 6063 Acked Read 2 Done Sent Done false
20005 6064 Acked Read 4 Done Sent Done false
20006 6065 Acked Write 3 Done Sent Done false
20007 6066 Acked Write 2 Done Sent Done false
20008 6067 Acked Write 1 Done Sent Done false
20009 6068 Acked Read 2 Done Sent Done false
20010 6069 Acked Read 3 Done Sent Done false
20011 6070 Acked Write 7 Done Sent Done false
20012 6071 Acked Read 3 Done Sent Done false
20013 6072 Acked Read 8 Done Sent Done false
20014 6073 Acked Write 10 Done Sent Done false
20015 6074 Acked Flush 0 Done Sent Done false
20016 6075 Acked Write 8 Done Sent Done false
20017 6076 Acked Write 5 Done Sent Done false
20018 6077 Acked Read 5 Done Sent Done false
20019 6078 Acked Read 2 Done Sent Done false
20020 6079 Acked Write 7 Done Sent Done false
20021 6080 Acked Read 6 Done Sent Done false
20022 6081 Acked Write 4 Done Sent Done false
20023 6082 Acked Read 7 Done Sent Done false
20024 6083 Acked Read 4 Done Sent Done false
20025 6084 Acked Read 5 Done Sent Done false
20026 6085 Acked Write 5 Done Sent Done false
20027 6086 Acked Read 2 Done Sent Done false
20028 6087 Acked Read 1 Done Sent Done false
20029 6088 Acked Read 9 Done Sent Done false
20030 6089 Acked Write 6 Done Sent Done false
20031 6090 Acked Flush 0 Done Sent Done false
20032 6091 Acked Read 8 Done Sent Done false
20033 6092 Acked Read 8 Done Sent Done false
20034 6093 Acked Read 4 Done Sent Done false
20035 6094 Acked Read 9 Done Sent Done false
20036 6095 Acked Write 1 Done Sent Done false
20037 6096 Acked Write 1 Done Sent Done false
20038 6097 Acked Read 5 Done Sent Done false
20039 6098 Acked Read 7 Done Sent Done false
20040 6099 Acked Write 7 Done Sent Done false
20041 6100 Acked Write 3 Done Sent Done false
20042 6101 Acked Read 5 Done Sent Done false
20043 6102 Acked Read 3 Done Sent Done false
20044 6103 Acked Read 10 Done Sent Done false
20045 6104 Acked Read 6 Done Sent Done false
20046 6105 Acked Write 8 Done Sent Done false
20047 6106 Acked Read 5 Done Sent Done false
20048 6107 Acked Read 3 Done Sent Done false
20049 6108 Acked Write 8 Done Sent Done false
20050 6109 Acked Read 8 Done Sent Done false
20051 6110 Acked Read 8 Done Sent Done false
20052 6111 Acked Read 2 Done Sent Done false
20053 6112 Acked Flush 0 Done Sent Done false
20054 6113 Acked Write 5 Done Sent Done false
20055 6114 Acked Read 1 Done Sent Done false
20056 6115 Acked Read 5 Done Sent Done false
20057 6116 Acked Write 6 Done Sent Done false
20058 6117 Acked Flush 0 Done Sent Done false
20059 6118 Acked Write 5 Done Sent Done false
20060 6119 Acked Read 3 Done Sent Done false
20061 6120 Acked Write 5 Done Sent Done false
20062 6121 Acked Write 5 Done Sent Done false
20063 6122 Acked Read 6 Done Sent Done false
20064 6123 Acked Read 2 Done Sent Done false
20065 6124 Acked Write 3 Done Sent Done false
20066 6125 Acked Write 5 Done Sent Done false
20067 6126 Acked Write 2 Done Sent Done false
20068 6127 Acked Flush 0 Done Sent Done false
20069 6128 Acked Read 1 Done Sent Done false
20070 6129 Acked Read 5 Done Sent Done false
20071 6130 Acked Read 4 Done Sent Done false
20072 6131 Acked Write 2 Done Sent Done false
20073 STATES DS:0 DS:1 DS:2 TOTAL
20074 Sent 0 301 0 301
20075 Done 301 0 301 602
20076 Skipped 0 0 0 0
20077 Error 0 0 0 0
20078 Last Flush: Some(JobId(6127)) Some(JobId(5830)) Some(JobId(6127))
20079 Downstairs last five completed: 5830 5829 5828 5827 5826
20080 Upstairs last five completed: 6131 6130 6129 6128 6127
20081 CLIENT: Up:0 ds:301 act:2
200822025-03-17T22:38:57.726ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200832025-03-17T22:38:58.903ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200842025-03-17T22:38:58.922ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200852025-03-17T22:38:58.922ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200862025-03-17T22:38:58.923ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200872025-03-17T22:38:58.923ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200882025-03-17T22:38:58.923ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200892025-03-17T22:38:59.730ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
200902025-03-17T22:39:00.732ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
20091 ----------------------------------------------------------------
20092 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20093 States: Active Offline Active
20094 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20095 5831 Acked Write 5 Done Sent Done false
20096 5832 Acked Read 1 Done Sent Done false
20097 5833 Acked Read 4 Done Sent Done false
20098 5834 Acked Write 1 Done Sent Done false
20099 5835 Acked Write 2 Done Sent Done false
20100 5836 Acked Read 2 Done Sent Done false
20101 5837 Acked Read 2 Done Sent Done false
20102 5838 Acked Write 8 Done Sent Done false
20103 5839 Acked Read 6 Done Sent Done false
20104 5840 Acked Read 2 Done Sent Done false
20105 5841 Acked Read 5 Done Sent Done false
20106 5842 Acked Read 1 Done Sent Done false
20107 5843 Acked Read 8 Done Sent Done false
20108 5844 Acked Read 5 Done Sent Done false
20109 5845 Acked Write 2 Done Sent Done false
20110 5846 Acked Write 3 Done Sent Done false
20111 5847 Acked Write 10 Done Sent Done false
20112 5848 Acked Read 9 Done Sent Done false
20113 5849 Acked Read 1 Done Sent Done false
20114 5850 Acked Write 8 Done Sent Done false
20115 5851 Acked Write 8 Done Sent Done false
20116 5852 Acked Read 9 Done Sent Done false
20117 5853 Acked Flush 0 Done Sent Done false
20118 5854 Acked Read 7 Done Sent Done false
20119 5855 Acked Read 8 Done Sent Done false
20120 5856 Acked Write 1 Done Sent Done false
20121 5857 Acked Write 2 Done Sent Done false
20122 5858 Acked Read 6 Done Sent Done false
20123 5859 Acked Read 8 Done Sent Done false
20124 5860 Acked Read 2 Done Sent Done false
20125 5861 Acked Flush 0 Done Sent Done false
20126 5862 Acked Write 6 Done Sent Done false
20127 5863 Acked Write 8 Done Sent Done false
20128 5864 Acked Write 1 Done Sent Done false
20129 5865 Acked Read 10 Done Sent Done false
20130 5866 Acked Read 10 Done Sent Done false
20131 5867 Acked Write 1 Done Sent Done false
20132 5868 Acked Flush 0 Done Sent Done false
20133 5869 Acked Read 4 Done Sent Done false
20134 5870 Acked Read 6 Done Sent Done false
20135 5871 Acked Write 8 Done Sent Done false
20136 5872 Acked Write 3 Done Sent Done false
20137 5873 Acked Read 2 Done Sent Done false
20138 5874 Acked Flush 0 Done Sent Done false
20139 5875 Acked Read 2 Done Sent Done false
20140 5876 Acked Read 5 Done Sent Done false
20141 5877 Acked Write 1 Done Sent Done false
20142 5878 Acked Read 3 Done Sent Done false
20143 5879 Acked Flush 0 Done Sent Done false
20144 5880 Acked Write 10 Done Sent Done false
20145 5881 Acked Read 10 Done Sent Done false
20146 5882 Acked Read 10 Done Sent Done false
20147 5883 Acked Flush 0 Done Sent Done false
20148 5884 Acked Read 2 Done Sent Done false
20149 5885 Acked Write 2 Done Sent Done false
20150 5886 Acked Write 8 Done Sent Done false
20151 5887 Acked Write 8 Done Sent Done false
20152 5888 Acked Read 5 Done Sent Done false
20153 5889 Acked Read 1 Done Sent Done false
20154 5890 Acked Write 7 Done Sent Done false
20155 5891 Acked Read 6 Done Sent Done false
20156 5892 Acked Read 6 Done Sent Done false
20157 5893 Acked Write 7 Done Sent Done false
20158 5894 Acked Write 7 Done Sent Done false
20159 5895 Acked Read 4 Done Sent Done false
20160 5896 Acked Write 10 Done Sent Done false
20161 5897 Acked Read 8 Done Sent Done false
20162 5898 Acked Write 9 Done Sent Done false
20163 5899 Acked Write 8 Done Sent Done false
20164 5900 Acked Read 5 Done Sent Done false
20165 5901 Acked Flush 0 Done Sent Done false
20166 5902 Acked Read 2 Done Sent Done false
20167 5903 Acked Write 1 Done Sent Done false
20168 5904 Acked Flush 0 Done Sent Done false
20169 5905 Acked Read 5 Done Sent Done false
20170 5906 Acked Read 1 Done Sent Done false
20171 5907 Acked Write 5 Done Sent Done false
20172 5908 Acked Read 2 Done Sent Done false
20173 5909 Acked Write 5 Done Sent Done false
20174 5910 Acked Flush 0 Done Sent Done false
20175 5911 Acked Read 10 Done Sent Done false
20176 5912 Acked Read 1 Done Sent Done false
20177 5913 Acked Read 4 Done Sent Done false
20178 5914 Acked Flush 0 Done Sent Done false
20179 5915 Acked Write 7 Done Sent Done false
20180 5916 Acked Read 9 Done Sent Done false
20181 5917 Acked Write 10 Done Sent Done false
20182 5918 Acked Read 8 Done Sent Done false
20183 5919 Acked Read 3 Done Sent Done false
20184 5920 Acked Write 9 Done Sent Done false
20185 5921 Acked Write 2 Done Sent Done false
20186 5922 Acked Read 1 Done Sent Done false
20187 5923 Acked Read 1 Done Sent Done false
20188 5924 Acked Read 5 Done Sent Done false
20189 5925 Acked Read 2 Done Sent Done false
20190 5926 Acked Write 4 Done Sent Done false
20191 5927 Acked Flush 0 Done Sent Done false
20192 5928 Acked Read 5 Done Sent Done false
20193 5929 Acked Write 4 Done Sent Done false
20194 5930 Acked Write 5 Done Sent Done false
20195 5931 Acked Read 6 Done Sent Done false
20196 5932 Acked Read 8 Done Sent Done false
20197 5933 Acked Write 7 Done Sent Done false
20198 5934 Acked Read 3 Done Sent Done false
20199 5935 Acked Read 5 Done Sent Done false
20200 5936 Acked Read 6 Done Sent Done false
20201 5937 Acked Write 2 Done Sent Done false
20202 5938 Acked Read 2 Done Sent Done false
20203 5939 Acked Flush 0 Done Sent Done false
20204 5940 Acked Flush 0 Done Sent Done false
20205 5941 Acked Read 3 Done Sent Done false
20206 5942 Acked Write 2 Done Sent Done false
20207 5943 Acked Write 3 Done Sent Done false
20208 5944 Acked Flush 0 Done Sent Done false
20209 5945 Acked Read 3 Done Sent Done false
20210 5946 Acked Write 4 Done Sent Done false
20211 5947 Acked Read 5 Done Sent Done false
20212 5948 Acked Read 4 Done Sent Done false
20213 5949 Acked Write 8 Done Sent Done false
20214 5950 Acked Read 7 Done Sent Done false
20215 5951 Acked Flush 0 Done Sent Done false
20216 5952 Acked Read 10 Done Sent Done false
20217 5953 Acked Write 6 Done Sent Done false
20218 5954 Acked Write 9 Done Sent Done false
20219 5955 Acked Write 3 Done Sent Done false
20220 5956 Acked Write 6 Done Sent Done false
20221 5957 Acked Write 8 Done Sent Done false
20222 5958 Acked Write 6 Done Sent Done false
20223 5959 Acked Read 2 Done Sent Done false
20224 5960 Acked Write 6 Done Sent Done false
20225 5961 Acked Flush 0 Done Sent Done false
20226 5962 Acked Read 8 Done Sent Done false
20227 5963 Acked Read 9 Done Sent Done false
20228 5964 Acked Read 6 Done Sent Done false
20229 5965 Acked Read 7 Done Sent Done false
20230 5966 Acked Flush 0 Done Sent Done false
20231 5967 Acked Read 1 Done Sent Done false
20232 5968 Acked Read 9 Done Sent Done false
20233 5969 Acked Write 10 Done Sent Done false
20234 5970 Acked Read 7 Done Sent Done false
20235 5971 Acked Write 1 Done Sent Done false
20236 5972 Acked Read 8 Done Sent Done false
20237 5973 Acked Write 5 Done Sent Done false
20238 5974 Acked Write 9 Done Sent Done false
20239 5975 Acked Write 7 Done Sent Done false
20240 5976 Acked Read 9 Done Sent Done false
20241 5977 Acked Flush 0 Done Sent Done false
20242 5978 Acked Write 9 Done Sent Done false
20243 5979 Acked Read 7 Done Sent Done false
20244 5980 Acked Write 5 Done Sent Done false
20245 5981 Acked Write 2 Done Sent Done false
20246 5982 Acked Read 8 Done Sent Done false
20247 5983 Acked Read 1 Done Sent Done false
20248 5984 Acked Read 4 Done Sent Done false
20249 5985 Acked Write 3 Done Sent Done false
20250 5986 Acked Read 10 Done Sent Done false
20251 5987 Acked Read 4 Done Sent Done false
20252 5988 Acked Read 2 Done Sent Done false
20253 5989 Acked Read 3 Done Sent Done false
20254 5990 Acked Read 3 Done Sent Done false
20255 5991 Acked Write 4 Done Sent Done false
20256 5992 Acked Write 8 Done Sent Done false
20257 5993 Acked Read 5 Done Sent Done false
20258 5994 Acked Flush 0 Done Sent Done false
20259 5995 Acked Write 1 Done Sent Done false
20260 5996 Acked Read 8 Done Sent Done false
20261 5997 Acked Write 1 Done Sent Done false
20262 5998 Acked Flush 0 Done Sent Done false
20263 5999 Acked Read 5 Done Sent Done false
20264 6000 Acked Read 2 Done Sent Done false
20265 6001 Acked Write 6 Done Sent Done false
20266 6002 Acked Read 5 Done Sent Done false
20267 6003 Acked Read 3 Done Sent Done false
20268 6004 Acked Write 4 Done Sent Done false
20269 6005 Acked Write 8 Done Sent Done false
20270 6006 Acked Write 8 Done Sent Done false
20271 6007 Acked Write 4 Done Sent Done false
20272 6008 Acked Write 3 Done Sent Done false
20273 6009 Acked Read 2 Done Sent Done false
20274 6010 Acked Write 4 Done Sent Done false
20275 6011 Acked Write 6 Done Sent Done false
20276 6012 Acked Read 5 Done Sent Done false
20277 6013 Acked Read 7 Done Sent Done false
20278 6014 Acked Write 6 Done Sent Done false
20279 6015 Acked Write 8 Done Sent Done false
20280 6016 Acked Read 7 Done Sent Done false
20281 6017 Acked Read 4 Done Sent Done false
20282 6018 Acked Read 5 Done Sent Done false
20283 6019 Acked Read 5 Done Sent Done false
20284 6020 Acked Read 4 Done Sent Done false
20285 6021 Acked Read 8 Done Sent Done false
20286 6022 Acked Read 10 Done Sent Done false
20287 6023 Acked Read 8 Done Sent Done false
20288 6024 Acked Flush 0 Done Sent Done false
20289 6025 Acked Read 2 Done Sent Done false
20290 6026 Acked Flush 0 Done Sent Done false
20291 6027 Acked Write 5 Done Sent Done false
20292 6028 Acked Read 5 Done Sent Done false
20293 6029 Acked Write 8 Done Sent Done false
20294 6030 Acked Write 10 Done Sent Done false
20295 6031 Acked Write 1 Done Sent Done false
20296 6032 Acked Read 2 Done Sent Done false
20297 6033 Acked Write 3 Done Sent Done false
20298 6034 Acked Write 4 Done Sent Done false
20299 6035 Acked Read 7 Done Sent Done false
20300 6036 Acked Read 2 Done Sent Done false
20301 6037 Acked Read 6 Done Sent Done false
20302 6038 Acked Read 8 Done Sent Done false
20303 6039 Acked Write 7 Done Sent Done false
20304 6040 Acked Write 9 Done Sent Done false
20305 6041 Acked Read 5 Done Sent Done false
20306 6042 Acked Write 3 Done Sent Done false
20307 6043 Acked Write 4 Done Sent Done false
20308 6044 Acked Flush 0 Done Sent Done false
20309 6045 Acked Read 1 Done Sent Done false
20310 6046 Acked Read 5 Done Sent Done false
20311 6047 Acked Write 2 Done Sent Done false
20312 6048 Acked Read 5 Done Sent Done false
20313 6049 Acked Read 8 Done Sent Done false
20314 6050 Acked Write 2 Done Sent Done false
20315 6051 Acked Read 3 Done Sent Done false
20316 6052 Acked Read 8 Done Sent Done false
20317 6053 Acked Read 3 Done Sent Done false
20318 6054 Acked Write 8 Done Sent Done false
20319 6055 Acked Read 6 Done Sent Done false
20320 6056 Acked Flush 0 Done Sent Done false
20321 6057 Acked Flush 0 Done Sent Done false
20322 6058 Acked Write 2 Done Sent Done false
20323 6059 Acked Read 9 Done Sent Done false
20324 6060 Acked Flush 0 Done Sent Done false
20325 6061 Acked Write 10 Done Sent Done false
20326 6062 Acked Read 7 Done Sent Done false
20327 6063 Acked Read 2 Done Sent Done false
20328 6064 Acked Read 4 Done Sent Done false
20329 6065 Acked Write 3 Done Sent Done false
20330 6066 Acked Write 2 Done Sent Done false
20331 6067 Acked Write 1 Done Sent Done false
20332 6068 Acked Read 2 Done Sent Done false
20333 6069 Acked Read 3 Done Sent Done false
20334 6070 Acked Write 7 Done Sent Done false
20335 6071 Acked Read 3 Done Sent Done false
20336 6072 Acked Read 8 Done Sent Done false
20337 6073 Acked Write 10 Done Sent Done false
20338 6074 Acked Flush 0 Done Sent Done false
20339 6075 Acked Write 8 Done Sent Done false
20340 6076 Acked Write 5 Done Sent Done false
20341 6077 Acked Read 5 Done Sent Done false
20342 6078 Acked Read 2 Done Sent Done false
20343 6079 Acked Write 7 Done Sent Done false
20344 6080 Acked Read 6 Done Sent Done false
20345 6081 Acked Write 4 Done Sent Done false
20346 6082 Acked Read 7 Done Sent Done false
20347 6083 Acked Read 4 Done Sent Done false
20348 6084 Acked Read 5 Done Sent Done false
20349 6085 Acked Write 5 Done Sent Done false
20350 6086 Acked Read 2 Done Sent Done false
20351 6087 Acked Read 1 Done Sent Done false
20352 6088 Acked Read 9 Done Sent Done false
20353 6089 Acked Write 6 Done Sent Done false
20354 6090 Acked Flush 0 Done Sent Done false
20355 6091 Acked Read 8 Done Sent Done false
20356 6092 Acked Read 8 Done Sent Done false
20357 6093 Acked Read 4 Done Sent Done false
20358 6094 Acked Read 9 Done Sent Done false
20359 6095 Acked Write 1 Done Sent Done false
20360 6096 Acked Write 1 Done Sent Done false
20361 6097 Acked Read 5 Done Sent Done false
20362 6098 Acked Read 7 Done Sent Done false
20363 6099 Acked Write 7 Done Sent Done false
20364 6100 Acked Write 3 Done Sent Done false
20365 6101 Acked Read 5 Done Sent Done false
20366 6102 Acked Read 3 Done Sent Done false
20367 6103 Acked Read 10 Done Sent Done false
20368 6104 Acked Read 6 Done Sent Done false
20369 6105 Acked Write 8 Done Sent Done false
20370 6106 Acked Read 5 Done Sent Done false
20371 6107 Acked Read 3 Done Sent Done false
20372 6108 Acked Write 8 Done Sent Done false
20373 6109 Acked Read 8 Done Sent Done false
20374 6110 Acked Read 8 Done Sent Done false
20375 6111 Acked Read 2 Done Sent Done false
20376 6112 Acked Flush 0 Done Sent Done false
20377 6113 Acked Write 5 Done Sent Done false
20378 6114 Acked Read 1 Done Sent Done false
20379 6115 Acked Read 5 Done Sent Done false
20380 6116 Acked Write 6 Done Sent Done false
20381 6117 Acked Flush 0 Done Sent Done false
20382 6118 Acked Write 5 Done Sent Done false
20383 6119 Acked Read 3 Done Sent Done false
20384 6120 Acked Write 5 Done Sent Done false
20385 6121 Acked Write 5 Done Sent Done false
20386 6122 Acked Read 6 Done Sent Done false
20387 6123 Acked Read 2 Done Sent Done false
20388 6124 Acked Write 3 Done Sent Done false
20389 6125 Acked Write 5 Done Sent Done false
20390 6126 Acked Write 2 Done Sent Done false
20391 6127 Acked Flush 0 Done Sent Done false
20392 6128 Acked Read 1 Done Sent Done false
20393 6129 Acked Read 5 Done Sent Done false
20394 6130 Acked Read 4 Done Sent Done false
20395 6131 Acked Write 2 Done Sent Done false
20396 STATES DS:0 DS:1 DS:2 TOTAL
20397 Sent 0 301 0 301
20398 Done 301 0 301 602
20399 Skipped 0 0 0 0
20400 Error 0 0 0 0
20401 Last Flush: Some(JobId(6127)) Some(JobId(5830)) Some(JobId(6127))
20402 Downstairs last five completed: 5830 5829 5828 5827 5826
20403 Upstairs last five completed: 6131 6130 6129 6128 6127
20404 CLIENT: Up:0 ds:301 act:2
204052025-03-17T22:39:02.176ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
204062025-03-17T22:39:02.736ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
20407 [8820] got start action
20408 [8820] I am going to restart
20409 Starting downstairs at port 8820
20410 Make output file at "/tmp/dsc/downstairs-8820.txt"
20411 [8820][1] reports Starting
20412 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
20413 [8820][1] reports Running
204142025-03-17T22:39:03.905ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
204152025-03-17T22:39:04.739ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
20416 ----------------------------------------------------------------
20417 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20418 States: Active Offline Active
20419 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20420 5831 Acked Write 5 Done Sent Done false
20421 5832 Acked Read 1 Done Sent Done false
20422 5833 Acked Read 4 Done Sent Done false
20423 5834 Acked Write 1 Done Sent Done false
20424 5835 Acked Write 2 Done Sent Done false
20425 5836 Acked Read 2 Done Sent Done false
20426 5837 Acked Read 2 Done Sent Done false
20427 5838 Acked Write 8 Done Sent Done false
20428 5839 Acked Read 6 Done Sent Done false
20429 5840 Acked Read 2 Done Sent Done false
20430 5841 Acked Read 5 Done Sent Done false
20431 5842 Acked Read 1 Done Sent Done false
20432 5843 Acked Read 8 Done Sent Done false
20433 5844 Acked Read 5 Done Sent Done false
20434 5845 Acked Write 2 Done Sent Done false
20435 5846 Acked Write 3 Done Sent Done false
20436 5847 Acked Write 10 Done Sent Done false
20437 5848 Acked Read 9 Done Sent Done false
20438 5849 Acked Read 1 Done Sent Done false
20439 5850 Acked Write 8 Done Sent Done false
20440 5851 Acked Write 8 Done Sent Done false
20441 5852 Acked Read 9 Done Sent Done false
20442 5853 Acked Flush 0 Done Sent Done false
20443 5854 Acked Read 7 Done Sent Done false
20444 5855 Acked Read 8 Done Sent Done false
20445 5856 Acked Write 1 Done Sent Done false
20446 5857 Acked Write 2 Done Sent Done false
20447 5858 Acked Read 6 Done Sent Done false
20448 5859 Acked Read 8 Done Sent Done false
20449 5860 Acked Read 2 Done Sent Done false
20450 5861 Acked Flush 0 Done Sent Done false
20451 5862 Acked Write 6 Done Sent Done false
20452 5863 Acked Write 8 Done Sent Done false
20453 5864 Acked Write 1 Done Sent Done false
20454 5865 Acked Read 10 Done Sent Done false
20455 5866 Acked Read 10 Done Sent Done false
20456 5867 Acked Write 1 Done Sent Done false
20457 5868 Acked Flush 0 Done Sent Done false
20458 5869 Acked Read 4 Done Sent Done false
20459 5870 Acked Read 6 Done Sent Done false
20460 5871 Acked Write 8 Done Sent Done false
20461 5872 Acked Write 3 Done Sent Done false
20462 5873 Acked Read 2 Done Sent Done false
20463 5874 Acked Flush 0 Done Sent Done false
20464 5875 Acked Read 2 Done Sent Done false
20465 5876 Acked Read 5 Done Sent Done false
20466 5877 Acked Write 1 Done Sent Done false
20467 5878 Acked Read 3 Done Sent Done false
20468 5879 Acked Flush 0 Done Sent Done false
20469 5880 Acked Write 10 Done Sent Done false
20470 5881 Acked Read 10 Done Sent Done false
20471 5882 Acked Read 10 Done Sent Done false
20472 5883 Acked Flush 0 Done Sent Done false
20473 5884 Acked Read 2 Done Sent Done false
20474 5885 Acked Write 2 Done Sent Done false
20475 5886 Acked Write 8 Done Sent Done false
20476 5887 Acked Write 8 Done Sent Done false
20477 5888 Acked Read 5 Done Sent Done false
20478 5889 Acked Read 1 Done Sent Done false
20479 5890 Acked Write 7 Done Sent Done false
20480 5891 Acked Read 6 Done Sent Done false
20481 5892 Acked Read 6 Done Sent Done false
20482 5893 Acked Write 7 Done Sent Done false
20483 5894 Acked Write 7 Done Sent Done false
20484 5895 Acked Read 4 Done Sent Done false
20485 5896 Acked Write 10 Done Sent Done false
20486 5897 Acked Read 8 Done Sent Done false
20487 5898 Acked Write 9 Done Sent Done false
20488 5899 Acked Write 8 Done Sent Done false
20489 5900 Acked Read 5 Done Sent Done false
20490 5901 Acked Flush 0 Done Sent Done false
20491 5902 Acked Read 2 Done Sent Done false
20492 5903 Acked Write 1 Done Sent Done false
20493 5904 Acked Flush 0 Done Sent Done false
20494 5905 Acked Read 5 Done Sent Done false
20495 5906 Acked Read 1 Done Sent Done false
20496 5907 Acked Write 5 Done Sent Done false
20497 5908 Acked Read 2 Done Sent Done false
20498 5909 Acked Write 5 Done Sent Done false
20499 5910 Acked Flush 0 Done Sent Done false
20500 5911 Acked Read 10 Done Sent Done false
20501 5912 Acked Read 1 Done Sent Done false
20502 5913 Acked Read 4 Done Sent Done false
20503 5914 Acked Flush 0 Done Sent Done false
20504 5915 Acked Write 7 Done Sent Done false
20505 5916 Acked Read 9 Done Sent Done false
20506 5917 Acked Write 10 Done Sent Done false
20507 5918 Acked Read 8 Done Sent Done false
20508 5919 Acked Read 3 Done Sent Done false
20509 5920 Acked Write 9 Done Sent Done false
20510 5921 Acked Write 2 Done Sent Done false
20511 5922 Acked Read 1 Done Sent Done false
20512 5923 Acked Read 1 Done Sent Done false
20513 5924 Acked Read 5 Done Sent Done false
20514 5925 Acked Read 2 Done Sent Done false
20515 5926 Acked Write 4 Done Sent Done false
20516 5927 Acked Flush 0 Done Sent Done false
20517 5928 Acked Read 5 Done Sent Done false
20518 5929 Acked Write 4 Done Sent Done false
20519 5930 Acked Write 5 Done Sent Done false
20520 5931 Acked Read 6 Done Sent Done false
20521 5932 Acked Read 8 Done Sent Done false
20522 5933 Acked Write 7 Done Sent Done false
20523 5934 Acked Read 3 Done Sent Done false
20524 5935 Acked Read 5 Done Sent Done false
20525 5936 Acked Read 6 Done Sent Done false
20526 5937 Acked Write 2 Done Sent Done false
20527 5938 Acked Read 2 Done Sent Done false
20528 5939 Acked Flush 0 Done Sent Done false
20529 5940 Acked Flush 0 Done Sent Done false
20530 5941 Acked Read 3 Done Sent Done false
20531 5942 Acked Write 2 Done Sent Done false
20532 5943 Acked Write 3 Done Sent Done false
20533 5944 Acked Flush 0 Done Sent Done false
20534 5945 Acked Read 3 Done Sent Done false
20535 5946 Acked Write 4 Done Sent Done false
20536 5947 Acked Read 5 Done Sent Done false
20537 5948 Acked Read 4 Done Sent Done false
20538 5949 Acked Write 8 Done Sent Done false
20539 5950 Acked Read 7 Done Sent Done false
20540 5951 Acked Flush 0 Done Sent Done false
20541 5952 Acked Read 10 Done Sent Done false
20542 5953 Acked Write 6 Done Sent Done false
20543 5954 Acked Write 9 Done Sent Done false
20544 5955 Acked Write 3 Done Sent Done false
20545 5956 Acked Write 6 Done Sent Done false
20546 5957 Acked Write 8 Done Sent Done false
20547 5958 Acked Write 6 Done Sent Done false
20548 5959 Acked Read 2 Done Sent Done false
20549 5960 Acked Write 6 Done Sent Done false
20550 5961 Acked Flush 0 Done Sent Done false
20551 5962 Acked Read 8 Done Sent Done false
20552 5963 Acked Read 9 Done Sent Done false
20553 5964 Acked Read 6 Done Sent Done false
20554 5965 Acked Read 7 Done Sent Done false
20555 5966 Acked Flush 0 Done Sent Done false
20556 5967 Acked Read 1 Done Sent Done false
20557 5968 Acked Read 9 Done Sent Done false
20558 5969 Acked Write 10 Done Sent Done false
20559 5970 Acked Read 7 Done Sent Done false
20560 5971 Acked Write 1 Done Sent Done false
20561 5972 Acked Read 8 Done Sent Done false
20562 5973 Acked Write 5 Done Sent Done false
20563 5974 Acked Write 9 Done Sent Done false
20564 5975 Acked Write 7 Done Sent Done false
20565 5976 Acked Read 9 Done Sent Done false
20566 5977 Acked Flush 0 Done Sent Done false
20567 5978 Acked Write 9 Done Sent Done false
20568 5979 Acked Read 7 Done Sent Done false
20569 5980 Acked Write 5 Done Sent Done false
20570 5981 Acked Write 2 Done Sent Done false
20571 5982 Acked Read 8 Done Sent Done false
20572 5983 Acked Read 1 Done Sent Done false
20573 5984 Acked Read 4 Done Sent Done false
20574 5985 Acked Write 3 Done Sent Done false
20575 5986 Acked Read 10 Done Sent Done false
20576 5987 Acked Read 4 Done Sent Done false
20577 5988 Acked Read 2 Done Sent Done false
20578 5989 Acked Read 3 Done Sent Done false
20579 5990 Acked Read 3 Done Sent Done false
20580 5991 Acked Write 4 Done Sent Done false
20581 5992 Acked Write 8 Done Sent Done false
20582 5993 Acked Read 5 Done Sent Done false
20583 5994 Acked Flush 0 Done Sent Done false
20584 5995 Acked Write 1 Done Sent Done false
20585 5996 Acked Read 8 Done Sent Done false
20586 5997 Acked Write 1 Done Sent Done false
20587 5998 Acked Flush 0 Done Sent Done false
20588 5999 Acked Read 5 Done Sent Done false
20589 6000 Acked Read 2 Done Sent Done false
20590 6001 Acked Write 6 Done Sent Done false
20591 6002 Acked Read 5 Done Sent Done false
20592 6003 Acked Read 3 Done Sent Done false
20593 6004 Acked Write 4 Done Sent Done false
20594 6005 Acked Write 8 Done Sent Done false
20595 6006 Acked Write 8 Done Sent Done false
20596 6007 Acked Write 4 Done Sent Done false
20597 6008 Acked Write 3 Done Sent Done false
20598 6009 Acked Read 2 Done Sent Done false
20599 6010 Acked Write 4 Done Sent Done false
20600 6011 Acked Write 6 Done Sent Done false
20601 6012 Acked Read 5 Done Sent Done false
20602 6013 Acked Read 7 Done Sent Done false
20603 6014 Acked Write 6 Done Sent Done false
20604 6015 Acked Write 8 Done Sent Done false
20605 6016 Acked Read 7 Done Sent Done false
20606 6017 Acked Read 4 Done Sent Done false
20607 6018 Acked Read 5 Done Sent Done false
20608 6019 Acked Read 5 Done Sent Done false
20609 6020 Acked Read 4 Done Sent Done false
20610 6021 Acked Read 8 Done Sent Done false
20611 6022 Acked Read 10 Done Sent Done false
20612 6023 Acked Read 8 Done Sent Done false
20613 6024 Acked Flush 0 Done Sent Done false
20614 6025 Acked Read 2 Done Sent Done false
20615 6026 Acked Flush 0 Done Sent Done false
20616 6027 Acked Write 5 Done Sent Done false
20617 6028 Acked Read 5 Done Sent Done false
20618 6029 Acked Write 8 Done Sent Done false
20619 6030 Acked Write 10 Done Sent Done false
20620 6031 Acked Write 1 Done Sent Done false
20621 6032 Acked Read 2 Done Sent Done false
20622 6033 Acked Write 3 Done Sent Done false
20623 6034 Acked Write 4 Done Sent Done false
20624 6035 Acked Read 7 Done Sent Done false
20625 6036 Acked Read 2 Done Sent Done false
20626 6037 Acked Read 6 Done Sent Done false
20627 6038 Acked Read 8 Done Sent Done false
20628 6039 Acked Write 7 Done Sent Done false
20629 6040 Acked Write 9 Done Sent Done false
20630 6041 Acked Read 5 Done Sent Done false
20631 6042 Acked Write 3 Done Sent Done false
20632 6043 Acked Write 4 Done Sent Done false
20633 6044 Acked Flush 0 Done Sent Done false
20634 6045 Acked Read 1 Done Sent Done false
20635 6046 Acked Read 5 Done Sent Done false
20636 6047 Acked Write 2 Done Sent Done false
20637 6048 Acked Read 5 Done Sent Done false
20638 6049 Acked Read 8 Done Sent Done false
20639 6050 Acked Write 2 Done Sent Done false
20640 6051 Acked Read 3 Done Sent Done false
20641 6052 Acked Read 8 Done Sent Done false
20642 6053 Acked Read 3 Done Sent Done false
20643 6054 Acked Write 8 Done Sent Done false
20644 6055 Acked Read 6 Done Sent Done false
20645 6056 Acked Flush 0 Done Sent Done false
20646 6057 Acked Flush 0 Done Sent Done false
20647 6058 Acked Write 2 Done Sent Done false
20648 6059 Acked Read 9 Done Sent Done false
20649 6060 Acked Flush 0 Done Sent Done false
20650 6061 Acked Write 10 Done Sent Done false
20651 6062 Acked Read 7 Done Sent Done false
20652 6063 Acked Read 2 Done Sent Done false
20653 6064 Acked Read 4 Done Sent Done false
20654 6065 Acked Write 3 Done Sent Done false
20655 6066 Acked Write 2 Done Sent Done false
20656 6067 Acked Write 1 Done Sent Done false
20657 6068 Acked Read 2 Done Sent Done false
20658 6069 Acked Read 3 Done Sent Done false
20659 6070 Acked Write 7 Done Sent Done false
20660 6071 Acked Read 3 Done Sent Done false
20661 6072 Acked Read 8 Done Sent Done false
20662 6073 Acked Write 10 Done Sent Done false
20663 6074 Acked Flush 0 Done Sent Done false
20664 6075 Acked Write 8 Done Sent Done false
20665 6076 Acked Write 5 Done Sent Done false
20666 6077 Acked Read 5 Done Sent Done false
20667 6078 Acked Read 2 Done Sent Done false
20668 6079 Acked Write 7 Done Sent Done false
20669 6080 Acked Read 6 Done Sent Done false
20670 6081 Acked Write 4 Done Sent Done false
20671 6082 Acked Read 7 Done Sent Done false
20672 6083 Acked Read 4 Done Sent Done false
20673 6084 Acked Read 5 Done Sent Done false
20674 6085 Acked Write 5 Done Sent Done false
20675 6086 Acked Read 2 Done Sent Done false
20676 6087 Acked Read 1 Done Sent Done false
20677 6088 Acked Read 9 Done Sent Done false
20678 6089 Acked Write 6 Done Sent Done false
20679 6090 Acked Flush 0 Done Sent Done false
20680 6091 Acked Read 8 Done Sent Done false
20681 6092 Acked Read 8 Done Sent Done false
20682 6093 Acked Read 4 Done Sent Done false
20683 6094 Acked Read 9 Done Sent Done false
20684 6095 Acked Write 1 Done Sent Done false
20685 6096 Acked Write 1 Done Sent Done false
20686 6097 Acked Read 5 Done Sent Done false
20687 6098 Acked Read 7 Done Sent Done false
20688 6099 Acked Write 7 Done Sent Done false
20689 6100 Acked Write 3 Done Sent Done false
20690 6101 Acked Read 5 Done Sent Done false
20691 6102 Acked Read 3 Done Sent Done false
20692 6103 Acked Read 10 Done Sent Done false
20693 6104 Acked Read 6 Done Sent Done false
20694 6105 Acked Write 8 Done Sent Done false
20695 6106 Acked Read 5 Done Sent Done false
20696 6107 Acked Read 3 Done Sent Done false
20697 6108 Acked Write 8 Done Sent Done false
20698 6109 Acked Read 8 Done Sent Done false
20699 6110 Acked Read 8 Done Sent Done false
20700 6111 Acked Read 2 Done Sent Done false
20701 6112 Acked Flush 0 Done Sent Done false
20702 6113 Acked Write 5 Done Sent Done false
20703 6114 Acked Read 1 Done Sent Done false
20704 6115 Acked Read 5 Done Sent Done false
20705 6116 Acked Write 6 Done Sent Done false
20706 6117 Acked Flush 0 Done Sent Done false
20707 6118 Acked Write 5 Done Sent Done false
20708 6119 Acked Read 3 Done Sent Done false
20709 6120 Acked Write 5 Done Sent Done false
20710 6121 Acked Write 5 Done Sent Done false
20711 6122 Acked Read 6 Done Sent Done false
20712 6123 Acked Read 2 Done Sent Done false
20713 6124 Acked Write 3 Done Sent Done false
20714 6125 Acked Write 5 Done Sent Done false
20715 6126 Acked Write 2 Done Sent Done false
20716 6127 Acked Flush 0 Done Sent Done false
20717 6128 Acked Read 1 Done Sent Done false
20718 6129 Acked Read 5 Done Sent Done false
20719 6130 Acked Read 4 Done Sent Done false
20720 6131 Acked Write 2 Done Sent Done false
20721 STATES DS:0 DS:1 DS:2 TOTAL
20722 Sent 0 301 0 301
20723 Done 301 0 301 602
20724 Skipped 0 0 0 0
20725 Error 0 0 0 0
20726 Last Flush: Some(JobId(6127)) Some(JobId(5830)) Some(JobId(6127))
20727 Downstairs last five completed: 5830 5829 5828 5827 5826
20728 Upstairs last five completed: 6131 6130 6129 6128 6127
20729 CLIENT: Up:0 ds:301 act:2
207302025-03-17T22:39:05.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207312025-03-17T22:39:07.175ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207322025-03-17T22:39:07.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207332025-03-17T22:39:08.906ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207342025-03-17T22:39:08.924ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207352025-03-17T22:39:08.924ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50394) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207362025-03-17T22:39:08.925ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207372025-03-17T22:39:08.925ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207382025-03-17T22:39:08.925ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207392025-03-17T22:39:08.925ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207402025-03-17T22:39:08.925ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207412025-03-17T22:39:08.925ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207422025-03-17T22:39:08.926ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
20743 ----------------------------------------------------------------
20744 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
20745 States: Active Active Active
20746 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20747 6128 Acked Read 1 Done Done Done true
20748 6129 Acked Read 5 Done Done Done true
20749 6130 Acked Read 4 Done Done Done true
20750 6131 Acked Write 2 Done Done Done true
20751 STATES DS:0 DS:1 DS:2 TOTAL
20752 Sent 0 0 0 0
20753 Done 4 4 4 12
20754 Skipped 0 0 0 0
20755 Error 0 0 0 0
20756 Last Flush: Some(JobId(6127)) Some(JobId(6127)) Some(JobId(6127))
20757 Downstairs last five completed: 6127 6126 6125 6124 6123
20758 Upstairs last five completed: 6131 6130 6129 6128 6127
20759 CLIENT: Up:0 ds:4 act:3
20760 ----------------------------------------------------------------
20761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20762 Downstairs last five completed: 6132 6131 6130 6129 6128
20763 Upstairs last five completed: 6132 6131 6130 6129 6128
20764 CLIENT: Up:0 ds:0 act:3
20765 Replay: All jobs finished
20766 Main task has work to do, go find it
20767 got dsc Stop(1)
20768 stop 1
20769 Mar 17 22:39:13.174[8820] Got stop action so:false kr:false
20770 INFO request completed, latency_us: 347, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c0c4735a-5ad8-444b-ba7c-6611ee12eadf, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
20771 Mar 17 22:39:13.175 INFO request completed, latency_us: 153, response_code: 200, uri: /state/cid/1, method: GET, req_id{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:39:13.175231008Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
20772 : 0a422703-002b-4831-a190-2fdc1c6cbfee, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:39:13.175303474Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
20773 .1:9998
207742025-03-17T22:39:13.175ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
207752025-03-17T22:39:13.175ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
20776 [8820] Exited with: ExitStatus(unix_wait_status(9))
20777 [8820][1] reports Exit
20778 Mar 17 22:39:17.177 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/1, method: GET, req_id: 69df6155-7e08-4109-a3cc-f4a62135a882, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
20779 001/300 Read block 2046 len 20480
20780 002/300 Write block 3511 len 8192 data: 3 3
20781 003/300 Read block 4522 len 8192
20782 004/300 Read block 4690 len 24576
20783 005/300 Read block 3244 len 24576
20784 006/300 Write block 3574 len 28672 data: 5 4 3 3 3 3 3
20785 007/300 Read block 192 len 40960
20786 008/300 Write block 4138 len 32768 data: 5 5 4 4 6 6 6 6
20787 009/300 Write block 956 len 8192 data: 2 2
20788 010/300 Write block 142 len 8192 data: 5 5
20789 011/300 Flush
20790 012/300 Read block 4457 len 40960
20791 013/300 Read block 3905 len 40960
20792 014/300 Write block 2054 len 4096 data: 5
20793 015/300 Write block 3221 len 8192 data: 3 3
20794 016/300 Flush
20795 017/300 Read block 1478 len 4096
20796 018/300 Read block 4948 len 40960
20797 019/300 Write block 307 len 12288 data: 5 6 5
20798 020/300 Write block 999 len 24576 data: 3 2 2 3 4 4
20799 021/300 Read block 2535 len 4096
20800 022/300 Read block 637 len 36864
20801 023/300 Read block 3930 len 32768
20802 024/300 Read block 2790 len 12288
20803 025/300 Write block 698 len 28672 data: 4 5 5 5 6 6 7
20804 026/300 Read block 2001 len 32768
20805 027/300 Write block 2452 len 20480 data: 3 3 3 3 3
20806 028/300 Flush
20807 029/300 Read block 2453 len 8192
20808 030/300 Write block 4208 len 8192 data: 5 4
20809 031/300 Read block 4952 len 20480
20810 032/300 Write block 2034 len 36864 data: 8 7 6 6 6 6 7 6 7
20811 033/300 Read block 4248 len 32768
20812 034/300 Write block 2884 len 32768 data: 3 3 2 3 3 3 3 4
20813 035/300 Flush
20814 036/300 Read block 1051 len 36864
20815 037/300 Read block 4850 len 20480
20816 038/300 Read block 639 len 8192
20817 039/300 Write block 4043 len 4096 data: 4
20818 040/300 Write block 4877 len 8192 data: 4 4
20819 041/300 Read block 4324 len 24576
20820 042/300 Write block 1502 len 16384 data: 4 4 5 3
20821 043/300 Flush
20822 044/300 Read block 1954 len 4096
20823 045/300 Flush
20824 046/300 Write block 4683 len 16384 data: 3 2 2 2
20825 047/300 Write block 1203 len 8192 data: 2 3
20826 048/300 Write block 3257 len 4096 data: 3
20827 049/300 Write block 2327 len 20480 data: 2 3 5 5 5
20828 050/300 Write block 891 len 4096 data: 2
20829 051/300 Write block 109 len 20480 data: 5 5 6 5 4
20830 052/300 Read block 3044 len 16384
20831 053/300 Write block 493 len 36864 data: 3 4 3 2 2 2 2 2 3
20832 054/300 Write block 202 len 12288 data: 3 3 3
20833 055/300 Write block 3616 len 4096 data: 3
20834 056/300 Write block 3068 len 28672 data: 4 4 4 4 4 4 3
20835 057/300 Write block 1491 len 36864 data: 2 2 2 2 3 4 5 6 6
20836 058/300 Write block 731 len 12288 data: 4 4 5
20837 059/300 Write block 2613 len 36864 data: 4 5 5 4 5 5 4 5 5
20838 060/300 Read block 3517 len 16384
20839 061/300 Read block 1852 len 40960
20840 062/300 Read block 638 len 28672
20841 063/300 Read block 695 len 40960
20842 064/300 Flush
20843 065/300 Read block 747 len 32768
20844 066/300 Read block 4942 len 28672
20845 067/300 Write block 3094 len 20480 data: 4 4 4 4 4
20846 068/300 Write block 1258 len 32768 data: 5 6 5 6 7 5 4 5
20847 069/300 Read block 3399 len 24576
20848 070/300 Read block 3821 len 16384
20849 071/300 Read block 373 len 20480
20850 072/300 Write block 1453 len 20480 data: 3 6 6 6 4
20851 073/300 Read block 242 len 32768
20852 074/300 Write block 1757 len 16384 data: 4 4 5 3
20853 075/300 Write block 2943 len 28672 data: 5 6 5 5 5 4 4
20854 076/300 Read block 1261 len 8192
20855 077/300 Write block 4512 len 8192 data: 2 4
20856 078/300 Read block 2488 len 16384
20857 079/300 Read block 4832 len 12288
20858 080/300 Read block 1151 len 4096
20859 081/300 Read block 1372 len 36864
20860 082/300 Write block 4084 len 28672 data: 2 2 3 3 5 4 4
20861 083/300 Write block 2570 len 12288 data: 5 6 6
20862 084/300 Read block 4539 len 28672
20863 085/300 Read block 652 len 20480
20864 086/300 Write block 191 len 20480 data: 4 5 6 6 6
20865 087/300 Write block 1982 len 12288 data: 7 9 8
20866 088/300 Write block 1023 len 32768 data: 2 2 2 2 3 3 3 3
20867 089/300 Write block 3433 len 32768 data: 7 8 6 5 4 3 3 4
20868 090/300 Write block 4354 len 16384 data: 9 7 7 7
20869 091/300 Read block 813 len 36864
20870 092/300 Read block 3098 len 36864
20871 093/300 Write block 1831 len 36864 data: 6 6 5 4 4 4 4 3 3
20872 094/300 Read block 472 len 40960
20873 095/300 Write block 53 len 12288 data: 4 5 6
20874 096/300 Write block 4967 len 32768 data: 6 6 6 6 6 6 5 4
20875 097/300 Read block 1248 len 16384
20876 098/300 Write block 4632 len 24576 data: 6 6 6 7 7 6
20877 099/300 Read block 62 len 8192
20878 100/300 Flush
20879 101/300 Read block 935 len 20480
20880 102/300 Write block 4584 len 20480 data: 4 4 4 4 3
20881 103/300 Read block 2807 len 36864
20882 104/300 Read block 2331 len 32768
20883 105/300 Write block 852 len 28672 data: 5 5 4 5 5 4 4
20884 106/300 Read block 1927 len 36864
20885 107/300 Read block 3631 len 12288
20886 108/300 Read block 1477 len 12288
20887 109/300 Read block 1482 len 16384
20888 110/300 Read block 1641 len 20480
20889 111/300 Flush
20890 112/300 Read block 3732 len 4096
20891 113/300 Read block 1903 len 40960
20892 114/300 Read block 1113 len 24576
20893 115/300 Write block 3722 len 12288 data: 7 6 5
20894 116/300 Write block 1947 len 16384 data: 3 4 4 4
20895 117/300 Read block 4952 len 40960
20896 118/300 Write block 1193 len 16384 data: 5 4 4 3
20897 119/300 Read block 1028 len 40960
20898 120/300 Read block 1361 len 4096
20899 121/300 Write block 2463 len 16384 data: 5 5 5 6
20900 122/300 Read block 1712 len 16384
20901 123/300 Read block 855 len 12288
20902 124/300 Flush
20903 125/300 Read block 1912 len 40960
20904 126/300 Write block 2961 len 24576 data: 3 4 5 4 4 3
20905 127/300 Flush
20906 128/300 Write block 468 len 4096 data: 5
20907 129/300 Write block 4412 len 8192 data: 4 4
20908 130/300 Write block 4180 len 28672 data: 5 5 5 5 4 6 5
20909 131/300 Read block 4721 len 24576
20910 132/300 Write block 2665 len 12288 data: 3 4 4
20911 133/300 Read block 2105 len 36864
20912 134/300 Read block 487 len 16384
20913 135/300 Read block 806 len 32768
20914 136/300 Read block 1259 len 36864
20915 137/300 Write block 89 len 12288 data: 2 2 2
20916 138/300 Flush
20917 139/300 Read block 383 len 12288
20918 140/300 Write block 1907 len 24576 data: 4 3 4 4 4 5
20919 141/300 Read block 3121 len 28672
20920 142/300 Read block 3486 len 32768
20921 143/300 Flush
20922 144/300 Write block 4679 len 28672 data: 3 4 4 4 4 3 3
20923 145/300 Flush
20924 146/300 Flush
20925 147/300 Write block 2482 len 28672 data: 6 6 6 7 7 6 6
20926 148/300 Write block 4560 len 16384 data: 3 4 5 4
20927 149/300 Write block 775 len 28672 data: 4 4 3 4 4 4 5
20928 150/300 Write block 4106 len 40960 data: 4 4 4 4 4 4 4 6 8 7
20929 151/300 Read block 407 len 36864
20930 152/300 Write block 2230 len 8192 data: 3 3
20931 153/300 Read block 3456 len 12288
20932 154/300 Write block 1098 len 36864 data: 3 4 5 5 5 4 4 3 3
20933 155/300 Read block 2184 len 40960
20934 156/300 Write block 2633 len 12288 data: 4 3 4
20935 157/300 Write block 1834 len 16384 data: 5 5 5 5
20936 158/300 Write block 4737 len 4096 data: 4
20937 159/300 Write block 4207 len 20480 data: 5 6 5 3 2
20938 160/300 Write block 4882 len 36864 data: 5 5 5 5 4 4 3 4 4
20939 161/300 Write block 3770 len 24576 data: 3 4 3 4 4 4
20940 162/300 Read block 3584 len 28672
20941 163/300 Read block 3949 len 36864
20942 164/300 Read block 2161 len 12288
20943 165/300 Write block 4759 len 40960 data: 3 3 3 3 3 3 3 3 3 3
20944 166/300 Write block 2256 len 36864 data: 3 3 4 4 3 3 4 3 3
20945 167/300 Write block 4109 len 20480 data: 5 5 5 5 7
20946 168/300 Write block 1432 len 32768 data: 3 3 3 5 7 6 4 4
20947 169/300 Read block 1341 len 40960
20948 170/300 Write block 2656 len 12288 data: 4 5 4
20949 171/300 Read block 4659 len 4096
20950 172/300 Write block 23 len 36864 data: 6 6 6 5 5 5 4 3 3
20951 173/300 Read block 226 len 20480
20952 174/300 Flush
20953 175/300 Write block 2930 len 40960 data: 3 2 2 2 2 2 2 2 3 4
20954 176/300 Write block 451 len 32768 data: 7 5 4 4 3 3 3 3
20955 177/300 Read block 3805 len 8192
20956 178/300 Write block 725 len 28672 data: 6 7 6 7 6 5 5
20957 179/300 Write block 63 len 24576 data: 2 3 4 5 6 5
20958 180/300 Read block 191 len 8192
20959 181/300 Read block 4070 len 36864
20960 182/300 Read block 2848 len 28672
20961 183/300 Read block 920 len 12288
20962 184/300 Write block 3130 len 20480 data: 5 5 4 5 5
20963 185/300 Read block 3548 len 24576
20964 186/300 Read block 2408 len 20480
20965 187/300 Write block 883 len 8192 data: 4 5
20966 188/300 Read block 4701 len 16384
20967 189/300 Read block 3573 len 40960
20968 190/300 Flush
20969 191/300 Write block 3436 len 4096 data: 6
20970 192/300 Read block 369 len 8192
20971 193/300 Write block 2551 len 4096 data: 6
20972 194/300 Read block 3119 len 28672
20973 195/300 Read block 4175 len 40960
20974 196/300 Read block 2918 len 32768
20975 197/300 Write block 4086 len 40960 data: 4 4 6 5 5 4 4 3 3 5
20976 198/300 Write block 2155 len 24576 data: 3 4 5 6 5 5
20977 199/300 Write block 832 len 36864 data: 3 3 3 3 3 3 3 3 3
20978 200/300 Read block 746 len 24576
20979 201/300 Flush
20980 202/300 Read block 310 len 8192
20981 203/300 Write block 4934 len 20480 data: 7 7 6 6 3
20982 204/300 Read block 4342 len 16384
20983 205/300 Read block 207 len 4096
20984 206/300 Write block 2489 len 40960 data: 4 2 2 3 4 3 3 4 2 2
20985 207/300 Write block 159 len 24576 data: 4 3 4 4 4 4
20986 208/300 Write block 2865 len 24576 data: 3 3 2 2 3 3
20987 209/300 Write block 2984 len 28672 data: 5 5 6 4 4 3 6
20988 210/300 Write block 97 len 32768 data: 4 4 5 4 4 4 5 5
20989 211/300 Write block 2356 len 28672 data: 4 3 4 4 3 3 3
20990 212/300 Read block 3374 len 4096
20991 213/300 Read block 80 len 36864
20992 214/300 Write block 2351 len 4096 data: 4
20993 215/300 Read block 4611 len 24576
20994 216/300 Write block 2908 len 16384 data: 4 6 4 5
20995 217/300 Read block 1696 len 8192
20996 218/300 Read block 4622 len 32768
20997 219/300 Read block 1681 len 32768
20998 220/300 Read block 4115 len 28672
20999 221/300 Write block 4605 len 28672 data: 7 6 5 3 3 3 4
21000 222/300 Write block 2202 len 40960 data: 7 7 7 9 8 7 5 5 5 4
21001 223/300 Flush
21002 224/300 Read block 2379 len 4096
21003 225/300 Write block 1783 len 24576 data: 3 5 6 6 6 5
21004 226/300 Read block 4070 len 24576
21005 227/300 Read block 850 len 16384
21006 228/300 Write block 2531 len 36864 data: 7 8 7 8 8 6 6 6 5
21007 229/300 Read block 4882 len 24576
21008 230/300 Read block 3929 len 20480
21009 231/300 Write block 1430 len 20480 data: 2 2 4 4 4
21010 232/300 Flush
21011 233/300 Write block 1604 len 32768 data: 3 3 3 3 3 4 4 3
21012 234/300 Read block 3134 len 16384
21013 235/300 Read block 1615 len 32768
21014 236/300 Read block 1583 len 32768
21015 237/300 Write block 3320 len 12288 data: 7 6 6
21016 238/300 Read block 4948 len 28672
21017 239/300 Read block 4280 len 40960
21018 240/300 Read block 1851 len 28672
21019 241/300 Read block 3349 len 8192
21020 242/300 Read block 3602 len 40960
21021 243/300 Write block 4809 len 4096 data: 4
21022 244/300 Read block 1433 len 32768
21023 245/300 Read block 4548 len 16384
21024 246/300 Read block 3077 len 40960
21025 247/300 Read block 3649 len 4096
21026 248/300 Read block 2695 len 24576
21027 249/300 Read block 3481 len 24576
21028 250/300 Flush
21029 251/300 Write block 1443 len 4096 data: 6
21030 252/300 Write block 2192 len 32768 data: 2 4 3 4 5 3 4 4
21031 253/300 Read block 4792 len 8192
21032 254/300 Flush
21033 255/300 Write block 2430 len 20480 data: 3 3 3 3 2
21034 256/300 Write block 4598 len 12288 data: 4 5 5
21035 257/300 Read block 2520 len 40960
21036 258/300 Read block 2656 len 4096
21037 259/300 Flush
21038 260/300 Read block 343 len 24576
21039 261/300 Write block 3824 len 20480 data: 4 4 3 2 2
21040 262/300 Read block 3401 len 16384
21041 263/300 Read block 4336 len 40960
21042 264/300 Write block 2818 len 12288 data: 3 3 3
21043 265/300 Write block 1675 len 8192 data: 2 2
21044 266/300 Read block 4709 len 36864
21045 267/300 Write block 2722 len 28672 data: 4 2 2 2 2 2 5
21046 268/300 Read block 18 len 40960
21047 269/300 Read block 1088 len 28672
21048 270/300 Write block 4254 len 20480 data: 4 4 4 3 4
21049 271/300 Write block 2640 len 16384 data: 3 3 3 2
21050 272/300 Flush
21051 273/300 Write block 2493 len 28672 data: 5 4 4 5 3 3 2
21052 274/300 Read block 1946 len 20480
21053 275/300 Write block 1997 len 24576 data: 4 6 5 5 5 5
21054 276/300 Read block 2999 len 28672
21055 277/300 Write block 1219 len 8192 data: 5 4
21056 278/300 Write block 779 len 40960 data: 5 5 6 7 7 7 6 4 2 3
21057 279/300 Read block 2154 len 8192
21058 280/300 Read block 3090 len 8192
21059 281/300 Write block 872 len 40960 data: 3 2 2 3 3 4 3 3 3 4
21060 282/300 Write block 4204 len 32768 data: 4 4 5 6 7 6 4 3
21061 283/300 Read block 2877 len 8192
21062 284/300 Write block 1838 len 12288 data: 4 4 3
21063 285/300 Write block 2468 len 36864 data: 6 5 6 6 6 6 7 7 5
21064 286/300 Read block 1456 len 8192
21065 287/300 Read block 1744 len 20480
21066 288/300 Read block 877 len 28672
21067 289/300 Write block 1148 len 24576 data: 2 2 4 4 5 4
21068 290/300 Read block 2567 len 12288
21069 291/300 Read block 913 len 28672
21070 292/300 Write block 4538 len 16384 data: 7 7 6 7
21071 293/300 Write block 4367 len 40960 data: 6 6 4 5 5 5 6 7 7 6
21072 294/300 Read block 3053 len 36864
21073 295/300 Write block 2924 len 12288 data: 5 3 2
21074 296/300 Flush
21075 297/300 Write block 1277 len 20480 data: 4 4 5 6 4
21076 298/300 Read block 3575 len 12288
21077 299/300 Write block 4659 len 8192 data: 5 5
21078 300/300 Read block 409 len 24576
21079 301/300 Read block 2830 len 4096
21080 Main task has work to do, go find it
21081 got dsc Start(1)
21082 start 1
21083 Mar 17 22:39:17.420 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/1, method: POST, req_id: 61869bc4-f87b-44ad-9b4b-c92ad9e35671, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
21084 Replay: started 1, returned:Ok(())
21085 ----------------------------------------------------------------
21086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21087 States: Active Offline Active
21088 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21089 6133 Acked Read 5 Done Sent Done false
21090 6134 Acked Write 2 Done Sent Done false
21091 6135 Acked Read 2 Done Sent Done false
21092 6136 Acked Read 6 Done Sent Done false
21093 6137 Acked Read 6 Done Sent Done false
21094 6138 Acked Write 7 Done Sent Done false
21095 6139 Acked Read 10 Done Sent Done false
21096 6140 Acked Write 8 Done Sent Done false
21097 6141 Acked Write 2 Done Sent Done false
21098 6142 Acked Write 2 Done Sent Done false
21099 6143 Acked Flush 0 Done Sent Done false
21100 6144 Acked Read 10 Done Sent Done false
21101 6145 Acked Read 10 Done Sent Done false
21102 6146 Acked Write 1 Done Sent Done false
21103 6147 Acked Write 2 Done Sent Done false
21104 6148 Acked Flush 0 Done Sent Done false
21105 6149 Acked Read 1 Done Sent Done false
21106 6150 Acked Read 10 Done Sent Done false
21107 6151 Acked Write 3 Done Sent Done false
21108 6152 Acked Write 6 Done Sent Done false
21109 6153 Acked Read 1 Done Sent Done false
21110 6154 Acked Read 9 Done Sent Done false
21111 6155 Acked Read 8 Done Sent Done false
21112 6156 Acked Read 3 Done Sent Done false
21113 6157 Acked Write 7 Done Sent Done false
21114 6158 Acked Read 8 Done Sent Done false
21115 6159 Acked Write 5 Done Sent Done false
21116 6160 Acked Flush 0 Done Sent Done false
21117 6161 Acked Read 2 Done Sent Done false
21118 6162 Acked Write 2 Done Sent Done false
21119 6163 Acked Read 5 Done Sent Done false
21120 6164 Acked Write 9 Done Sent Done false
21121 6165 Acked Read 8 Done Sent Done false
21122 6166 Acked Write 8 Done Sent Done false
21123 6167 Acked Flush 0 Done Sent Done false
21124 6168 Acked Read 9 Done Sent Done false
21125 6169 Acked Read 5 Done Sent Done false
21126 6170 Acked Read 2 Done Sent Done false
21127 6171 Acked Write 1 Done Sent Done false
21128 6172 Acked Write 2 Done Sent Done false
21129 6173 Acked Read 6 Done Sent Done false
21130 6174 Acked Write 4 Done Sent Done false
21131 6175 Acked Flush 0 Done Sent Done false
21132 6176 Acked Read 1 Done Sent Done false
21133 6177 Acked Flush 0 Done Sent Done false
21134 6178 Acked Write 4 Done Sent Done false
21135 6179 Acked Write 2 Done Sent Done false
21136 6180 Acked Write 1 Done Sent Done false
21137 6181 Acked Write 5 Done Sent Done false
21138 6182 Acked Write 1 Done Sent Done false
21139 6183 Acked Write 5 Done Sent Done false
21140 6184 Acked Read 4 Done Sent Done false
21141 6185 Acked Write 9 Done Sent Done false
21142 6186 Acked Write 3 Done Sent Done false
21143 6187 Acked Write 1 Done Sent Done false
21144 6188 Acked Write 7 Done Sent Done false
21145 6189 Acked Write 9 Done Sent Done false
21146 6190 Acked Write 3 Done Sent Done false
21147 6191 Acked Write 9 Done Sent Done false
21148 6192 Acked Read 4 Done Sent Done false
21149 6193 Acked Read 10 Done Sent Done false
21150 6194 Acked Read 7 Done Sent Done false
21151 6195 Acked Read 10 Done Sent Done false
21152 6196 Acked Flush 0 Done Sent Done false
21153 6197 Acked Read 8 Done Sent Done false
21154 6198 Acked Read 7 Done Sent Done false
21155 6199 Acked Write 5 Done Sent Done false
21156 6200 Acked Write 8 Done Sent Done false
21157 6201 Acked Read 6 Done Sent Done false
21158 6202 Acked Read 4 Done Sent Done false
21159 6203 Acked Read 5 Done Sent Done false
21160 6204 Acked Write 5 Done Sent Done false
21161 6205 Acked Read 8 Done Sent Done false
21162 6206 Acked Write 4 Done Sent Done false
21163 6207 Acked Write 7 Done Sent Done false
21164 6208 Acked Read 2 Done Sent Done false
21165 6209 Acked Write 2 Done Sent Done false
21166 6210 Acked Read 4 Done Sent Done false
21167 6211 Acked Read 3 Done Sent Done false
21168 6212 Acked Read 1 Done Sent Done false
21169 6213 Acked Read 9 Done Sent Done false
21170 6214 Acked Write 7 Done Sent Done false
21171 6215 Acked Write 3 Done Sent Done false
21172 6216 Acked Read 7 Done Sent Done false
21173 6217 Acked Read 5 Done Sent Done false
21174 6218 Acked Write 5 Done Sent Done false
21175 6219 Acked Write 3 Done Sent Done false
21176 6220 Acked Write 8 Done Sent Done false
21177 6221 Acked Write 8 Done Sent Done false
21178 6222 Acked Write 4 Done Sent Done false
21179 6223 Acked Read 9 Done Sent Done false
21180 6224 Acked Read 9 Done Sent Done false
21181 6225 Acked Write 9 Done Sent Done false
21182 6226 Acked Read 10 Done Sent Done false
21183 6227 Acked Write 3 Done Sent Done false
21184 6228 Acked Write 8 Done Sent Done false
21185 6229 Acked Read 4 Done Sent Done false
21186 6230 Acked Write 6 Done Sent Done false
21187 6231 Acked Read 2 Done Sent Done false
21188 6232 Acked Flush 0 Done Sent Done false
21189 6233 Acked Read 5 Done Sent Done false
21190 6234 Acked Write 5 Done Sent Done false
21191 6235 Acked Read 9 Done Sent Done false
21192 6236 Acked Read 8 Done Sent Done false
21193 6237 Acked Write 7 Done Sent Done false
21194 6238 Acked Read 9 Done Sent Done false
21195 6239 Acked Read 3 Done Sent Done false
21196 6240 Acked Read 3 Done Sent Done false
21197 6241 Acked Read 4 Done Sent Done false
21198 6242 Acked Read 5 Done Sent Done false
21199 6243 Acked Flush 0 Done Sent Done false
21200 6244 Acked Read 1 Done Sent Done false
21201 6245 Acked Read 10 Done Sent Done false
21202 6246 Acked Read 6 Done Sent Done false
21203 6247 Acked Write 3 Done Sent Done false
21204 6248 Acked Write 4 Done Sent Done false
21205 6249 Acked Read 10 Done Sent Done false
21206 6250 Acked Write 4 Done Sent Done false
21207 6251 Acked Read 10 Done Sent Done false
21208 6252 Acked Read 1 Done Sent Done false
21209 6253 Acked Write 4 Done Sent Done false
21210 6254 Acked Read 4 Done Sent Done false
21211 6255 Acked Read 3 Done Sent Done false
21212 6256 Acked Flush 0 Done Sent Done false
21213 6257 Acked Read 10 Done Sent Done false
21214 6258 Acked Write 6 Done Sent Done false
21215 6259 Acked Flush 0 Done Sent Done false
21216 6260 Acked Write 1 Done Sent Done false
21217 6261 Acked Write 2 Done Sent Done false
21218 6262 Acked Write 7 Done Sent Done false
21219 6263 Acked Read 6 Done Sent Done false
21220 6264 Acked Write 3 Done Sent Done false
21221 6265 Acked Read 9 Done Sent Done false
21222 6266 Acked Read 4 Done Sent Done false
21223 6267 Acked Read 8 Done Sent Done false
21224 6268 Acked Read 9 Done Sent Done false
21225 6269 Acked Write 3 Done Sent Done false
21226 6270 Acked Flush 0 Done Sent Done false
21227 6271 Acked Read 3 Done Sent Done false
21228 6272 Acked Write 6 Done Sent Done false
21229 6273 Acked Read 7 Done Sent Done false
21230 6274 Acked Read 8 Done Sent Done false
21231 6275 Acked Flush 0 Done Sent Done false
21232 6276 Acked Write 7 Done Sent Done false
21233 6277 Acked Flush 0 Done Sent Done false
21234 6278 Acked Flush 0 Done Sent Done false
21235 6279 Acked Write 7 Done Sent Done false
21236 6280 Acked Write 4 Done Sent Done false
21237 6281 Acked Write 7 Done Sent Done false
21238 6282 Acked Write 10 Done Sent Done false
21239 6283 Acked Read 9 Done Sent Done false
21240 6284 Acked Write 2 Done Sent Done false
21241 6285 Acked Read 3 Done Sent Done false
21242 6286 Acked Write 9 Done Sent Done false
21243 6287 Acked Read 10 Done Sent Done false
21244 6288 Acked Write 3 Done Sent Done false
21245 6289 Acked Write 4 Done Sent Done false
21246 6290 Acked Write 1 Done Sent Done false
21247 6291 Acked Write 5 Done Sent Done false
21248 6292 Acked Write 9 Done Sent Done false
21249 6293 Acked Write 6 Done Sent Done false
21250 6294 Acked Read 7 Done Sent Done false
21251 6295 Acked Read 9 Done Sent Done false
21252 6296 Acked Read 3 Done Sent Done false
21253 6297 Acked Write 10 Done Sent Done false
21254 6298 Acked Write 9 Done Sent Done false
21255 6299 Acked Write 5 Done Sent Done false
21256 6300 Acked Write 8 Done Sent Done false
21257 6301 Acked Read 10 Done Sent Done false
21258 6302 Acked Write 3 Done Sent Done false
21259 6303 Acked Read 1 Done Sent Done false
21260 6304 Acked Write 9 Done Sent Done false
21261 6305 Acked Read 5 Done Sent Done false
21262 6306 Acked Flush 0 Done Sent Done false
21263 6307 Acked Write 10 Done Sent Done false
21264 6308 Acked Write 8 Done Sent Done false
21265 6309 Acked Read 2 Done Sent Done false
21266 6310 Acked Write 7 Done Sent Done false
21267 6311 Acked Write 6 Done Sent Done false
21268 6312 Acked Read 2 Done Sent Done false
21269 6313 Acked Read 9 Done Sent Done false
21270 6314 Acked Read 7 Done Sent Done false
21271 6315 Acked Read 3 Done Sent Done false
21272 6316 Acked Write 5 Done Sent Done false
21273 6317 Acked Read 6 Done Sent Done false
21274 6318 Acked Read 5 Done Sent Done false
21275 6319 Acked Write 2 Done Sent Done false
21276 6320 Acked Read 4 Done Sent Done false
21277 6321 Acked Read 10 Done Sent Done false
21278 6322 Acked Flush 0 Done Sent Done false
21279 6323 Acked Write 1 Done Sent Done false
21280 6324 Acked Read 2 Done Sent Done false
21281 6325 Acked Write 1 Done Sent Done false
21282 6326 Acked Read 7 Done Sent Done false
21283 6327 Acked Read 10 Done Sent Done false
21284 6328 Acked Read 8 Done Sent Done false
21285 6329 Acked Write 10 Done Sent Done false
21286 6330 Acked Write 6 Done Sent Done false
21287 6331 Acked Write 9 Done Sent Done false
21288 6332 Acked Read 6 Done Sent Done false
21289 6333 Acked Flush 0 Done Sent Done false
21290 6334 Acked Read 2 Done Sent Done false
21291 6335 Acked Write 5 Done Sent Done false
21292 6336 Acked Read 4 Done Sent Done false
21293 6337 Acked Read 1 Done Sent Done false
21294 6338 Acked Write 10 Done Sent Done false
21295 6339 Acked Write 6 Done Sent Done false
21296 6340 Acked Write 6 Done Sent Done false
21297 6341 Acked Write 7 Done Sent Done false
21298 6342 Acked Write 8 Done Sent Done false
21299 6343 Acked Write 7 Done Sent Done false
21300 6344 Acked Read 1 Done Sent Done false
21301 6345 Acked Read 9 Done Sent Done false
21302 6346 Acked Write 1 Done Sent Done false
21303 6347 Acked Read 6 Done Sent Done false
21304 6348 Acked Write 4 Done Sent Done false
21305 6349 Acked Read 2 Done Sent Done false
21306 6350 Acked Read 8 Done Sent Done false
21307 6351 Acked Read 8 Done Sent Done false
21308 6352 Acked Read 7 Done Sent Done false
21309 6353 Acked Write 7 Done Sent Done false
21310 6354 Acked Write 10 Done Sent Done false
21311 6355 Acked Flush 0 Done Sent Done false
21312 6356 Acked Read 1 Done Sent Done false
21313 6357 Acked Write 6 Done Sent Done false
21314 6358 Acked Read 6 Done Sent Done false
21315 6359 Acked Read 4 Done Sent Done false
21316 6360 Acked Write 9 Done Sent Done false
21317 6361 Acked Read 6 Done Sent Done false
21318 6362 Acked Read 5 Done Sent Done false
21319 6363 Acked Write 5 Done Sent Done false
21320 6364 Acked Flush 0 Done Sent Done false
21321 6365 Acked Write 8 Done Sent Done false
21322 6366 Acked Read 4 Done Sent Done false
21323 6367 Acked Read 8 Done Sent Done false
21324 6368 Acked Read 8 Done Sent Done false
21325 6369 Acked Write 3 Done Sent Done false
21326 6370 Acked Read 7 Done Sent Done false
21327 6371 Acked Read 10 Done Sent Done false
21328 6372 Acked Read 7 Done Sent Done false
21329 6373 Acked Read 2 Done Sent Done false
21330 6374 Acked Read 10 Done Sent Done false
21331 6375 Acked Write 1 Done Sent Done false
21332 6376 Acked Read 8 Done Sent Done false
21333 6377 Acked Read 4 Done Sent Done false
21334 6378 Acked Read 10 Done Sent Done false
21335 6379 Acked Read 1 Done Sent Done false
21336 6380 Acked Read 6 Done Sent Done false
21337 6381 Acked Read 6 Done Sent Done false
21338 6382 Acked Flush 0 Done Sent Done false
21339 6383 Acked Write 1 Done Sent Done false
21340 6384 Acked Write 8 Done Sent Done false
21341 6385 Acked Read 2 Done Sent Done false
21342 6386 Acked Flush 0 Done Sent Done false
21343 6387 Acked Write 5 Done Sent Done false
21344 6388 Acked Write 3 Done Sent Done false
21345 6389 Acked Read 10 Done Sent Done false
21346 6390 Acked Read 1 Done Sent Done false
21347 6391 Acked Flush 0 Done Sent Done false
21348 6392 Acked Read 6 Done Sent Done false
21349 6393 Acked Write 5 Done Sent Done false
21350 6394 Acked Read 4 Done Sent Done false
21351 6395 Acked Read 10 Done Sent Done false
21352 6396 Acked Write 3 Done Sent Done false
21353 6397 Acked Write 2 Done Sent Done false
21354 6398 Acked Read 9 Done Sent Done false
21355 6399 Acked Write 7 Done Sent Done false
21356 6400 Acked Read 10 Done Sent Done false
21357 6401 Acked Read 7 Done Sent Done false
21358 6402 Acked Write 5 Done Sent Done false
21359 6403 Acked Write 4 Done Sent Done false
21360 6404 Acked Flush 0 Done Sent Done false
21361 6405 Acked Write 7 Done Sent Done false
21362 6406 Acked Read 5 Done Sent Done false
21363 6407 Acked Write 6 Done Sent Done false
21364 6408 Acked Read 7 Done Sent Done false
21365 6409 Acked Write 2 Done Sent Done false
21366 6410 Acked Write 10 Done Sent Done false
21367 6411 Acked Read 2 Done Sent Done false
21368 6412 Acked Read 2 Done Sent Done false
21369 6413 Acked Write 10 Done Sent Done false
21370 6414 Acked Write 8 Done Sent Done false
21371 6415 Acked Read 2 Done Sent Done false
21372 6416 Acked Write 3 Done Sent Done false
21373 6417 Acked Write 9 Done Sent Done false
21374 6418 Acked Read 2 Done Sent Done false
21375 6419 Acked Read 5 Done Sent Done false
21376 6420 Acked Read 7 Done Sent Done false
21377 6421 Acked Write 6 Done Sent Done false
21378 6422 Acked Read 3 Done Sent Done false
21379 6423 Acked Read 7 Done Sent Done false
21380 6424 Acked Write 4 Done Sent Done false
21381 6425 Acked Write 10 Done Sent Done false
21382 6426 Acked Read 9 Done Sent Done false
21383 6427 Acked Write 3 Done Sent Done false
21384 6428 Acked Flush 0 Done Sent Done false
21385 6429 Acked Write 5 Done Sent Done false
21386 6430 Acked Read 3 Done Sent Done false
21387 6431 Acked Write 2 Done Sent Done false
21388 6432 Acked Read 6 Done Sent Done false
21389 6433 Acked Read 1 Done Sent Done false
21390 STATES DS:0 DS:1 DS:2 TOTAL
21391 Sent 0 301 0 301
21392 Done 301 0 301 602
21393 Skipped 0 0 0 0
21394 Error 0 0 0 0
21395 Last Flush: Some(JobId(6428)) Some(JobId(6132)) Some(JobId(6428))
21396 Downstairs last five completed: 6132 6131 6130 6129 6128
21397 Upstairs last five completed: 6433 6432 6431 6430 6429
21398 CLIENT: Up:0 ds:301 act:2
213992025-03-17T22:39:17.922ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
214002025-03-17T22:39:18.909ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21401 [8820] got start action
21402 [8820] I am going to restart
21403 Starting downstairs at port 8820
21404 Make output file at "/tmp/dsc/downstairs-8820.txt"
21405 [8820][1] reports Starting
21406 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1226)
21407 [8820][1] reports Running
214082025-03-17T22:39:19.760ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
214092025-03-17T22:39:20.761ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21410 ----------------------------------------------------------------
21411 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21412 States: Active Offline Active
21413 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21414 6133 Acked Read 5 Done Sent Done false
21415 6134 Acked Write 2 Done Sent Done false
21416 6135 Acked Read 2 Done Sent Done false
21417 6136 Acked Read 6 Done Sent Done false
21418 6137 Acked Read 6 Done Sent Done false
21419 6138 Acked Write 7 Done Sent Done false
21420 6139 Acked Read 10 Done Sent Done false
21421 6140 Acked Write 8 Done Sent Done false
21422 6141 Acked Write 2 Done Sent Done false
21423 6142 Acked Write 2 Done Sent Done false
21424 6143 Acked Flush 0 Done Sent Done false
21425 6144 Acked Read 10 Done Sent Done false
21426 6145 Acked Read 10 Done Sent Done false
21427 6146 Acked Write 1 Done Sent Done false
21428 6147 Acked Write 2 Done Sent Done false
21429 6148 Acked Flush 0 Done Sent Done false
21430 6149 Acked Read 1 Done Sent Done false
21431 6150 Acked Read 10 Done Sent Done false
21432 6151 Acked Write 3 Done Sent Done false
21433 6152 Acked Write 6 Done Sent Done false
21434 6153 Acked Read 1 Done Sent Done false
21435 6154 Acked Read 9 Done Sent Done false
21436 6155 Acked Read 8 Done Sent Done false
21437 6156 Acked Read 3 Done Sent Done false
21438 6157 Acked Write 7 Done Sent Done false
21439 6158 Acked Read 8 Done Sent Done false
21440 6159 Acked Write 5 Done Sent Done false
21441 6160 Acked Flush 0 Done Sent Done false
21442 6161 Acked Read 2 Done Sent Done false
21443 6162 Acked Write 2 Done Sent Done false
21444 6163 Acked Read 5 Done Sent Done false
21445 6164 Acked Write 9 Done Sent Done false
21446 6165 Acked Read 8 Done Sent Done false
21447 6166 Acked Write 8 Done Sent Done false
21448 6167 Acked Flush 0 Done Sent Done false
21449 6168 Acked Read 9 Done Sent Done false
21450 6169 Acked Read 5 Done Sent Done false
21451 6170 Acked Read 2 Done Sent Done false
21452 6171 Acked Write 1 Done Sent Done false
21453 6172 Acked Write 2 Done Sent Done false
21454 6173 Acked Read 6 Done Sent Done false
21455 6174 Acked Write 4 Done Sent Done false
21456 6175 Acked Flush 0 Done Sent Done false
21457 6176 Acked Read 1 Done Sent Done false
21458 6177 Acked Flush 0 Done Sent Done false
21459 6178 Acked Write 4 Done Sent Done false
21460 6179 Acked Write 2 Done Sent Done false
21461 6180 Acked Write 1 Done Sent Done false
21462 6181 Acked Write 5 Done Sent Done false
21463 6182 Acked Write 1 Done Sent Done false
21464 6183 Acked Write 5 Done Sent Done false
21465 6184 Acked Read 4 Done Sent Done false
21466 6185 Acked Write 9 Done Sent Done false
21467 6186 Acked Write 3 Done Sent Done false
21468 6187 Acked Write 1 Done Sent Done false
21469 6188 Acked Write 7 Done Sent Done false
21470 6189 Acked Write 9 Done Sent Done false
21471 6190 Acked Write 3 Done Sent Done false
21472 6191 Acked Write 9 Done Sent Done false
21473 6192 Acked Read 4 Done Sent Done false
21474 6193 Acked Read 10 Done Sent Done false
21475 6194 Acked Read 7 Done Sent Done false
21476 6195 Acked Read 10 Done Sent Done false
21477 6196 Acked Flush 0 Done Sent Done false
21478 6197 Acked Read 8 Done Sent Done false
21479 6198 Acked Read 7 Done Sent Done false
21480 6199 Acked Write 5 Done Sent Done false
21481 6200 Acked Write 8 Done Sent Done false
21482 6201 Acked Read 6 Done Sent Done false
21483 6202 Acked Read 4 Done Sent Done false
21484 6203 Acked Read 5 Done Sent Done false
21485 6204 Acked Write 5 Done Sent Done false
21486 6205 Acked Read 8 Done Sent Done false
21487 6206 Acked Write 4 Done Sent Done false
21488 6207 Acked Write 7 Done Sent Done false
21489 6208 Acked Read 2 Done Sent Done false
21490 6209 Acked Write 2 Done Sent Done false
21491 6210 Acked Read 4 Done Sent Done false
21492 6211 Acked Read 3 Done Sent Done false
21493 6212 Acked Read 1 Done Sent Done false
21494 6213 Acked Read 9 Done Sent Done false
21495 6214 Acked Write 7 Done Sent Done false
21496 6215 Acked Write 3 Done Sent Done false
21497 6216 Acked Read 7 Done Sent Done false
21498 6217 Acked Read 5 Done Sent Done false
21499 6218 Acked Write 5 Done Sent Done false
21500 6219 Acked Write 3 Done Sent Done false
21501 6220 Acked Write 8 Done Sent Done false
21502 6221 Acked Write 8 Done Sent Done false
21503 6222 Acked Write 4 Done Sent Done false
21504 6223 Acked Read 9 Done Sent Done false
21505 6224 Acked Read 9 Done Sent Done false
21506 6225 Acked Write 9 Done Sent Done false
21507 6226 Acked Read 10 Done Sent Done false
21508 6227 Acked Write 3 Done Sent Done false
21509 6228 Acked Write 8 Done Sent Done false
21510 6229 Acked Read 4 Done Sent Done false
21511 6230 Acked Write 6 Done Sent Done false
21512 6231 Acked Read 2 Done Sent Done false
21513 6232 Acked Flush 0 Done Sent Done false
21514 6233 Acked Read 5 Done Sent Done false
21515 6234 Acked Write 5 Done Sent Done false
21516 6235 Acked Read 9 Done Sent Done false
21517 6236 Acked Read 8 Done Sent Done false
21518 6237 Acked Write 7 Done Sent Done false
21519 6238 Acked Read 9 Done Sent Done false
21520 6239 Acked Read 3 Done Sent Done false
21521 6240 Acked Read 3 Done Sent Done false
21522 6241 Acked Read 4 Done Sent Done false
21523 6242 Acked Read 5 Done Sent Done false
21524 6243 Acked Flush 0 Done Sent Done false
21525 6244 Acked Read 1 Done Sent Done false
21526 6245 Acked Read 10 Done Sent Done false
21527 6246 Acked Read 6 Done Sent Done false
21528 6247 Acked Write 3 Done Sent Done false
21529 6248 Acked Write 4 Done Sent Done false
21530 6249 Acked Read 10 Done Sent Done false
21531 6250 Acked Write 4 Done Sent Done false
21532 6251 Acked Read 10 Done Sent Done false
21533 6252 Acked Read 1 Done Sent Done false
21534 6253 Acked Write 4 Done Sent Done false
21535 6254 Acked Read 4 Done Sent Done false
21536 6255 Acked Read 3 Done Sent Done false
21537 6256 Acked Flush 0 Done Sent Done false
21538 6257 Acked Read 10 Done Sent Done false
21539 6258 Acked Write 6 Done Sent Done false
21540 6259 Acked Flush 0 Done Sent Done false
21541 6260 Acked Write 1 Done Sent Done false
21542 6261 Acked Write 2 Done Sent Done false
21543 6262 Acked Write 7 Done Sent Done false
21544 6263 Acked Read 6 Done Sent Done false
21545 6264 Acked Write 3 Done Sent Done false
21546 6265 Acked Read 9 Done Sent Done false
21547 6266 Acked Read 4 Done Sent Done false
21548 6267 Acked Read 8 Done Sent Done false
21549 6268 Acked Read 9 Done Sent Done false
21550 6269 Acked Write 3 Done Sent Done false
21551 6270 Acked Flush 0 Done Sent Done false
21552 6271 Acked Read 3 Done Sent Done false
21553 6272 Acked Write 6 Done Sent Done false
21554 6273 Acked Read 7 Done Sent Done false
21555 6274 Acked Read 8 Done Sent Done false
21556 6275 Acked Flush 0 Done Sent Done false
21557 6276 Acked Write 7 Done Sent Done false
21558 6277 Acked Flush 0 Done Sent Done false
21559 6278 Acked Flush 0 Done Sent Done false
21560 6279 Acked Write 7 Done Sent Done false
21561 6280 Acked Write 4 Done Sent Done false
21562 6281 Acked Write 7 Done Sent Done false
21563 6282 Acked Write 10 Done Sent Done false
21564 6283 Acked Read 9 Done Sent Done false
21565 6284 Acked Write 2 Done Sent Done false
21566 6285 Acked Read 3 Done Sent Done false
21567 6286 Acked Write 9 Done Sent Done false
21568 6287 Acked Read 10 Done Sent Done false
21569 6288 Acked Write 3 Done Sent Done false
21570 6289 Acked Write 4 Done Sent Done false
21571 6290 Acked Write 1 Done Sent Done false
21572 6291 Acked Write 5 Done Sent Done false
21573 6292 Acked Write 9 Done Sent Done false
21574 6293 Acked Write 6 Done Sent Done false
21575 6294 Acked Read 7 Done Sent Done false
21576 6295 Acked Read 9 Done Sent Done false
21577 6296 Acked Read 3 Done Sent Done false
21578 6297 Acked Write 10 Done Sent Done false
21579 6298 Acked Write 9 Done Sent Done false
21580 6299 Acked Write 5 Done Sent Done false
21581 6300 Acked Write 8 Done Sent Done false
21582 6301 Acked Read 10 Done Sent Done false
21583 6302 Acked Write 3 Done Sent Done false
21584 6303 Acked Read 1 Done Sent Done false
21585 6304 Acked Write 9 Done Sent Done false
21586 6305 Acked Read 5 Done Sent Done false
21587 6306 Acked Flush 0 Done Sent Done false
21588 6307 Acked Write 10 Done Sent Done false
21589 6308 Acked Write 8 Done Sent Done false
21590 6309 Acked Read 2 Done Sent Done false
21591 6310 Acked Write 7 Done Sent Done false
21592 6311 Acked Write 6 Done Sent Done false
21593 6312 Acked Read 2 Done Sent Done false
21594 6313 Acked Read 9 Done Sent Done false
21595 6314 Acked Read 7 Done Sent Done false
21596 6315 Acked Read 3 Done Sent Done false
21597 6316 Acked Write 5 Done Sent Done false
21598 6317 Acked Read 6 Done Sent Done false
21599 6318 Acked Read 5 Done Sent Done false
21600 6319 Acked Write 2 Done Sent Done false
21601 6320 Acked Read 4 Done Sent Done false
21602 6321 Acked Read 10 Done Sent Done false
21603 6322 Acked Flush 0 Done Sent Done false
21604 6323 Acked Write 1 Done Sent Done false
21605 6324 Acked Read 2 Done Sent Done false
21606 6325 Acked Write 1 Done Sent Done false
21607 6326 Acked Read 7 Done Sent Done false
21608 6327 Acked Read 10 Done Sent Done false
21609 6328 Acked Read 8 Done Sent Done false
21610 6329 Acked Write 10 Done Sent Done false
21611 6330 Acked Write 6 Done Sent Done false
21612 6331 Acked Write 9 Done Sent Done false
21613 6332 Acked Read 6 Done Sent Done false
21614 6333 Acked Flush 0 Done Sent Done false
21615 6334 Acked Read 2 Done Sent Done false
21616 6335 Acked Write 5 Done Sent Done false
21617 6336 Acked Read 4 Done Sent Done false
21618 6337 Acked Read 1 Done Sent Done false
21619 6338 Acked Write 10 Done Sent Done false
21620 6339 Acked Write 6 Done Sent Done false
21621 6340 Acked Write 6 Done Sent Done false
21622 6341 Acked Write 7 Done Sent Done false
21623 6342 Acked Write 8 Done Sent Done false
21624 6343 Acked Write 7 Done Sent Done false
21625 6344 Acked Read 1 Done Sent Done false
21626 6345 Acked Read 9 Done Sent Done false
21627 6346 Acked Write 1 Done Sent Done false
21628 6347 Acked Read 6 Done Sent Done false
21629 6348 Acked Write 4 Done Sent Done false
21630 6349 Acked Read 2 Done Sent Done false
21631 6350 Acked Read 8 Done Sent Done false
21632 6351 Acked Read 8 Done Sent Done false
21633 6352 Acked Read 7 Done Sent Done false
21634 6353 Acked Write 7 Done Sent Done false
21635 6354 Acked Write 10 Done Sent Done false
21636 6355 Acked Flush 0 Done Sent Done false
21637 6356 Acked Read 1 Done Sent Done false
21638 6357 Acked Write 6 Done Sent Done false
21639 6358 Acked Read 6 Done Sent Done false
21640 6359 Acked Read 4 Done Sent Done false
21641 6360 Acked Write 9 Done Sent Done false
21642 6361 Acked Read 6 Done Sent Done false
21643 6362 Acked Read 5 Done Sent Done false
21644 6363 Acked Write 5 Done Sent Done false
21645 6364 Acked Flush 0 Done Sent Done false
21646 6365 Acked Write 8 Done Sent Done false
21647 6366 Acked Read 4 Done Sent Done false
21648 6367 Acked Read 8 Done Sent Done false
21649 6368 Acked Read 8 Done Sent Done false
21650 6369 Acked Write 3 Done Sent Done false
21651 6370 Acked Read 7 Done Sent Done false
21652 6371 Acked Read 10 Done Sent Done false
21653 6372 Acked Read 7 Done Sent Done false
21654 6373 Acked Read 2 Done Sent Done false
21655 6374 Acked Read 10 Done Sent Done false
21656 6375 Acked Write 1 Done Sent Done false
21657 6376 Acked Read 8 Done Sent Done false
21658 6377 Acked Read 4 Done Sent Done false
21659 6378 Acked Read 10 Done Sent Done false
21660 6379 Acked Read 1 Done Sent Done false
21661 6380 Acked Read 6 Done Sent Done false
21662 6381 Acked Read 6 Done Sent Done false
21663 6382 Acked Flush 0 Done Sent Done false
21664 6383 Acked Write 1 Done Sent Done false
21665 6384 Acked Write 8 Done Sent Done false
21666 6385 Acked Read 2 Done Sent Done false
21667 6386 Acked Flush 0 Done Sent Done false
21668 6387 Acked Write 5 Done Sent Done false
21669 6388 Acked Write 3 Done Sent Done false
21670 6389 Acked Read 10 Done Sent Done false
21671 6390 Acked Read 1 Done Sent Done false
21672 6391 Acked Flush 0 Done Sent Done false
21673 6392 Acked Read 6 Done Sent Done false
21674 6393 Acked Write 5 Done Sent Done false
21675 6394 Acked Read 4 Done Sent Done false
21676 6395 Acked Read 10 Done Sent Done false
21677 6396 Acked Write 3 Done Sent Done false
21678 6397 Acked Write 2 Done Sent Done false
21679 6398 Acked Read 9 Done Sent Done false
21680 6399 Acked Write 7 Done Sent Done false
21681 6400 Acked Read 10 Done Sent Done false
21682 6401 Acked Read 7 Done Sent Done false
21683 6402 Acked Write 5 Done Sent Done false
21684 6403 Acked Write 4 Done Sent Done false
21685 6404 Acked Flush 0 Done Sent Done false
21686 6405 Acked Write 7 Done Sent Done false
21687 6406 Acked Read 5 Done Sent Done false
21688 6407 Acked Write 6 Done Sent Done false
21689 6408 Acked Read 7 Done Sent Done false
21690 6409 Acked Write 2 Done Sent Done false
21691 6410 Acked Write 10 Done Sent Done false
21692 6411 Acked Read 2 Done Sent Done false
21693 6412 Acked Read 2 Done Sent Done false
21694 6413 Acked Write 10 Done Sent Done false
21695 6414 Acked Write 8 Done Sent Done false
21696 6415 Acked Read 2 Done Sent Done false
21697 6416 Acked Write 3 Done Sent Done false
21698 6417 Acked Write 9 Done Sent Done false
21699 6418 Acked Read 2 Done Sent Done false
21700 6419 Acked Read 5 Done Sent Done false
21701 6420 Acked Read 7 Done Sent Done false
21702 6421 Acked Write 6 Done Sent Done false
21703 6422 Acked Read 3 Done Sent Done false
21704 6423 Acked Read 7 Done Sent Done false
21705 6424 Acked Write 4 Done Sent Done false
21706 6425 Acked Write 10 Done Sent Done false
21707 6426 Acked Read 9 Done Sent Done false
21708 6427 Acked Write 3 Done Sent Done false
21709 6428 Acked Flush 0 Done Sent Done false
21710 6429 Acked Write 5 Done Sent Done false
21711 6430 Acked Read 3 Done Sent Done false
21712 6431 Acked Write 2 Done Sent Done false
21713 6432 Acked Read 6 Done Sent Done false
21714 6433 Acked Read 1 Done Sent Done false
21715 STATES DS:0 DS:1 DS:2 TOTAL
21716 Sent 0 301 0 301
21717 Done 301 0 301 602
21718 Skipped 0 0 0 0
21719 Error 0 0 0 0
21720 Last Flush: Some(JobId(6428)) Some(JobId(6132)) Some(JobId(6428))
21721 Downstairs last five completed: 6132 6131 6130 6129 6128
21722 Upstairs last five completed: 6433 6432 6431 6430 6429
21723 CLIENT: Up:0 ds:301 act:2
217242025-03-17T22:39:22.179ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217252025-03-17T22:39:22.763ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217262025-03-17T22:39:23.176ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217272025-03-17T22:39:23.176ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39814) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217282025-03-17T22:39:23.177ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217292025-03-17T22:39:23.177ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217302025-03-17T22:39:23.177ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217312025-03-17T22:39:23.177ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217322025-03-17T22:39:23.177ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217332025-03-17T22:39:23.177ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
217342025-03-17T22:39:23.178ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21735 ----------------------------------------------------------------
21736 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21737 Downstairs last five completed: 6434 6433 6432 6431 6430
21738 Upstairs last five completed: 6434 6433 6432 6431 6430
21739 CLIENT: Up:0 ds:0 act:3
21740 Replay: All jobs finished
21741 Main task has work to do, go find it
21742 got dsc Stop(2)
21743 stop 2
21744 Mar 17 22:39:25.425 INFO request completed[8830] Got stop action so:false kr:false
21745 , latency_us: 311, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 29af20be-734f-46a8-9748-2108d4215749, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
21746 Mar 17 22:39:25.426 INFO request completed, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:39:25.426644373Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
21747 latency_us: 151, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:39:25.426697307Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
21748 200, uri: /state/cid/2, method: GET, req_id:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:39:25.426713168Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
21749 8c2e1bdd-6982-4a35-a9f8-87b78516ea69, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
217502025-03-17T22:39:25.426ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
21751 [8830] Exited with: ExitStatus(unix_wait_status(9))
21752 [8830][2] reports Exit
21753 Mar 17 22:39:29.429 INFO request completed, latency_us: 324, response_code: 200, uri: /state/cid/2, method: GET, req_id: 968af958-9920-435e-9bb1-e403246a7d1f, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
21754 001/300 Write block 1197 len 36864 data: 3 3 3 3 2 2 3 4 3
21755 002/300 Read block 2932 len 40960
21756 003/300 Flush
21757 004/300 Flush
21758 005/300 Write block 4846 len 16384 data: 6 6 4 4
21759 006/300 Read block 4848 len 24576
21760 007/300 Write block 4976 len 4096 data: 2
21761 008/300 Read block 4587 len 28672
21762 009/300 Read block 3691 len 40960
21763 010/300 Write block 1620 len 12288 data: 4 5 6
21764 011/300 Read block 4032 len 40960
21765 012/300 Write block 1934 len 16384 data: 2 2 3 3
21766 013/300 Read block 4464 len 4096
21767 014/300 Read block 3876 len 4096
21768 015/300 Read block 1015 len 12288
21769 016/300 Write block 1962 len 4096 data: 4
21770 017/300 Write block 2965 len 20480 data: 5 4 3 3 3
21771 018/300 Write block 663 len 36864 data: 7 7 6 6 5 5 4 4 6
21772 019/300 Read block 3937 len 8192
21773 020/300 Write block 2056 len 20480 data: 3 3 4 3 3
21774 021/300 Write block 4287 len 4096 data: 4
21775 022/300 Read block 1992 len 4096
21776 023/300 Write block 4619 len 24576 data: 2 2 2 2 3 4
21777 024/300 Read block 1338 len 20480
21778 025/300 Write block 639 len 16384 data: 4 4 5 4
21779 026/300 Write block 607 len 28672 data: 4 3 4 4 4 5 6
21780 027/300 Read block 647 len 28672
21781 028/300 Write block 100 len 40960 data: 5 5 5 6 6 5 5 3 5 6
21782 029/300 Write block 2976 len 20480 data: 6 6 8 9 8
21783 030/300 Read block 2943 len 28672
21784 031/300 Read block 202 len 20480
21785 032/300 Read block 3514 len 28672
21786 033/300 Read block 1244 len 36864
21787 034/300 Write block 4679 len 8192 data: 4 5
21788 035/300 Write block 1188 len 40960 data: 3 4 5 5 5 6 5 5 4 4
21789 036/300 Flush
21790 037/300 Read block 3383 len 8192
21791 038/300 Flush
21792 039/300 Flush
21793 040/300 Flush
21794 041/300 Read block 1890 len 16384
21795 042/300 Write block 2296 len 4096 data: 5
21796 043/300 Read block 3893 len 4096
21797 044/300 Read block 3736 len 36864
21798 045/300 Write block 846 len 40960 data: 6 5 4 3 4 5 6 6 5 6
21799 046/300 Write block 1659 len 16384 data: 5 6 5 4
21800 047/300 Write block 1632 len 32768 data: 3 3 4 5 4 5 5 5
21801 048/300 Read block 1004 len 36864
21802 049/300 Write block 4829 len 40960 data: 4 5 5 5 4 4 4 3 3 4
21803 050/300 Read block 2152 len 40960
21804 051/300 Flush
21805 052/300 Read block 2689 len 36864
21806 053/300 Read block 371 len 32768
21807 054/300 Write block 4471 len 28672 data: 2 2 2 2 2 2 2
21808 055/300 Flush
21809 056/300 Write block 3493 len 12288 data: 3 3 3
21810 057/300 Read block 3237 len 8192
21811 058/300 Read block 4419 len 12288
21812 059/300 Read block 2500 len 16384
21813 060/300 Read block 3692 len 16384
21814 061/300 Read block 1563 len 36864
21815 062/300 Read block 4911 len 40960
21816 063/300 Write block 554 len 36864 data: 5 6 6 4 4 5 5 3 4
21817 064/300 Write block 2547 len 32768 data: 7 5 5 7 7 5 5 5
21818 065/300 Write block 2034 len 36864 data: 9 8 7 7 7 7 8 7 8
21819 066/300 Flush
21820 067/300 Read block 3497 len 28672
21821 068/300 Write block 4092 len 12288 data: 5 4 4
21822 069/300 Write block 3964 len 36864 data: 5 5 5 3 3 3 3 3 2
21823 070/300 Write block 1036 len 28672 data: 2 3 3 3 3 3 4
21824 071/300 Write block 3322 len 24576 data: 7 6 5 5 3 3
21825 072/300 Flush
21826 073/300 Read block 795 len 28672
21827 074/300 Write block 3030 len 32768 data: 6 5 5 5 6 4 4 3
21828 075/300 Write block 1136 len 28672 data: 4 4 4 4 4 5 5
21829 076/300 Read block 3835 len 24576
21830 077/300 Read block 4753 len 8192
21831 078/300 Read block 3520 len 20480
21832 079/300 Write block 3722 len 24576 data: 8 7 6 5 5 5
21833 080/300 Flush
21834 081/300 Read block 1348 len 4096
21835 082/300 Write block 1593 len 24576 data: 7 8 9 9 8 7
21836 083/300 Read block 2718 len 24576
21837 084/300 Write block 4765 len 40960 data: 4 4 4 4 3 3 3 3 6 5
21838 085/300 Write block 83 len 32768 data: 6 4 4 3 3 2 3 3
21839 086/300 Read block 3307 len 4096
21840 087/300 Read block 2706 len 24576
21841 088/300 Flush
21842 089/300 Write block 3745 len 20480 data: 2 2 2 2 2
21843 090/300 Read block 2774 len 32768
21844 091/300 Read block 3358 len 40960
21845 092/300 Write block 4400 len 4096 data: 4
21846 093/300 Read block 2635 len 24576
21847 094/300 Flush
21848 095/300 Read block 4193 len 24576
21849 096/300 Read block 3607 len 40960
21850 097/300 Read block 780 len 36864
21851 098/300 Read block 1491 len 32768
21852 099/300 Write block 3652 len 12288 data: 4 5 5
21853 100/300 Read block 4202 len 40960
21854 101/300 Read block 1440 len 16384
21855 102/300 Read block 852 len 36864
21856 103/300 Flush
21857 104/300 Write block 4480 len 28672 data: 2 2 2 2 3 3 4
21858 105/300 Write block 451 len 4096 data: 8
21859 106/300 Read block 901 len 32768
21860 107/300 Write block 1957 len 28672 data: 5 5 5 5 4 5 5
21861 108/300 Write block 3142 len 4096 data: 3
21862 109/300 Flush
21863 110/300 Read block 4703 len 32768
21864 111/300 Read block 61 len 16384
21865 112/300 Read block 579 len 32768
21866 113/300 Read block 84 len 16384
21867 114/300 Read block 2828 len 4096
21868 115/300 Read block 132 len 16384
21869 116/300 Read block 1881 len 32768
21870 117/300 Read block 2235 len 40960
21871 118/300 Read block 1660 len 40960
21872 119/300 Write block 4281 len 24576 data: 3 4 4 4 4 4
21873 120/300 Write block 617 len 16384 data: 6 5 4 4
21874 121/300 Flush
21875 122/300 Read block 4813 len 40960
21876 123/300 Write block 4918 len 24576 data: 6 6 5 6 5 6
21877 124/300 Write block 3429 len 8192 data: 5 5
21878 125/300 Write block 574 len 28672 data: 3 3 3 3 3 4 4
21879 126/300 Write block 3102 len 8192 data: 3 3
21880 127/300 Read block 3859 len 36864
21881 128/300 Read block 4485 len 16384
21882 129/300 Read block 3448 len 28672
21883 130/300 Write block 2252 len 36864 data: 2 3 4 4 4 4 5 5 4
21884 131/300 Read block 3422 len 12288
21885 132/300 Read block 1626 len 24576
21886 133/300 Read block 4600 len 20480
21887 134/300 Read block 2206 len 20480
21888 135/300 Read block 4058 len 8192
21889 136/300 Read block 823 len 36864
21890 137/300 Write block 4919 len 40960 data: 7 6 7 6 7 7 5 5 5 6
21891 138/300 Flush
21892 139/300 Read block 664 len 40960
21893 140/300 Read block 4033 len 28672
21894 141/300 Read block 253 len 16384
21895 142/300 Write block 342 len 12288 data: 7 6 5
21896 143/300 Read block 4844 len 16384
21897 144/300 Write block 3420 len 40960 data: 7 8 7 6 6 6 5 5 5 6
21898 145/300 Read block 4272 len 8192
21899 146/300 Read block 1291 len 40960
21900 147/300 Write block 1821 len 12288 data: 4 4 4
21901 148/300 Read block 3920 len 28672
21902 149/300 Read block 2193 len 24576
21903 150/300 Read block 2152 len 36864
21904 151/300 Read block 2071 len 20480
21905 152/300 Read block 2055 len 40960
21906 153/300 Read block 4075 len 24576
21907 154/300 Flush
21908 155/300 Write block 1253 len 4096 data: 3
21909 156/300 Write block 1945 len 32768 data: 4 4 4 5 5 5 2 2
21910 157/300 Read block 1085 len 12288
21911 158/300 Read block 663 len 4096
21912 159/300 Read block 878 len 32768
21913 160/300 Read block 2668 len 20480
21914 161/300 Write block 718 len 12288 data: 4 3 3
21915 162/300 Write block 1000 len 4096 data: 3
21916 163/300 Write block 167 len 32768 data: 3 3 2 3 4 4 4 5
21917 164/300 Flush
21918 165/300 Read block 4438 len 32768
21919 166/300 Write block 3008 len 4096 data: 3
21920 167/300 Write block 847 len 4096 data: 6
21921 168/300 Flush
21922 169/300 Write block 126 len 8192 data: 7 6
21923 170/300 Read block 1519 len 16384
21924 171/300 Read block 1474 len 4096
21925 172/300 Read block 4970 len 20480
21926 173/300 Flush
21927 174/300 Write block 2503 len 16384 data: 2 2 2 2
21928 175/300 Write block 4458 len 24576 data: 2 3 3 3 3 3
21929 176/300 Read block 3556 len 8192
21930 177/300 Write block 1485 len 4096 data: 5
21931 178/300 Flush
21932 179/300 Flush
21933 180/300 Read block 4383 len 4096
21934 181/300 Write block 748 len 12288 data: 9 7 7
21935 182/300 Flush
21936 183/300 Write block 1452 len 20480 data: 2 4 7 7 7
21937 184/300 Write block 1686 len 36864 data: 4 4 4 4 4 4 4 3 3
21938 185/300 Write block 2276 len 24576 data: 3 2 2 2 2 3
21939 186/300 Flush
21940 187/300 Write block 729 len 32768 data: 7 6 6 5 6 5 5 5
21941 188/300 Flush
21942 189/300 Read block 3250 len 12288
21943 190/300 Write block 2838 len 20480 data: 5 4 3 2 3
21944 191/300 Read block 96 len 12288
21945 192/300 Write block 1605 len 12288 data: 4 4 4
21946 193/300 Read block 2104 len 8192
21947 194/300 Write block 3246 len 12288 data: 4 4 4
21948 195/300 Write block 1547 len 36864 data: 4 4 3 2 3 3 3 3 3
21949 196/300 Read block 1676 len 20480
21950 197/300 Write block 4013 len 8192 data: 6 5
21951 198/300 Read block 2416 len 32768
21952 199/300 Read block 3936 len 12288
21953 200/300 Read block 3700 len 40960
21954 201/300 Read block 1440 len 8192
21955 202/300 Write block 33 len 28672 data: 4 4 5 4 4 3 4
21956 203/300 Read block 2565 len 24576
21957 204/300 Write block 1372 len 40960 data: 3 3 4 5 5 6 6 5 6 5
21958 205/300 Read block 693 len 12288
21959 206/300 Write block 4365 len 32768 data: 4 5 7 7 5 6 6 6
21960 207/300 Write block 4171 len 36864 data: 4 4 3 3 2 3 4 4 5
21961 208/300 Read block 1903 len 16384
21962 209/300 Read block 3107 len 16384
21963 210/300 Flush
21964 211/300 Write block 3714 len 24576 data: 5 5 6 4 5 6
21965 212/300 Flush
21966 213/300 Read block 3384 len 32768
21967 214/300 Flush
21968 215/300 Read block 863 len 12288
21969 216/300 Flush
21970 217/300 Write block 1679 len 24576 data: 3 3 4 4 5 5
21971 218/300 Write block 4093 len 32768 data: 5 5 6 7 7 6 6 7
21972 219/300 Write block 2785 len 32768 data: 4 4 4 6 6 7 7 7
21973 220/300 Write block 254 len 20480 data: 7 6 6 5 5
21974 221/300 Write block 1598 len 4096 data: 8
21975 222/300 Read block 3005 len 4096
21976 223/300 Read block 1187 len 28672
21977 224/300 Write block 760 len 32768 data: 2 2 2 2 4 4 6 7
21978 225/300 Write block 4038 len 28672 data: 5 5 5 6 5 5 4
21979 226/300 Write block 3271 len 40960 data: 2 2 2 2 3 4 4 4 3 5
21980 227/300 Read block 4420 len 24576
21981 228/300 Read block 212 len 20480
21982 229/300 Flush
21983 230/300 Write block 2864 len 28672 data: 3 4 4 3 3 4 4
21984 231/300 Write block 1187 len 4096 data: 3
21985 232/300 Write block 545 len 40960 data: 4 4 3 3 4 5 6 6 5 6
21986 233/300 Write block 4013 len 20480 data: 7 6 5 6 6
21987 234/300 Read block 3809 len 24576
21988 235/300 Write block 1274 len 16384 data: 4 4 4 5
21989 236/300 Read block 3021 len 28672
21990 237/300 Write block 1427 len 40960 data: 4 2 2 3 3 5 5 5 6 8
21991 238/300 Read block 2064 len 20480
21992 239/300 Write block 4375 len 4096 data: 8
21993 240/300 Write block 4527 len 24576 data: 5 6 6 5 5 5
21994 241/300 Read block 3165 len 24576
21995 242/300 Read block 118 len 32768
21996 243/300 Read block 4368 len 32768
21997 244/300 Write block 4802 len 4096 data: 4
21998 245/300 Read block 652 len 36864
21999 246/300 Read block 1872 len 32768
22000 247/300 Read block 3883 len 32768
22001 248/300 Write block 1013 len 8192 data: 3 3
22002 249/300 Write block 1464 len 12288 data: 4 4 4
22003 250/300 Write block 3465 len 32768 data: 4 4 5 5 5 6 8 7
22004 251/300 Read block 2489 len 20480
22005 252/300 Write block 876 len 28672 data: 4 5 4 4 4 5 4
22006 253/300 Flush
22007 254/300 Read block 935 len 24576
22008 255/300 Read block 1634 len 20480
22009 256/300 Write block 2802 len 12288 data: 3 3 5
22010 257/300 Write block 4988 len 32768 data: 3 4 3 3 2 2 2 2
22011 258/300 Flush
22012 259/300 Read block 999 len 36864
22013 260/300 Read block 1790 len 8192
22014 261/300 Read block 1838 len 20480
22015 262/300 Read block 4578 len 24576
22016 263/300 Read block 777 len 12288
22017 264/300 Flush
22018 265/300 Read block 3540 len 12288
22019 266/300 Read block 3163 len 8192
22020 267/300 Read block 2584 len 32768
22021 268/300 Read block 3654 len 16384
22022 269/300 Read block 2914 len 36864
22023 270/300 Read block 1773 len 12288
22024 271/300 Write block 1022 len 12288 data: 2 3 3
22025 272/300 Read block 2996 len 20480
22026 273/300 Flush
22027 274/300 Read block 3106 len 40960
22028 275/300 Read block 1452 len 32768
22029 276/300 Write block 777 len 16384 data: 4 5 6 6
22030 277/300 Write block 375 len 4096 data: 4
22031 278/300 Write block 2135 len 36864 data: 6 6 5 4 3 4 4 4 4
22032 279/300 Write block 43 len 20480 data: 4 4 4 4 4
22033 280/300 Write block 4022 len 16384 data: 5 6 5 6
22034 281/300 Flush
22035 282/300 Write block 2430 len 32768 data: 4 4 4 4 3 2 5 5
22036 283/300 Write block 4456 len 40960 data: 3 2 3 4 4 4 4 4 6 6
22037 284/300 Read block 1874 len 40960
22038 285/300 Flush
22039 286/300 Read block 4455 len 28672
22040 287/300 Write block 1233 len 24576 data: 4 4 4 3 2 3
22041 288/300 Read block 1697 len 8192
22042 289/300 Read block 1249 len 16384
22043 290/300 Read block 3318 len 40960
22044 291/300 Read block 4592 len 40960
22045 292/300 Write block 928 len 32768 data: 5 5 4 4 4 4 5 6
22046 293/300 Flush
22047 294/300 Read block 1484 len 20480
22048 295/300 Read block 2792 len 8192
22049 296/300 Read block 3507 len 4096
22050 297/300 Write block 246 len 8192 data: 3 3
22051 298/300 Write block 669 len 32768 data: 5 5 7 6 5 5 6 6
22052 299/300 Flush
22053 300/300 Read block 2175 len 28672
22054 301/300 Write block 4796 len 36864 data: 5 4 5 6 5 4 5 5 5
22055 Main task has work to do, go find it
22056 got dsc Start(2)
22057 start 2
22058 Mar 17 22:39:29.674 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 18977829-1e76-4a83-a44a-0783b29a92db, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
22059 Replay: started 2, returned:Ok(())
22060 ----------------------------------------------------------------
22061 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22062 States: Active Active Offline
22063 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22064 6435 Acked Write 9 Done Done Sent false
22065 6436 Acked Read 10 Done Done Sent false
22066 6437 Acked Flush 0 Done Done Sent false
22067 6438 Acked Flush 0 Done Done Sent false
22068 6439 Acked Write 4 Done Done Sent false
22069 6440 Acked Read 6 Done Done Sent false
22070 6441 Acked Write 1 Done Done Sent false
22071 6442 Acked Read 7 Done Done Sent false
22072 6443 Acked Read 10 Done Done Sent false
22073 6444 Acked Write 3 Done Done Sent false
22074 6445 Acked Read 10 Done Done Sent false
22075 6446 Acked Write 4 Done Done Sent false
22076 6447 Acked Read 1 Done Done Sent false
22077 6448 Acked Read 1 Done Done Sent false
22078 6449 Acked Read 3 Done Done Sent false
22079 6450 Acked Write 1 Done Done Sent false
22080 6451 Acked Write 5 Done Done Sent false
22081 6452 Acked Write 9 Done Done Sent false
22082 6453 Acked Read 2 Done Done Sent false
22083 6454 Acked Write 5 Done Done Sent false
22084 6455 Acked Write 1 Done Done Sent false
22085 6456 Acked Read 1 Done Done Sent false
22086 6457 Acked Write 6 Done Done Sent false
22087 6458 Acked Read 5 Done Done Sent false
22088 6459 Acked Write 4 Done Done Sent false
22089 6460 Acked Write 7 Done Done Sent false
22090 6461 Acked Read 7 Done Done Sent false
22091 6462 Acked Write 10 Done Done Sent false
22092 6463 Acked Write 5 Done Done Sent false
22093 6464 Acked Read 7 Done Done Sent false
22094 6465 Acked Read 5 Done Done Sent false
22095 6466 Acked Read 7 Done Done Sent false
22096 6467 Acked Read 9 Done Done Sent false
22097 6468 Acked Write 2 Done Done Sent false
22098 6469 Acked Write 10 Done Done Sent false
22099 6470 Acked Flush 0 Done Done Sent false
22100 6471 Acked Read 2 Done Done Sent false
22101 6472 Acked Flush 0 Done Done Sent false
22102 6473 Acked Flush 0 Done Done Sent false
22103 6474 Acked Flush 0 Done Done Sent false
22104 6475 Acked Read 4 Done Done Sent false
22105 6476 Acked Write 1 Done Done Sent false
22106 6477 Acked Read 1 Done Done Sent false
22107 6478 Acked Read 9 Done Done Sent false
22108 6479 Acked Write 10 Done Done Sent false
22109 6480 Acked Write 4 Done Done Sent false
22110 6481 Acked Write 8 Done Done Sent false
22111 6482 Acked Read 9 Done Done Sent false
22112 6483 Acked Write 10 Done Done Sent false
22113 6484 Acked Read 10 Done Done Sent false
22114 6485 Acked Flush 0 Done Done Sent false
22115 6486 Acked Read 9 Done Done Sent false
22116 6487 Acked Read 8 Done Done Sent false
22117 6488 Acked Write 7 Done Done Sent false
22118 6489 Acked Flush 0 Done Done Sent false
22119 6490 Acked Write 3 Done Done Sent false
22120 6491 Acked Read 2 Done Done Sent false
22121 6492 Acked Read 3 Done Done Sent false
22122 6493 Acked Read 4 Done Done Sent false
22123 6494 Acked Read 4 Done Done Sent false
22124 6495 Acked Read 9 Done Done Sent false
22125 6496 Acked Read 10 Done Done Sent false
22126 6497 Acked Write 9 Done Done Sent false
22127 6498 Acked Write 8 Done Done Sent false
22128 6499 Acked Write 9 Done Done Sent false
22129 6500 Acked Flush 0 Done Done Sent false
22130 6501 Acked Read 7 Done Done Sent false
22131 6502 Acked Write 3 Done Done Sent false
22132 6503 Acked Write 9 Done Done Sent false
22133 6504 Acked Write 7 Done Done Sent false
22134 6505 Acked Write 6 Done Done Sent false
22135 6506 Acked Flush 0 Done Done Sent false
22136 6507 Acked Read 7 Done Done Sent false
22137 6508 Acked Write 8 Done Done Sent false
22138 6509 Acked Write 7 Done Done Sent false
22139 6510 Acked Read 6 Done Done Sent false
22140 6511 Acked Read 2 Done Done Sent false
22141 6512 Acked Read 5 Done Done Sent false
22142 6513 Acked Write 6 Done Done Sent false
22143 6514 Acked Flush 0 Done Done Sent false
22144 6515 Acked Read 1 Done Done Sent false
22145 6516 Acked Write 6 Done Done Sent false
22146 6517 Acked Read 6 Done Done Sent false
22147 6518 Acked Write 10 Done Done Sent false
22148 6519 Acked Write 8 Done Done Sent false
22149 6520 Acked Read 1 Done Done Sent false
22150 6521 Acked Read 6 Done Done Sent false
22151 6522 Acked Flush 0 Done Done Sent false
22152 6523 Acked Write 5 Done Done Sent false
22153 6524 Acked Read 8 Done Done Sent false
22154 6525 Acked Read 10 Done Done Sent false
22155 6526 Acked Write 1 Done Done Sent false
22156 6527 Acked Read 6 Done Done Sent false
22157 6528 Acked Flush 0 Done Done Sent false
22158 6529 Acked Read 6 Done Done Sent false
22159 6530 Acked Read 10 Done Done Sent false
22160 6531 Acked Read 9 Done Done Sent false
22161 6532 Acked Read 8 Done Done Sent false
22162 6533 Acked Write 3 Done Done Sent false
22163 6534 Acked Read 10 Done Done Sent false
22164 6535 Acked Read 4 Done Done Sent false
22165 6536 Acked Read 9 Done Done Sent false
22166 6537 Acked Flush 0 Done Done Sent false
22167 6538 Acked Write 7 Done Done Sent false
22168 6539 Acked Write 1 Done Done Sent false
22169 6540 Acked Read 8 Done Done Sent false
22170 6541 Acked Write 7 Done Done Sent false
22171 6542 Acked Write 1 Done Done Sent false
22172 6543 Acked Flush 0 Done Done Sent false
22173 6544 Acked Read 8 Done Done Sent false
22174 6545 Acked Read 4 Done Done Sent false
22175 6546 Acked Read 8 Done Done Sent false
22176 6547 Acked Read 4 Done Done Sent false
22177 6548 Acked Read 1 Done Done Sent false
22178 6549 Acked Read 4 Done Done Sent false
22179 6550 Acked Read 8 Done Done Sent false
22180 6551 Acked Read 10 Done Done Sent false
22181 6552 Acked Read 10 Done Done Sent false
22182 6553 Acked Write 6 Done Done Sent false
22183 6554 Acked Write 4 Done Done Sent false
22184 6555 Acked Flush 0 Done Done Sent false
22185 6556 Acked Read 10 Done Done Sent false
22186 6557 Acked Write 6 Done Done Sent false
22187 6558 Acked Write 2 Done Done Sent false
22188 6559 Acked Write 7 Done Done Sent false
22189 6560 Acked Write 2 Done Done Sent false
22190 6561 Acked Read 9 Done Done Sent false
22191 6562 Acked Read 4 Done Done Sent false
22192 6563 Acked Read 7 Done Done Sent false
22193 6564 Acked Write 9 Done Done Sent false
22194 6565 Acked Read 3 Done Done Sent false
22195 6566 Acked Read 6 Done Done Sent false
22196 6567 Acked Read 5 Done Done Sent false
22197 6568 Acked Read 5 Done Done Sent false
22198 6569 Acked Read 2 Done Done Sent false
22199 6570 Acked Read 9 Done Done Sent false
22200 6571 Acked Write 10 Done Done Sent false
22201 6572 Acked Flush 0 Done Done Sent false
22202 6573 Acked Read 10 Done Done Sent false
22203 6574 Acked Read 7 Done Done Sent false
22204 6575 Acked Read 4 Done Done Sent false
22205 6576 Acked Write 3 Done Done Sent false
22206 6577 Acked Read 4 Done Done Sent false
22207 6578 Acked Write 10 Done Done Sent false
22208 6579 Acked Read 2 Done Done Sent false
22209 6580 Acked Read 10 Done Done Sent false
22210 6581 Acked Write 3 Done Done Sent false
22211 6582 Acked Read 7 Done Done Sent false
22212 6583 Acked Read 6 Done Done Sent false
22213 6584 Acked Read 9 Done Done Sent false
22214 6585 Acked Read 5 Done Done Sent false
22215 6586 Acked Read 10 Done Done Sent false
22216 6587 Acked Read 6 Done Done Sent false
22217 6588 Acked Flush 0 Done Done Sent false
22218 6589 Acked Write 1 Done Done Sent false
22219 6590 Acked Write 8 Done Done Sent false
22220 6591 Acked Read 3 Done Done Sent false
22221 6592 Acked Read 1 Done Done Sent false
22222 6593 Acked Read 8 Done Done Sent false
22223 6594 Acked Read 5 Done Done Sent false
22224 6595 Acked Write 3 Done Done Sent false
22225 6596 Acked Write 1 Done Done Sent false
22226 6597 Acked Write 8 Done Done Sent false
22227 6598 Acked Flush 0 Done Done Sent false
22228 6599 Acked Read 8 Done Done Sent false
22229 6600 Acked Write 1 Done Done Sent false
22230 6601 Acked Write 1 Done Done Sent false
22231 6602 Acked Flush 0 Done Done Sent false
22232 6603 Acked Write 2 Done Done Sent false
22233 6604 Acked Read 4 Done Done Sent false
22234 6605 Acked Read 1 Done Done Sent false
22235 6606 Acked Read 5 Done Done Sent false
22236 6607 Acked Flush 0 Done Done Sent false
22237 6608 Acked Write 4 Done Done Sent false
22238 6609 Acked Write 6 Done Done Sent false
22239 6610 Acked Read 2 Done Done Sent false
22240 6611 Acked Write 1 Done Done Sent false
22241 6612 Acked Flush 0 Done Done Sent false
22242 6613 Acked Flush 0 Done Done Sent false
22243 6614 Acked Read 1 Done Done Sent false
22244 6615 Acked Write 3 Done Done Sent false
22245 6616 Acked Flush 0 Done Done Sent false
22246 6617 Acked Write 5 Done Done Sent false
22247 6618 Acked Write 9 Done Done Sent false
22248 6619 Acked Write 6 Done Done Sent false
22249 6620 Acked Flush 0 Done Done Sent false
22250 6621 Acked Write 8 Done Done Sent false
22251 6622 Acked Flush 0 Done Done Sent false
22252 6623 Acked Read 3 Done Done Sent false
22253 6624 Acked Write 5 Done Done Sent false
22254 6625 Acked Read 3 Done Done Sent false
22255 6626 Acked Write 3 Done Done Sent false
22256 6627 Acked Read 2 Done Done Sent false
22257 6628 Acked Write 3 Done Done Sent false
22258 6629 Acked Write 9 Done Done Sent false
22259 6630 Acked Read 5 Done Done Sent false
22260 6631 Acked Write 2 Done Done Sent false
22261 6632 Acked Read 8 Done Done Sent false
22262 6633 Acked Read 3 Done Done Sent false
22263 6634 Acked Read 10 Done Done Sent false
22264 6635 Acked Read 2 Done Done Sent false
22265 6636 Acked Write 7 Done Done Sent false
22266 6637 Acked Read 6 Done Done Sent false
22267 6638 Acked Write 10 Done Done Sent false
22268 6639 Acked Read 3 Done Done Sent false
22269 6640 Acked Write 8 Done Done Sent false
22270 6641 Acked Write 9 Done Done Sent false
22271 6642 Acked Read 4 Done Done Sent false
22272 6643 Acked Read 4 Done Done Sent false
22273 6644 Acked Flush 0 Done Done Sent false
22274 6645 Acked Write 6 Done Done Sent false
22275 6646 Acked Flush 0 Done Done Sent false
22276 6647 Acked Read 8 Done Done Sent false
22277 6648 Acked Flush 0 Done Done Sent false
22278 6649 Acked Read 3 Done Done Sent false
22279 6650 Acked Flush 0 Done Done Sent false
22280 6651 Acked Write 6 Done Done Sent false
22281 6652 Acked Write 8 Done Done Sent false
22282 6653 Acked Write 8 Done Done Sent false
22283 6654 Acked Write 5 Done Done Sent false
22284 6655 Acked Write 1 Done Done Sent false
22285 6656 Acked Read 1 Done Done Sent false
22286 6657 Acked Read 7 Done Done Sent false
22287 6658 Acked Write 8 Done Done Sent false
22288 6659 Acked Write 7 Done Done Sent false
22289 6660 Acked Write 10 Done Done Sent false
22290 6661 Acked Read 6 Done Done Sent false
22291 6662 Acked Read 5 Done Done Sent false
22292 6663 Acked Flush 0 Done Done Sent false
22293 6664 Acked Write 7 Done Done Sent false
22294 6665 Acked Write 1 Done Done Sent false
22295 6666 Acked Write 10 Done Done Sent false
22296 6667 Acked Write 5 Done Done Sent false
22297 6668 Acked Read 6 Done Done Sent false
22298 6669 Acked Write 4 Done Done Sent false
22299 6670 Acked Read 7 Done Done Sent false
22300 6671 Acked Write 10 Done Done Sent false
22301 6672 Acked Read 5 Done Done Sent false
22302 6673 Acked Write 1 Done Done Sent false
22303 6674 Acked Write 6 Done Done Sent false
22304 6675 Acked Read 6 Done Done Sent false
22305 6676 Acked Read 8 Done Done Sent false
22306 6677 Acked Read 8 Done Done Sent false
22307 6678 Acked Write 1 Done Done Sent false
22308 6679 Acked Read 9 Done Done Sent false
22309 6680 Acked Read 8 Done Done Sent false
22310 6681 Acked Read 8 Done Done Sent false
22311 6682 Acked Write 2 Done Done Sent false
22312 6683 Acked Write 3 Done Done Sent false
22313 6684 Acked Write 8 Done Done Sent false
22314 6685 Acked Read 5 Done Done Sent false
22315 6686 Acked Write 7 Done Done Sent false
22316 6687 Acked Flush 0 Done Done Sent false
22317 6688 Acked Read 6 Done Done Sent false
22318 6689 Acked Read 5 Done Done Sent false
22319 6690 Acked Write 3 Done Done Sent false
22320 6691 Acked Write 8 Done Done Sent false
22321 6692 Acked Flush 0 Done Done Sent false
22322 6693 Acked Read 9 Done Done Sent false
22323 6694 Acked Read 2 Done Done Sent false
22324 6695 Acked Read 5 Done Done Sent false
22325 6696 Acked Read 6 Done Done Sent false
22326 6697 Acked Read 3 Done Done Sent false
22327 6698 Acked Flush 0 Done Done Sent false
22328 6699 Acked Read 3 Done Done Sent false
22329 6700 Acked Read 2 Done Done Sent false
22330 6701 Acked Read 8 Done Done Sent false
22331 6702 Acked Read 4 Done Done Sent false
22332 6703 Acked Read 9 Done Done Sent false
22333 6704 Acked Read 3 Done Done Sent false
22334 6705 Acked Write 3 Done Done Sent false
22335 6706 Acked Read 5 Done Done Sent false
22336 6707 Acked Flush 0 Done Done Sent false
22337 6708 Acked Read 10 Done Done Sent false
22338 6709 Acked Read 8 Done Done Sent false
22339 6710 Acked Write 4 Done Done Sent false
22340 6711 Acked Write 1 Done Done Sent false
22341 6712 Acked Write 9 Done Done Sent false
22342 6713 Acked Write 5 Done Done Sent false
22343 6714 Acked Write 4 Done Done Sent false
22344 6715 Acked Flush 0 Done Done Sent false
22345 6716 Acked Write 8 Done Done Sent false
22346 6717 Acked Write 10 Done Done Sent false
22347 6718 Acked Read 10 Done Done Sent false
22348 6719 Acked Flush 0 Done Done Sent false
22349 6720 Acked Read 7 Done Done Sent false
22350 6721 Acked Write 6 Done Done Sent false
22351 6722 Acked Read 2 Done Done Sent false
22352 6723 Acked Read 4 Done Done Sent false
22353 6724 Acked Read 10 Done Done Sent false
22354 6725 Acked Read 10 Done Done Sent false
22355 6726 Acked Write 8 Done Done Sent false
22356 6727 Acked Flush 0 Done Done Sent false
22357 6728 Acked Read 5 Done Done Sent false
22358 6729 Acked Read 2 Done Done Sent false
22359 6730 Acked Read 1 Done Done Sent false
22360 6731 Acked Write 2 Done Done Sent false
22361 6732 Acked Write 8 Done Done Sent false
22362 6733 Acked Flush 0 Done Done Sent false
22363 6734 Acked Read 7 Done Done Sent false
22364 6735 Acked Write 9 Done Done Sent false
22365 STATES DS:0 DS:1 DS:2 TOTAL
22366 Sent 0 0 301 301
22367 Done 301 301 0 602
22368 Skipped 0 0 0 0
22369 Error 0 0 0 0
22370 Last Flush: Some(JobId(6733)) Some(JobId(6733)) Some(JobId(6434))
22371 Downstairs last five completed: 6434 6433 6432 6431 6430
22372 Upstairs last five completed: 6735 6734 6733 6732 6731
22373 CLIENT: Up:0 ds:301 act:2
223742025-03-17T22:39:30.176ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
223752025-03-17T22:39:30.776ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
22376 [8830] got start action
22377 [8830] I am going to restart
22378 Starting downstairs at port 8830
22379 Make output file at "/tmp/dsc/downstairs-8830.txt"
22380 [8830][2] reports Starting
22381 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1227)
22382 [8830][2] reports Running
223832025-03-17T22:39:32.181ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
223842025-03-17T22:39:32.779ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
22385 ----------------------------------------------------------------
22386 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22387 States: Active Active Offline
22388 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22389 6435 Acked Write 9 Done Done Sent false
22390 6436 Acked Read 10 Done Done Sent false
22391 6437 Acked Flush 0 Done Done Sent false
22392 6438 Acked Flush 0 Done Done Sent false
22393 6439 Acked Write 4 Done Done Sent false
22394 6440 Acked Read 6 Done Done Sent false
22395 6441 Acked Write 1 Done Done Sent false
22396 6442 Acked Read 7 Done Done Sent false
22397 6443 Acked Read 10 Done Done Sent false
22398 6444 Acked Write 3 Done Done Sent false
22399 6445 Acked Read 10 Done Done Sent false
22400 6446 Acked Write 4 Done Done Sent false
22401 6447 Acked Read 1 Done Done Sent false
22402 6448 Acked Read 1 Done Done Sent false
22403 6449 Acked Read 3 Done Done Sent false
22404 6450 Acked Write 1 Done Done Sent false
22405 6451 Acked Write 5 Done Done Sent false
22406 6452 Acked Write 9 Done Done Sent false
22407 6453 Acked Read 2 Done Done Sent false
22408 6454 Acked Write 5 Done Done Sent false
22409 6455 Acked Write 1 Done Done Sent false
22410 6456 Acked Read 1 Done Done Sent false
22411 6457 Acked Write 6 Done Done Sent false
22412 6458 Acked Read 5 Done Done Sent false
22413 6459 Acked Write 4 Done Done Sent false
22414 6460 Acked Write 7 Done Done Sent false
22415 6461 Acked Read 7 Done Done Sent false
22416 6462 Acked Write 10 Done Done Sent false
22417 6463 Acked Write 5 Done Done Sent false
22418 6464 Acked Read 7 Done Done Sent false
22419 6465 Acked Read 5 Done Done Sent false
22420 6466 Acked Read 7 Done Done Sent false
22421 6467 Acked Read 9 Done Done Sent false
22422 6468 Acked Write 2 Done Done Sent false
22423 6469 Acked Write 10 Done Done Sent false
22424 6470 Acked Flush 0 Done Done Sent false
22425 6471 Acked Read 2 Done Done Sent false
22426 6472 Acked Flush 0 Done Done Sent false
22427 6473 Acked Flush 0 Done Done Sent false
22428 6474 Acked Flush 0 Done Done Sent false
22429 6475 Acked Read 4 Done Done Sent false
22430 6476 Acked Write 1 Done Done Sent false
22431 6477 Acked Read 1 Done Done Sent false
22432 6478 Acked Read 9 Done Done Sent false
22433 6479 Acked Write 10 Done Done Sent false
22434 6480 Acked Write 4 Done Done Sent false
22435 6481 Acked Write 8 Done Done Sent false
22436 6482 Acked Read 9 Done Done Sent false
22437 6483 Acked Write 10 Done Done Sent false
22438 6484 Acked Read 10 Done Done Sent false
22439 6485 Acked Flush 0 Done Done Sent false
22440 6486 Acked Read 9 Done Done Sent false
22441 6487 Acked Read 8 Done Done Sent false
22442 6488 Acked Write 7 Done Done Sent false
22443 6489 Acked Flush 0 Done Done Sent false
22444 6490 Acked Write 3 Done Done Sent false
22445 6491 Acked Read 2 Done Done Sent false
22446 6492 Acked Read 3 Done Done Sent false
22447 6493 Acked Read 4 Done Done Sent false
22448 6494 Acked Read 4 Done Done Sent false
22449 6495 Acked Read 9 Done Done Sent false
22450 6496 Acked Read 10 Done Done Sent false
22451 6497 Acked Write 9 Done Done Sent false
22452 6498 Acked Write 8 Done Done Sent false
22453 6499 Acked Write 9 Done Done Sent false
22454 6500 Acked Flush 0 Done Done Sent false
22455 6501 Acked Read 7 Done Done Sent false
22456 6502 Acked Write 3 Done Done Sent false
22457 6503 Acked Write 9 Done Done Sent false
22458 6504 Acked Write 7 Done Done Sent false
22459 6505 Acked Write 6 Done Done Sent false
22460 6506 Acked Flush 0 Done Done Sent false
22461 6507 Acked Read 7 Done Done Sent false
22462 6508 Acked Write 8 Done Done Sent false
22463 6509 Acked Write 7 Done Done Sent false
22464 6510 Acked Read 6 Done Done Sent false
22465 6511 Acked Read 2 Done Done Sent false
22466 6512 Acked Read 5 Done Done Sent false
22467 6513 Acked Write 6 Done Done Sent false
22468 6514 Acked Flush 0 Done Done Sent false
22469 6515 Acked Read 1 Done Done Sent false
22470 6516 Acked Write 6 Done Done Sent false
22471 6517 Acked Read 6 Done Done Sent false
22472 6518 Acked Write 10 Done Done Sent false
22473 6519 Acked Write 8 Done Done Sent false
22474 6520 Acked Read 1 Done Done Sent false
22475 6521 Acked Read 6 Done Done Sent false
22476 6522 Acked Flush 0 Done Done Sent false
22477 6523 Acked Write 5 Done Done Sent false
22478 6524 Acked Read 8 Done Done Sent false
22479 6525 Acked Read 10 Done Done Sent false
22480 6526 Acked Write 1 Done Done Sent false
22481 6527 Acked Read 6 Done Done Sent false
22482 6528 Acked Flush 0 Done Done Sent false
22483 6529 Acked Read 6 Done Done Sent false
22484 6530 Acked Read 10 Done Done Sent false
22485 6531 Acked Read 9 Done Done Sent false
22486 6532 Acked Read 8 Done Done Sent false
22487 6533 Acked Write 3 Done Done Sent false
22488 6534 Acked Read 10 Done Done Sent false
22489 6535 Acked Read 4 Done Done Sent false
22490 6536 Acked Read 9 Done Done Sent false
22491 6537 Acked Flush 0 Done Done Sent false
22492 6538 Acked Write 7 Done Done Sent false
22493 6539 Acked Write 1 Done Done Sent false
22494 6540 Acked Read 8 Done Done Sent false
22495 6541 Acked Write 7 Done Done Sent false
22496 6542 Acked Write 1 Done Done Sent false
22497 6543 Acked Flush 0 Done Done Sent false
22498 6544 Acked Read 8 Done Done Sent false
22499 6545 Acked Read 4 Done Done Sent false
22500 6546 Acked Read 8 Done Done Sent false
22501 6547 Acked Read 4 Done Done Sent false
22502 6548 Acked Read 1 Done Done Sent false
22503 6549 Acked Read 4 Done Done Sent false
22504 6550 Acked Read 8 Done Done Sent false
22505 6551 Acked Read 10 Done Done Sent false
22506 6552 Acked Read 10 Done Done Sent false
22507 6553 Acked Write 6 Done Done Sent false
22508 6554 Acked Write 4 Done Done Sent false
22509 6555 Acked Flush 0 Done Done Sent false
22510 6556 Acked Read 10 Done Done Sent false
22511 6557 Acked Write 6 Done Done Sent false
22512 6558 Acked Write 2 Done Done Sent false
22513 6559 Acked Write 7 Done Done Sent false
22514 6560 Acked Write 2 Done Done Sent false
22515 6561 Acked Read 9 Done Done Sent false
22516 6562 Acked Read 4 Done Done Sent false
22517 6563 Acked Read 7 Done Done Sent false
22518 6564 Acked Write 9 Done Done Sent false
22519 6565 Acked Read 3 Done Done Sent false
22520 6566 Acked Read 6 Done Done Sent false
22521 6567 Acked Read 5 Done Done Sent false
22522 6568 Acked Read 5 Done Done Sent false
22523 6569 Acked Read 2 Done Done Sent false
22524 6570 Acked Read 9 Done Done Sent false
22525 6571 Acked Write 10 Done Done Sent false
22526 6572 Acked Flush 0 Done Done Sent false
22527 6573 Acked Read 10 Done Done Sent false
22528 6574 Acked Read 7 Done Done Sent false
22529 6575 Acked Read 4 Done Done Sent false
22530 6576 Acked Write 3 Done Done Sent false
22531 6577 Acked Read 4 Done Done Sent false
22532 6578 Acked Write 10 Done Done Sent false
22533 6579 Acked Read 2 Done Done Sent false
22534 6580 Acked Read 10 Done Done Sent false
22535 6581 Acked Write 3 Done Done Sent false
22536 6582 Acked Read 7 Done Done Sent false
22537 6583 Acked Read 6 Done Done Sent false
22538 6584 Acked Read 9 Done Done Sent false
22539 6585 Acked Read 5 Done Done Sent false
22540 6586 Acked Read 10 Done Done Sent false
22541 6587 Acked Read 6 Done Done Sent false
22542 6588 Acked Flush 0 Done Done Sent false
22543 6589 Acked Write 1 Done Done Sent false
22544 6590 Acked Write 8 Done Done Sent false
22545 6591 Acked Read 3 Done Done Sent false
22546 6592 Acked Read 1 Done Done Sent false
22547 6593 Acked Read 8 Done Done Sent false
22548 6594 Acked Read 5 Done Done Sent false
22549 6595 Acked Write 3 Done Done Sent false
22550 6596 Acked Write 1 Done Done Sent false
22551 6597 Acked Write 8 Done Done Sent false
22552 6598 Acked Flush 0 Done Done Sent false
22553 6599 Acked Read 8 Done Done Sent false
22554 6600 Acked Write 1 Done Done Sent false
22555 6601 Acked Write 1 Done Done Sent false
22556 6602 Acked Flush 0 Done Done Sent false
22557 6603 Acked Write 2 Done Done Sent false
22558 6604 Acked Read 4 Done Done Sent false
22559 6605 Acked Read 1 Done Done Sent false
22560 6606 Acked Read 5 Done Done Sent false
22561 6607 Acked Flush 0 Done Done Sent false
22562 6608 Acked Write 4 Done Done Sent false
22563 6609 Acked Write 6 Done Done Sent false
22564 6610 Acked Read 2 Done Done Sent false
22565 6611 Acked Write 1 Done Done Sent false
22566 6612 Acked Flush 0 Done Done Sent false
22567 6613 Acked Flush 0 Done Done Sent false
22568 6614 Acked Read 1 Done Done Sent false
22569 6615 Acked Write 3 Done Done Sent false
22570 6616 Acked Flush 0 Done Done Sent false
22571 6617 Acked Write 5 Done Done Sent false
22572 6618 Acked Write 9 Done Done Sent false
22573 6619 Acked Write 6 Done Done Sent false
22574 6620 Acked Flush 0 Done Done Sent false
22575 6621 Acked Write 8 Done Done Sent false
22576 6622 Acked Flush 0 Done Done Sent false
22577 6623 Acked Read 3 Done Done Sent false
22578 6624 Acked Write 5 Done Done Sent false
22579 6625 Acked Read 3 Done Done Sent false
22580 6626 Acked Write 3 Done Done Sent false
22581 6627 Acked Read 2 Done Done Sent false
22582 6628 Acked Write 3 Done Done Sent false
22583 6629 Acked Write 9 Done Done Sent false
22584 6630 Acked Read 5 Done Done Sent false
22585 6631 Acked Write 2 Done Done Sent false
22586 6632 Acked Read 8 Done Done Sent false
22587 6633 Acked Read 3 Done Done Sent false
22588 6634 Acked Read 10 Done Done Sent false
22589 6635 Acked Read 2 Done Done Sent false
22590 6636 Acked Write 7 Done Done Sent false
22591 6637 Acked Read 6 Done Done Sent false
22592 6638 Acked Write 10 Done Done Sent false
22593 6639 Acked Read 3 Done Done Sent false
22594 6640 Acked Write 8 Done Done Sent false
22595 6641 Acked Write 9 Done Done Sent false
22596 6642 Acked Read 4 Done Done Sent false
22597 6643 Acked Read 4 Done Done Sent false
22598 6644 Acked Flush 0 Done Done Sent false
22599 6645 Acked Write 6 Done Done Sent false
22600 6646 Acked Flush 0 Done Done Sent false
22601 6647 Acked Read 8 Done Done Sent false
22602 6648 Acked Flush 0 Done Done Sent false
22603 6649 Acked Read 3 Done Done Sent false
22604 6650 Acked Flush 0 Done Done Sent false
22605 6651 Acked Write 6 Done Done Sent false
22606 6652 Acked Write 8 Done Done Sent false
22607 6653 Acked Write 8 Done Done Sent false
22608 6654 Acked Write 5 Done Done Sent false
22609 6655 Acked Write 1 Done Done Sent false
22610 6656 Acked Read 1 Done Done Sent false
22611 6657 Acked Read 7 Done Done Sent false
22612 6658 Acked Write 8 Done Done Sent false
22613 6659 Acked Write 7 Done Done Sent false
22614 6660 Acked Write 10 Done Done Sent false
22615 6661 Acked Read 6 Done Done Sent false
22616 6662 Acked Read 5 Done Done Sent false
22617 6663 Acked Flush 0 Done Done Sent false
22618 6664 Acked Write 7 Done Done Sent false
22619 6665 Acked Write 1 Done Done Sent false
22620 6666 Acked Write 10 Done Done Sent false
22621 6667 Acked Write 5 Done Done Sent false
22622 6668 Acked Read 6 Done Done Sent false
22623 6669 Acked Write 4 Done Done Sent false
22624 6670 Acked Read 7 Done Done Sent false
22625 6671 Acked Write 10 Done Done Sent false
22626 6672 Acked Read 5 Done Done Sent false
22627 6673 Acked Write 1 Done Done Sent false
22628 6674 Acked Write 6 Done Done Sent false
22629 6675 Acked Read 6 Done Done Sent false
22630 6676 Acked Read 8 Done Done Sent false
22631 6677 Acked Read 8 Done Done Sent false
22632 6678 Acked Write 1 Done Done Sent false
22633 6679 Acked Read 9 Done Done Sent false
22634 6680 Acked Read 8 Done Done Sent false
22635 6681 Acked Read 8 Done Done Sent false
22636 6682 Acked Write 2 Done Done Sent false
22637 6683 Acked Write 3 Done Done Sent false
22638 6684 Acked Write 8 Done Done Sent false
22639 6685 Acked Read 5 Done Done Sent false
22640 6686 Acked Write 7 Done Done Sent false
22641 6687 Acked Flush 0 Done Done Sent false
22642 6688 Acked Read 6 Done Done Sent false
22643 6689 Acked Read 5 Done Done Sent false
22644 6690 Acked Write 3 Done Done Sent false
22645 6691 Acked Write 8 Done Done Sent false
22646 6692 Acked Flush 0 Done Done Sent false
22647 6693 Acked Read 9 Done Done Sent false
22648 6694 Acked Read 2 Done Done Sent false
22649 6695 Acked Read 5 Done Done Sent false
22650 6696 Acked Read 6 Done Done Sent false
22651 6697 Acked Read 3 Done Done Sent false
22652 6698 Acked Flush 0 Done Done Sent false
22653 6699 Acked Read 3 Done Done Sent false
22654 6700 Acked Read 2 Done Done Sent false
22655 6701 Acked Read 8 Done Done Sent false
22656 6702 Acked Read 4 Done Done Sent false
22657 6703 Acked Read 9 Done Done Sent false
22658 6704 Acked Read 3 Done Done Sent false
22659 6705 Acked Write 3 Done Done Sent false
22660 6706 Acked Read 5 Done Done Sent false
22661 6707 Acked Flush 0 Done Done Sent false
22662 6708 Acked Read 10 Done Done Sent false
22663 6709 Acked Read 8 Done Done Sent false
22664 6710 Acked Write 4 Done Done Sent false
22665 6711 Acked Write 1 Done Done Sent false
22666 6712 Acked Write 9 Done Done Sent false
22667 6713 Acked Write 5 Done Done Sent false
22668 6714 Acked Write 4 Done Done Sent false
22669 6715 Acked Flush 0 Done Done Sent false
22670 6716 Acked Write 8 Done Done Sent false
22671 6717 Acked Write 10 Done Done Sent false
22672 6718 Acked Read 10 Done Done Sent false
22673 6719 Acked Flush 0 Done Done Sent false
22674 6720 Acked Read 7 Done Done Sent false
22675 6721 Acked Write 6 Done Done Sent false
22676 6722 Acked Read 2 Done Done Sent false
22677 6723 Acked Read 4 Done Done Sent false
22678 6724 Acked Read 10 Done Done Sent false
22679 6725 Acked Read 10 Done Done Sent false
22680 6726 Acked Write 8 Done Done Sent false
22681 6727 Acked Flush 0 Done Done Sent false
22682 6728 Acked Read 5 Done Done Sent false
22683 6729 Acked Read 2 Done Done Sent false
22684 6730 Acked Read 1 Done Done Sent false
22685 6731 Acked Write 2 Done Done Sent false
22686 6732 Acked Write 8 Done Done Sent false
22687 6733 Acked Flush 0 Done Done Sent false
22688 6734 Acked Read 7 Done Done Sent false
22689 6735 Acked Write 9 Done Done Sent false
22690 STATES DS:0 DS:1 DS:2 TOTAL
22691 Sent 0 0 301 301
22692 Done 301 301 0 602
22693 Skipped 0 0 0 0
22694 Error 0 0 0 0
22695 Last Flush: Some(JobId(6733)) Some(JobId(6733)) Some(JobId(6434))
22696 Downstairs last five completed: 6434 6433 6432 6431 6430
22697 Upstairs last five completed: 6735 6734 6733 6732 6731
22698 CLIENT: Up:0 ds:301 act:2
226992025-03-17T22:39:34.177ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227002025-03-17T22:39:34.783ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227012025-03-17T22:39:35.428ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227022025-03-17T22:39:35.428ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44211) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227032025-03-17T22:39:35.429ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227042025-03-17T22:39:35.430ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227052025-03-17T22:39:35.430ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227062025-03-17T22:39:35.430ZINFOcrucible: send last flush ID to this DS: Some(JobId(6434)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227072025-03-17T22:39:35.430ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227082025-03-17T22:39:35.430ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6434)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227092025-03-17T22:39:35.430ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6434)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
22710 ----------------------------------------------------------------
22711 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22712 Downstairs last five completed: 6736 6735 6734 6733 6732
22713 Upstairs last five completed: 6736 6735 6734 6733 6732
22714 CLIENT: Up:0 ds:0 act:3
22715 Replay: All jobs finished
22716 Main task has work to do, go find it
22717 got dsc Stop(1)
22718 stop 1
22719 Mar 17 22:39:37.678 INFO request completed[8820] Got stop action so:false kr:false
22720 , latency_us: 367, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8692ed02-3c49-4ba3-a7cf-e25d51197984, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
227212025-03-17T22:39:37.679ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227222025-03-17T22:39:37.679ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
227232025-03-17T22:39:37.679ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
22724 Mar 17 22:39:37.679 INFO request completed, latency_us: 168, response_code: 200, uri: /state/cid/1, method{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:39:37.679582931Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
22725 : GET, req_id: b0621d21-2089-43a2-8565-a1f16cd157b7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
22726 [8820] Exited with: ExitStatus(unix_wait_status(9))
22727 [8820][1] reports Exit
22728 Mar 17 22:39:41.681 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/1, method: GET, req_id: e80d273b-2bcf-4b37-8636-58995b36267a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
22729 001/300 Read block 420 len 28672
22730 002/300 Write block 2462 len 4096 data: 4
22731 003/300 Write block 4258 len 28672 data: 5 4 4 6 6 8 8
22732 004/300 Read block 2719 len 20480
22733 005/300 Write block 2982 len 32768 data: 6 6 6 6 7 5 5 4
22734 006/300 Read block 3605 len 36864
22735 007/300 Read block 2463 len 32768
22736 008/300 Read block 4326 len 24576
22737 009/300 Write block 3031 len 40960 data: 6 6 6 7 5 5 4 3 3 3
22738 010/300 Write block 3195 len 8192 data: 5 5
22739 011/300 Write block 2968 len 28672 data: 4 4 2 2 2 2 3
22740 012/300 Read block 1199 len 8192
22741 013/300 Write block 4571 len 4096 data: 3
22742 014/300 Read block 1660 len 16384
22743 015/300 Read block 4907 len 28672
22744 016/300 Write block 220 len 28672 data: 2 3 3 4 3 3 3
22745 017/300 Read block 4979 len 40960
22746 018/300 Write block 994 len 40960 data: 3 4 4 3 3 4 4 3 4 5
22747 019/300 Read block 4573 len 40960
22748 020/300 Write block 3002 len 36864 data: 4 3 3 3 3 3 4 3 4
22749 021/300 Write block 4841 len 4096 data: 4
22750 022/300 Read block 1819 len 28672
22751 023/300 Write block 345 len 12288 data: 3 3 3
22752 024/300 Write block 4035 len 40960 data: 4 5 5 6 6 6 7 6 6 5
22753 025/300 Read block 1974 len 24576
22754 026/300 Write block 3388 len 4096 data: 8
22755 027/300 Read block 1832 len 40960
22756 028/300 Read block 3839 len 8192
22757 029/300 Write block 1570 len 24576 data: 4 3 3 3 3 3
22758 030/300 Flush
22759 031/300 Read block 2749 len 20480
22760 032/300 Write block 4006 len 36864 data: 5 3 4 5 5 5 7 8 7
22761 033/300 Read block 585 len 36864
22762 034/300 Read block 2061 len 20480
22763 035/300 Read block 2828 len 4096
22764 036/300 Read block 926 len 36864
22765 037/300 Flush
22766 038/300 Flush
22767 039/300 Write block 2681 len 4096 data: 3
22768 040/300 Write block 2652 len 28672 data: 2 3 2 3 5 6 5
22769 041/300 Read block 2286 len 4096
22770 042/300 Write block 3089 len 4096 data: 4
22771 043/300 Write block 1885 len 28672 data: 4 4 3 3 3 3 2
22772 044/300 Flush
22773 045/300 Write block 2695 len 40960 data: 2 3 3 3 3 3 3 2 2 2
22774 046/300 Read block 2320 len 16384
22775 047/300 Write block 756 len 32768 data: 3 2 2 2 3 3 3 3
22776 048/300 Read block 3077 len 36864
22777 049/300 Read block 3873 len 28672
22778 050/300 Read block 1373 len 20480
22779 051/300 Write block 3035 len 24576 data: 6 6 5 4 4 4
22780 052/300 Flush
22781 053/300 Write block 3297 len 24576 data: 4 3 3 3 4 4
22782 054/300 Flush
22783 055/300 Read block 748 len 36864
22784 056/300 Write block 3206 len 16384 data: 4 6 6 7
22785 057/300 Write block 4690 len 36864 data: 5 6 5 5 5 5 5 4 4
22786 058/300 Write block 2462 len 12288 data: 5 6 6
22787 059/300 Write block 3594 len 28672 data: 6 6 4 5 5 4 4
22788 060/300 Read block 3595 len 28672
22789 061/300 Write block 1302 len 28672 data: 4 2 3 2 3 3 4
22790 062/300 Write block 3055 len 28672 data: 5 5 5 4 5 6 5
22791 063/300 Write block 2238 len 8192 data: 2 2
22792 064/300 Flush
22793 065/300 Read block 2336 len 20480
22794 066/300 Write block 3900 len 12288 data: 6 7 5
22795 067/300 Write block 4560 len 24576 data: 4 5 6 5 5 4
22796 068/300 Write block 4439 len 36864 data: 4 4 4 5 6 4 4 3 3
22797 069/300 Write block 1768 len 20480 data: 3 3 3 5 6
22798 070/300 Write block 968 len 16384 data: 3 4 5 5
22799 071/300 Read block 1931 len 32768
22800 072/300 Read block 3827 len 24576
22801 073/300 Write block 1556 len 8192 data: 3 4
22802 074/300 Write block 3591 len 4096 data: 4
22803 075/300 Read block 2303 len 36864
22804 076/300 Read block 3000 len 16384
22805 077/300 Read block 3032 len 4096
22806 078/300 Read block 631 len 8192
22807 079/300 Write block 2520 len 20480 data: 4 4 4 4 5
22808 080/300 Write block 4115 len 28672 data: 8 7 5 6 7 6 4
22809 081/300 Flush
22810 082/300 Read block 677 len 28672
22811 083/300 Write block 1531 len 24576 data: 3 7 8 6 5 5
22812 084/300 Write block 1404 len 40960 data: 4 5 3 3 3 4 4 5 4 4
22813 085/300 Write block 1965 len 16384 data: 3 3 3 3
22814 086/300 Read block 211 len 20480
22815 087/300 Write block 3069 len 16384 data: 5 5 5 5
22816 088/300 Read block 1663 len 20480
22817 089/300 Write block 2147 len 12288 data: 6 5 5
22818 090/300 Read block 4853 len 12288
22819 091/300 Read block 960 len 32768
22820 092/300 Read block 331 len 24576
22821 093/300 Write block 2715 len 32768 data: 3 3 3 4 4 4 4 5
22822 094/300 Read block 3193 len 20480
22823 095/300 Write block 1644 len 12288 data: 5 5 7
22824 096/300 Read block 757 len 4096
22825 097/300 Write block 4172 len 20480 data: 5 4 4 3 4
22826 098/300 Write block 748 len 16384 data: 10 8 8 4
22827 099/300 Write block 3780 len 16384 data: 7 5 5 7
22828 100/300 Read block 4978 len 24576
22829 101/300 Read block 3155 len 16384
22830 102/300 Write block 1548 len 20480 data: 5 4 3 4 4
22831 103/300 Read block 2491 len 40960
22832 104/300 Read block 4675 len 16384
22833 105/300 Write block 3323 len 20480 data: 7 6 6 4 4
22834 106/300 Write block 1746 len 20480 data: 6 4 4 3 2
22835 107/300 Write block 1082 len 32768 data: 4 4 4 3 3 5 6 5
22836 108/300 Read block 3946 len 8192
22837 109/300 Read block 3827 len 40960
22838 110/300 Read block 414 len 4096
22839 111/300 Flush
22840 112/300 Read block 3532 len 12288
22841 113/300 Read block 1581 len 40960
22842 114/300 Write block 156 len 20480 data: 3 4 4 5 4
22843 115/300 Read block 1847 len 4096
22844 116/300 Read block 981 len 20480
22845 117/300 Write block 439 len 32768 data: 5 6 5 4 5 6 5 5
22846 118/300 Read block 1562 len 8192
22847 119/300 Write block 1940 len 4096 data: 4
22848 120/300 Flush
22849 121/300 Read block 2925 len 4096
22850 122/300 Write block 1319 len 36864 data: 5 4 4 5 5 5 6 7 6
22851 123/300 Read block 3511 len 40960
22852 124/300 Write block 866 len 8192 data: 4 3
22853 125/300 Read block 2658 len 28672
22854 126/300 Write block 1137 len 36864 data: 5 5 5 5 6 6 4 3 3
22855 127/300 Write block 1633 len 16384 data: 4 5 6 5
22856 128/300 Write block 1220 len 12288 data: 5 3 3
22857 129/300 Read block 1318 len 12288
22858 130/300 Read block 3129 len 8192
22859 131/300 Write block 3225 len 20480 data: 2 2 3 2 2
22860 132/300 Write block 1861 len 24576 data: 2 3 3 3 3 4
22861 133/300 Read block 4874 len 8192
22862 134/300 Write block 2752 len 24576 data: 8 8 7 7 5 4
22863 135/300 Read block 1573 len 8192
22864 136/300 Read block 2121 len 8192
22865 137/300 Write block 83 len 12288 data: 7 5 5
22866 138/300 Read block 4667 len 32768
22867 139/300 Read block 2969 len 36864
22868 140/300 Write block 513 len 24576 data: 5 4 3 3 4 4
22869 141/300 Read block 2110 len 16384
22870 142/300 Read block 1735 len 16384
22871 143/300 Write block 2121 len 20480 data: 3 2 2 2 3
22872 144/300 Read block 1848 len 16384
22873 145/300 Read block 1753 len 16384
22874 146/300 Flush
22875 147/300 Read block 1982 len 20480
22876 148/300 Read block 3735 len 36864
22877 149/300 Read block 2599 len 32768
22878 150/300 Write block 42 len 32768 data: 4 5 5 5 5 5 4 2
22879 151/300 Flush
22880 152/300 Write block 4021 len 24576 data: 5 6 7 6 7 6
22881 153/300 Read block 3514 len 40960
22882 154/300 Read block 3981 len 36864
22883 155/300 Read block 700 len 36864
22884 156/300 Write block 1517 len 12288 data: 4 3 3
22885 157/300 Read block 1666 len 12288
22886 158/300 Read block 583 len 4096
22887 159/300 Write block 2545 len 12288 data: 4 7 8
22888 160/300 Write block 1730 len 16384 data: 5 6 6 6
22889 161/300 Write block 3558 len 24576 data: 4 4 4 4 5 6
22890 162/300 Write block 990 len 20480 data: 3 4 5 5 4
22891 163/300 Write block 4109 len 28672 data: 6 6 6 6 8 9 9
22892 164/300 Write block 164 len 16384 data: 5 4 4 4
22893 165/300 Write block 4913 len 4096 data: 6
22894 166/300 Write block 541 len 24576 data: 4 4 3 4 5 5
22895 167/300 Write block 2705 len 32768 data: 2 2 4 3 3 4 3 3
22896 168/300 Read block 2033 len 36864
22897 169/300 Write block 1878 len 20480 data: 5 4 5 5 5
22898 170/300 Read block 4222 len 4096
22899 171/300 Write block 3927 len 24576 data: 5 4 4 5 5 6
22900 172/300 Read block 3239 len 16384
22901 173/300 Read block 128 len 8192
22902 174/300 Read block 1475 len 24576
22903 175/300 Read block 443 len 24576
22904 176/300 Write block 23 len 8192 data: 7 7
22905 177/300 Flush
22906 178/300 Read block 3366 len 16384
22907 179/300 Read block 1066 len 40960
22908 180/300 Read block 2178 len 16384
22909 181/300 Read block 2818 len 36864
22910 182/300 Read block 955 len 24576
22911 183/300 Write block 2863 len 20480 data: 3 4 5 5 4
22912 184/300 Write block 1852 len 40960 data: 5 5 4 4 3 3 3 2 2 3
22913 185/300 Read block 1654 len 8192
22914 186/300 Read block 3154 len 20480
22915 187/300 Read block 2237 len 24576
22916 188/300 Write block 3681 len 4096 data: 8
22917 189/300 Read block 1537 len 36864
22918 190/300 Write block 3818 len 32768 data: 2 2 3 3 3 3 5 5
22919 191/300 Write block 1684 len 32768 data: 6 5 5 5 5 5 5 5
22920 192/300 Read block 886 len 16384
22921 193/300 Read block 2230 len 16384
22922 194/300 Write block 3569 len 16384 data: 7 8 7 7
22923 195/300 Read block 2413 len 4096
22924 196/300 Write block 4807 len 32768 data: 4 3 5 4 4 3 3 3
22925 197/300 Write block 1013 len 4096 data: 4
22926 198/300 Read block 4240 len 4096
22927 199/300 Read block 776 len 12288
22928 200/300 Read block 1902 len 8192
22929 201/300 Read block 897 len 32768
22930 202/300 Write block 4509 len 32768 data: 3 2 2 3 5 5 4 5
22931 203/300 Write block 2451 len 32768 data: 2 4 4 4 4 4 3 2
22932 204/300 Read block 3006 len 28672
22933 205/300 Write block 657 len 4096 data: 6
22934 206/300 Read block 4320 len 12288
22935 207/300 Flush
22936 208/300 Write block 3222 len 8192 data: 4 2
22937 209/300 Read block 1508 len 4096
22938 210/300 Read block 3086 len 20480
22939 211/300 Read block 2351 len 36864
22940 212/300 Flush
22941 213/300 Write block 3192 len 28672 data: 3 4 5 6 6 4 4
22942 214/300 Flush
22943 215/300 Write block 3284 len 32768 data: 4 4 4 4 5 5 6 6
22944 216/300 Write block 4145 len 36864 data: 7 5 3 3 2 2 3 4 4
22945 217/300 Write block 1787 len 32768 data: 7 6 5 4 4 3 3 4
22946 218/300 Write block 4673 len 40960 data: 7 5 4 4 4 3 5 6 5 5
22947 219/300 Write block 4712 len 20480 data: 4 3 3 3 2
22948 220/300 Read block 2215 len 24576
22949 221/300 Read block 684 len 12288
22950 222/300 Read block 4958 len 28672
22951 223/300 Write block 1133 len 8192 data: 2 3
22952 224/300 Write block 928 len 12288 data: 6 6 5
22953 225/300 Read block 2129 len 24576
22954 226/300 Write block 2062 len 12288 data: 2 2 2
22955 227/300 Write block 4320 len 36864 data: 3 3 3 4 4 4 4 5 4
22956 228/300 Read block 2699 len 8192
22957 229/300 Read block 3030 len 24576
22958 230/300 Flush
22959 231/300 Write block 4696 len 32768 data: 6 5 5 4 6 6 6 5
22960 232/300 Read block 2417 len 24576
22961 233/300 Write block 1046 len 12288 data: 3 2 3
22962 234/300 Read block 4155 len 36864
22963 235/300 Write block 1251 len 16384 data: 3 3 4 4
22964 236/300 Read block 2302 len 28672
22965 237/300 Write block 4033 len 12288 data: 3 4 5
22966 238/300 Read block 1834 len 16384
22967 239/300 Write block 4803 len 24576 data: 6 6 5 5 5 4
22968 240/300 Read block 3354 len 40960
22969 241/300 Read block 2148 len 28672
22970 242/300 Read block 1724 len 8192
22971 243/300 Read block 2284 len 12288
22972 244/300 Read block 2694 len 20480
22973 245/300 Flush
22974 246/300 Read block 2444 len 36864
22975 247/300 Write block 1362 len 12288 data: 6 6 6
22976 248/300 Read block 1908 len 24576
22977 249/300 Read block 288 len 8192
22978 250/300 Write block 1554 len 4096 data: 4
22979 251/300 Read block 3807 len 12288
22980 252/300 Read block 1805 len 16384
22981 253/300 Read block 301 len 20480
22982 254/300 Flush
22983 255/300 Write block 416 len 32768 data: 7 7 7 7 6 6 6 5
22984 256/300 Write block 1640 len 24576 data: 5 4 3 2 6 6
22985 257/300 Read block 917 len 4096
22986 258/300 Read block 4837 len 8192
22987 259/300 Write block 2285 len 32768 data: 4 3 4 3 2 2 3 3
22988 260/300 Write block 1312 len 36864 data: 3 5 5 4 4 3 3 6 5
22989 261/300 Read block 1558 len 8192
22990 262/300 Flush
22991 263/300 Write block 4201 len 24576 data: 3 3 4 5 5 6
22992 264/300 Read block 1358 len 12288
22993 265/300 Read block 3461 len 16384
22994 266/300 Read block 4460 len 16384
22995 267/300 Write block 870 len 20480 data: 4 3 4 3 3
22996 268/300 Read block 1096 len 32768
22997 269/300 Write block 3108 len 8192 data: 5 4
22998 270/300 Write block 4816 len 16384 data: 3 3 5 5
22999 271/300 Write block 3661 len 8192 data: 4 5
23000 272/300 Read block 1536 len 12288
23001 273/300 Read block 3896 len 12288
23002 274/300 Write block 3227 len 32768 data: 4 3 3 2 3 4 4 4
23003 275/300 Read block 1980 len 20480
23004 276/300 Read block 3039 len 4096
23005 277/300 Read block 4578 len 16384
23006 278/300 Write block 3156 len 28672 data: 4 3 4 3 3 5 5
23007 279/300 Read block 3366 len 12288
23008 280/300 Read block 521 len 4096
23009 281/300 Write block 3150 len 8192 data: 6 6
23010 282/300 Write block 660 len 16384 data: 4 5 6 8
23011 283/300 Read block 1581 len 12288
23012 284/300 Write block 3719 len 36864 data: 7 6 7 9 8 7 6 6 6
23013 285/300 Write block 1956 len 28672 data: 4 6 6 6 6 5 6
23014 286/300 Read block 3939 len 8192
23015 287/300 Flush
23016 288/300 Read block 2457 len 16384
23017 289/300 Write block 3821 len 32768 data: 4 4 4 6 6 4 3 3
23018 290/300 Read block 2851 len 8192
23019 291/300 Write block 272 len 16384 data: 5 5 5 6
23020 292/300 Write block 2468 len 28672 data: 7 6 7 7 7 7 8
23021 293/300 Read block 3468 len 12288
23022 294/300 Write block 1839 len 20480 data: 5 4 4 4 4
23023 295/300 Read block 795 len 12288
23024 296/300 Write block 2447 len 16384 data: 3 3 3 2
23025 297/300 Read block 708 len 4096
23026 298/300 Write block 4785 len 16384 data: 5 5 4 4
23027 299/300 Read block 2132 len 24576
23028 300/300 Flush
23029 301/300 Write block 4355 len 16384 data: 8 8 8 6
23030 Main task has work to do, go find it
23031 got dsc Start(1)
23032 start 1
23033 Mar 17 22:39:41.918 INFO request completed, latency_us: 259, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8a7ba8bd-48ec-4c91-a617-430c36e11a14, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
23034 Replay: started 1, returned:Ok(())
23035 ----------------------------------------------------------------
23036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23037 States: Active Offline Active
23038 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23039 6737 Acked Read 7 Done Sent Done false
23040 6738 Acked Write 1 Done Sent Done false
23041 6739 Acked Write 7 Done Sent Done false
23042 6740 Acked Read 5 Done Sent Done false
23043 6741 Acked Write 8 Done Sent Done false
23044 6742 Acked Read 9 Done Sent Done false
23045 6743 Acked Read 8 Done Sent Done false
23046 6744 Acked Read 6 Done Sent Done false
23047 6745 Acked Write 10 Done Sent Done false
23048 6746 Acked Write 2 Done Sent Done false
23049 6747 Acked Write 7 Done Sent Done false
23050 6748 Acked Read 2 Done Sent Done false
23051 6749 Acked Write 1 Done Sent Done false
23052 6750 Acked Read 4 Done Sent Done false
23053 6751 Acked Read 7 Done Sent Done false
23054 6752 Acked Write 7 Done Sent Done false
23055 6753 Acked Read 10 Done Sent Done false
23056 6754 Acked Write 10 Done Sent Done false
23057 6755 Acked Read 10 Done Sent Done false
23058 6756 Acked Write 9 Done Sent Done false
23059 6757 Acked Write 1 Done Sent Done false
23060 6758 Acked Read 7 Done Sent Done false
23061 6759 Acked Write 3 Done Sent Done false
23062 6760 Acked Write 10 Done Sent Done false
23063 6761 Acked Read 6 Done Sent Done false
23064 6762 Acked Write 1 Done Sent Done false
23065 6763 Acked Read 10 Done Sent Done false
23066 6764 Acked Read 2 Done Sent Done false
23067 6765 Acked Write 6 Done Sent Done false
23068 6766 Acked Flush 0 Done Sent Done false
23069 6767 Acked Read 5 Done Sent Done false
23070 6768 Acked Write 9 Done Sent Done false
23071 6769 Acked Read 9 Done Sent Done false
23072 6770 Acked Read 5 Done Sent Done false
23073 6771 Acked Read 1 Done Sent Done false
23074 6772 Acked Read 9 Done Sent Done false
23075 6773 Acked Flush 0 Done Sent Done false
23076 6774 Acked Flush 0 Done Sent Done false
23077 6775 Acked Write 1 Done Sent Done false
23078 6776 Acked Write 7 Done Sent Done false
23079 6777 Acked Read 1 Done Sent Done false
23080 6778 Acked Write 1 Done Sent Done false
23081 6779 Acked Write 7 Done Sent Done false
23082 6780 Acked Flush 0 Done Sent Done false
23083 6781 Acked Write 10 Done Sent Done false
23084 6782 Acked Read 4 Done Sent Done false
23085 6783 Acked Write 8 Done Sent Done false
23086 6784 Acked Read 9 Done Sent Done false
23087 6785 Acked Read 7 Done Sent Done false
23088 6786 Acked Read 5 Done Sent Done false
23089 6787 Acked Write 6 Done Sent Done false
23090 6788 Acked Flush 0 Done Sent Done false
23091 6789 Acked Write 6 Done Sent Done false
23092 6790 Acked Flush 0 Done Sent Done false
23093 6791 Acked Read 9 Done Sent Done false
23094 6792 Acked Write 4 Done Sent Done false
23095 6793 Acked Write 9 Done Sent Done false
23096 6794 Acked Write 3 Done Sent Done false
23097 6795 Acked Write 7 Done Sent Done false
23098 6796 Acked Read 7 Done Sent Done false
23099 6797 Acked Write 7 Done Sent Done false
23100 6798 Acked Write 7 Done Sent Done false
23101 6799 Acked Write 2 Done Sent Done false
23102 6800 Acked Flush 0 Done Sent Done false
23103 6801 Acked Read 5 Done Sent Done false
23104 6802 Acked Write 3 Done Sent Done false
23105 6803 Acked Write 6 Done Sent Done false
23106 6804 Acked Write 9 Done Sent Done false
23107 6805 Acked Write 5 Done Sent Done false
23108 6806 Acked Write 4 Done Sent Done false
23109 6807 Acked Read 8 Done Sent Done false
23110 6808 Acked Read 6 Done Sent Done false
23111 6809 Acked Write 2 Done Sent Done false
23112 6810 Acked Write 1 Done Sent Done false
23113 6811 Acked Read 9 Done Sent Done false
23114 6812 Acked Read 4 Done Sent Done false
23115 6813 Acked Read 1 Done Sent Done false
23116 6814 Acked Read 2 Done Sent Done false
23117 6815 Acked Write 5 Done Sent Done false
23118 6816 Acked Write 7 Done Sent Done false
23119 6817 Acked Flush 0 Done Sent Done false
23120 6818 Acked Read 7 Done Sent Done false
23121 6819 Acked Write 6 Done Sent Done false
23122 6820 Acked Write 10 Done Sent Done false
23123 6821 Acked Write 4 Done Sent Done false
23124 6822 Acked Read 5 Done Sent Done false
23125 6823 Acked Write 4 Done Sent Done false
23126 6824 Acked Read 5 Done Sent Done false
23127 6825 Acked Write 3 Done Sent Done false
23128 6826 Acked Read 3 Done Sent Done false
23129 6827 Acked Read 8 Done Sent Done false
23130 6828 Acked Read 6 Done Sent Done false
23131 6829 Acked Write 8 Done Sent Done false
23132 6830 Acked Read 5 Done Sent Done false
23133 6831 Acked Write 3 Done Sent Done false
23134 6832 Acked Read 1 Done Sent Done false
23135 6833 Acked Write 5 Done Sent Done false
23136 6834 Acked Write 4 Done Sent Done false
23137 6835 Acked Write 4 Done Sent Done false
23138 6836 Acked Read 6 Done Sent Done false
23139 6837 Acked Read 4 Done Sent Done false
23140 6838 Acked Write 5 Done Sent Done false
23141 6839 Acked Read 10 Done Sent Done false
23142 6840 Acked Read 4 Done Sent Done false
23143 6841 Acked Write 5 Done Sent Done false
23144 6842 Acked Write 5 Done Sent Done false
23145 6843 Acked Write 8 Done Sent Done false
23146 6844 Acked Read 2 Done Sent Done false
23147 6845 Acked Read 10 Done Sent Done false
23148 6846 Acked Read 1 Done Sent Done false
23149 6847 Acked Flush 0 Done Sent Done false
23150 6848 Acked Read 3 Done Sent Done false
23151 6849 Acked Read 10 Done Sent Done false
23152 6850 Acked Write 5 Done Sent Done false
23153 6851 Acked Read 1 Done Sent Done false
23154 6852 Acked Read 5 Done Sent Done false
23155 6853 Acked Write 8 Done Sent Done false
23156 6854 Acked Read 2 Done Sent Done false
23157 6855 Acked Write 1 Done Sent Done false
23158 6856 Acked Flush 0 Done Sent Done false
23159 6857 Acked Read 1 Done Sent Done false
23160 6858 Acked Write 9 Done Sent Done false
23161 6859 Acked Read 10 Done Sent Done false
23162 6860 Acked Write 2 Done Sent Done false
23163 6861 Acked Read 7 Done Sent Done false
23164 6862 Acked Write 9 Done Sent Done false
23165 6863 Acked Write 4 Done Sent Done false
23166 6864 Acked Write 3 Done Sent Done false
23167 6865 Acked Read 3 Done Sent Done false
23168 6866 Acked Read 2 Done Sent Done false
23169 6867 Acked Write 5 Done Sent Done false
23170 6868 Acked Write 6 Done Sent Done false
23171 6869 Acked Read 2 Done Sent Done false
23172 6870 Acked Write 6 Done Sent Done false
23173 6871 Acked Read 2 Done Sent Done false
23174 6872 Acked Read 2 Done Sent Done false
23175 6873 Acked Write 3 Done Sent Done false
23176 6874 Acked Read 8 Done Sent Done false
23177 6875 Acked Read 9 Done Sent Done false
23178 6876 Acked Write 6 Done Sent Done false
23179 6877 Acked Read 4 Done Sent Done false
23180 6878 Acked Read 4 Done Sent Done false
23181 6879 Acked Write 5 Done Sent Done false
23182 6880 Acked Read 4 Done Sent Done false
23183 6881 Acked Read 4 Done Sent Done false
23184 6882 Acked Flush 0 Done Sent Done false
23185 6883 Acked Read 5 Done Sent Done false
23186 6884 Acked Read 9 Done Sent Done false
23187 6885 Acked Read 8 Done Sent Done false
23188 6886 Acked Write 8 Done Sent Done false
23189 6887 Acked Flush 0 Done Sent Done false
23190 6888 Acked Write 6 Done Sent Done false
23191 6889 Acked Read 10 Done Sent Done false
23192 6890 Acked Read 9 Done Sent Done false
23193 6891 Acked Read 9 Done Sent Done false
23194 6892 Acked Write 3 Done Sent Done false
23195 6893 Acked Read 3 Done Sent Done false
23196 6894 Acked Read 1 Done Sent Done false
23197 6895 Acked Write 3 Done Sent Done false
23198 6896 Acked Write 4 Done Sent Done false
23199 6897 Acked Write 6 Done Sent Done false
23200 6898 Acked Write 5 Done Sent Done false
23201 6899 Acked Write 7 Done Sent Done false
23202 6900 Acked Write 4 Done Sent Done false
23203 6901 Acked Write 1 Done Sent Done false
23204 6902 Acked Write 6 Done Sent Done false
23205 6903 Acked Write 8 Done Sent Done false
23206 6904 Acked Read 9 Done Sent Done false
23207 6905 Acked Write 5 Done Sent Done false
23208 6906 Acked Read 1 Done Sent Done false
23209 6907 Acked Write 6 Done Sent Done false
23210 6908 Acked Read 4 Done Sent Done false
23211 6909 Acked Read 2 Done Sent Done false
23212 6910 Acked Read 6 Done Sent Done false
23213 6911 Acked Read 6 Done Sent Done false
23214 6912 Acked Write 2 Done Sent Done false
23215 6913 Acked Flush 0 Done Sent Done false
23216 6914 Acked Read 4 Done Sent Done false
23217 6915 Acked Read 10 Done Sent Done false
23218 6916 Acked Read 4 Done Sent Done false
23219 6917 Acked Read 9 Done Sent Done false
23220 6918 Acked Read 6 Done Sent Done false
23221 6919 Acked Write 5 Done Sent Done false
23222 6920 Acked Write 10 Done Sent Done false
23223 6921 Acked Read 2 Done Sent Done false
23224 6922 Acked Read 5 Done Sent Done false
23225 6923 Acked Read 6 Done Sent Done false
23226 6924 Acked Write 1 Done Sent Done false
23227 6925 Acked Read 9 Done Sent Done false
23228 6926 Acked Write 8 Done Sent Done false
23229 6927 Acked Write 8 Done Sent Done false
23230 6928 Acked Read 4 Done Sent Done false
23231 6929 Acked Read 4 Done Sent Done false
23232 6930 Acked Write 4 Done Sent Done false
23233 6931 Acked Read 1 Done Sent Done false
23234 6932 Acked Write 8 Done Sent Done false
23235 6933 Acked Write 1 Done Sent Done false
23236 6934 Acked Read 1 Done Sent Done false
23237 6935 Acked Read 3 Done Sent Done false
23238 6936 Acked Read 2 Done Sent Done false
23239 6937 Acked Read 8 Done Sent Done false
23240 6938 Acked Write 8 Done Sent Done false
23241 6939 Acked Write 8 Done Sent Done false
23242 6940 Acked Read 7 Done Sent Done false
23243 6941 Acked Write 1 Done Sent Done false
23244 6942 Acked Read 3 Done Sent Done false
23245 6943 Acked Flush 0 Done Sent Done false
23246 6944 Acked Write 2 Done Sent Done false
23247 6945 Acked Read 1 Done Sent Done false
23248 6946 Acked Read 5 Done Sent Done false
23249 6947 Acked Read 9 Done Sent Done false
23250 6948 Acked Flush 0 Done Sent Done false
23251 6949 Acked Write 7 Done Sent Done false
23252 6950 Acked Flush 0 Done Sent Done false
23253 6951 Acked Write 8 Done Sent Done false
23254 6952 Acked Write 9 Done Sent Done false
23255 6953 Acked Write 8 Done Sent Done false
23256 6954 Acked Write 10 Done Sent Done false
23257 6955 Acked Write 5 Done Sent Done false
23258 6956 Acked Read 6 Done Sent Done false
23259 6957 Acked Read 3 Done Sent Done false
23260 6958 Acked Read 7 Done Sent Done false
23261 6959 Acked Write 2 Done Sent Done false
23262 6960 Acked Write 3 Done Sent Done false
23263 6961 Acked Read 6 Done Sent Done false
23264 6962 Acked Write 3 Done Sent Done false
23265 6963 Acked Write 9 Done Sent Done false
23266 6964 Acked Read 2 Done Sent Done false
23267 6965 Acked Read 6 Done Sent Done false
23268 6966 Acked Flush 0 Done Sent Done false
23269 6967 Acked Write 8 Done Sent Done false
23270 6968 Acked Read 6 Done Sent Done false
23271 6969 Acked Write 3 Done Sent Done false
23272 6970 Acked Read 9 Done Sent Done false
23273 6971 Acked Write 4 Done Sent Done false
23274 6972 Acked Read 7 Done Sent Done false
23275 6973 Acked Write 3 Done Sent Done false
23276 6974 Acked Read 4 Done Sent Done false
23277 6975 Acked Write 6 Done Sent Done false
23278 6976 Acked Read 10 Done Sent Done false
23279 6977 Acked Read 7 Done Sent Done false
23280 6978 Acked Read 2 Done Sent Done false
23281 6979 Acked Read 3 Done Sent Done false
23282 6980 Acked Read 5 Done Sent Done false
23283 6981 Acked Flush 0 Done Sent Done false
23284 6982 Acked Read 9 Done Sent Done false
23285 6983 Acked Write 3 Done Sent Done false
23286 6984 Acked Read 6 Done Sent Done false
23287 6985 Acked Read 2 Done Sent Done false
23288 6986 Acked Write 1 Done Sent Done false
23289 6987 Acked Read 3 Done Sent Done false
23290 6988 Acked Read 4 Done Sent Done false
23291 6989 Acked Read 5 Done Sent Done false
23292 6990 Acked Flush 0 Done Sent Done false
23293 6991 Acked Write 8 Done Sent Done false
23294 6992 Acked Write 6 Done Sent Done false
23295 6993 Acked Read 1 Done Sent Done false
23296 6994 Acked Read 2 Done Sent Done false
23297 6995 Acked Write 8 Done Sent Done false
23298 6996 Acked Write 9 Done Sent Done false
23299 6997 Acked Read 2 Done Sent Done false
23300 6998 Acked Flush 0 Done Sent Done false
23301 6999 Acked Write 6 Done Sent Done false
23302 7000 Acked Read 3 Done Sent Done false
23303 7001 Acked Read 4 Done Sent Done false
23304 7002 Acked Read 4 Done Sent Done false
23305 7003 Acked Write 5 Done Sent Done false
23306 7004 Acked Read 8 Done Sent Done false
23307 7005 Acked Write 2 Done Sent Done false
23308 7006 Acked Write 4 Done Sent Done false
23309 7007 Acked Write 2 Done Sent Done false
23310 7008 Acked Read 3 Done Sent Done false
23311 7009 Acked Read 3 Done Sent Done false
23312 7010 Acked Write 8 Done Sent Done false
23313 7011 Acked Read 5 Done Sent Done false
23314 7012 Acked Read 1 Done Sent Done false
23315 7013 Acked Read 4 Done Sent Done false
23316 7014 Acked Write 7 Done Sent Done false
23317 7015 Acked Read 3 Done Sent Done false
23318 7016 Acked Read 1 Done Sent Done false
23319 7017 Acked Write 2 Done Sent Done false
23320 7018 Acked Write 4 Done Sent Done false
23321 7019 Acked Read 3 Done Sent Done false
23322 7020 Acked Write 9 Done Sent Done false
23323 7021 Acked Write 7 Done Sent Done false
23324 7022 Acked Read 2 Done Sent Done false
23325 7023 Acked Flush 0 Done Sent Done false
23326 7024 Acked Read 4 Done Sent Done false
23327 7025 Acked Write 8 Done Sent Done false
23328 7026 Acked Read 2 Done Sent Done false
23329 7027 Acked Write 4 Done Sent Done false
23330 7028 Acked Write 7 Done Sent Done false
23331 7029 Acked Read 3 Done Sent Done false
23332 7030 Acked Write 5 Done Sent Done false
23333 7031 Acked Read 3 Done Sent Done false
23334 7032 Acked Write 4 Done Sent Done false
23335 7033 Acked Read 1 Done Sent Done false
23336 7034 Acked Write 4 Done Sent Done false
23337 7035 Acked Read 6 Done Sent Done false
23338 7036 Acked Flush 0 Done Sent Done false
23339 7037 Acked Write 4 Done Sent Done false
23340 STATES DS:0 DS:1 DS:2 TOTAL
23341 Sent 0 301 0 301
23342 Done 301 0 301 602
23343 Skipped 0 0 0 0
23344 Error 0 0 0 0
23345 Last Flush: Some(JobId(7036)) Some(JobId(6736)) Some(JobId(7036))
23346 Downstairs last five completed: 6736 6735 6734 6733 6732
23347 Upstairs last five completed: 7037 7036 7035 7034 7033
23348 CLIENT: Up:0 ds:301 act:2
233492025-03-17T22:39:42.795ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
23350 [8820] got start action
23351 [8820] I am going to restart
23352 Starting downstairs at port 8820
23353 Make output file at "/tmp/dsc/downstairs-8820.txt"
23354 [8820][1] reports Starting
23355 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1228)
23356 [8820][1] reports Running
233572025-03-17T22:39:43.797ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
233582025-03-17T22:39:44.799ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
23359 ----------------------------------------------------------------
23360 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23361 States: Active Offline Active
23362 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23363 6737 Acked Read 7 Done Sent Done false
23364 6738 Acked Write 1 Done Sent Done false
23365 6739 Acked Write 7 Done Sent Done false
23366 6740 Acked Read 5 Done Sent Done false
23367 6741 Acked Write 8 Done Sent Done false
23368 6742 Acked Read 9 Done Sent Done false
23369 6743 Acked Read 8 Done Sent Done false
23370 6744 Acked Read 6 Done Sent Done false
23371 6745 Acked Write 10 Done Sent Done false
23372 6746 Acked Write 2 Done Sent Done false
23373 6747 Acked Write 7 Done Sent Done false
23374 6748 Acked Read 2 Done Sent Done false
23375 6749 Acked Write 1 Done Sent Done false
23376 6750 Acked Read 4 Done Sent Done false
23377 6751 Acked Read 7 Done Sent Done false
23378 6752 Acked Write 7 Done Sent Done false
23379 6753 Acked Read 10 Done Sent Done false
23380 6754 Acked Write 10 Done Sent Done false
23381 6755 Acked Read 10 Done Sent Done false
23382 6756 Acked Write 9 Done Sent Done false
23383 6757 Acked Write 1 Done Sent Done false
23384 6758 Acked Read 7 Done Sent Done false
23385 6759 Acked Write 3 Done Sent Done false
23386 6760 Acked Write 10 Done Sent Done false
23387 6761 Acked Read 6 Done Sent Done false
23388 6762 Acked Write 1 Done Sent Done false
23389 6763 Acked Read 10 Done Sent Done false
23390 6764 Acked Read 2 Done Sent Done false
23391 6765 Acked Write 6 Done Sent Done false
23392 6766 Acked Flush 0 Done Sent Done false
23393 6767 Acked Read 5 Done Sent Done false
23394 6768 Acked Write 9 Done Sent Done false
23395 6769 Acked Read 9 Done Sent Done false
23396 6770 Acked Read 5 Done Sent Done false
23397 6771 Acked Read 1 Done Sent Done false
23398 6772 Acked Read 9 Done Sent Done false
23399 6773 Acked Flush 0 Done Sent Done false
23400 6774 Acked Flush 0 Done Sent Done false
23401 6775 Acked Write 1 Done Sent Done false
23402 6776 Acked Write 7 Done Sent Done false
23403 6777 Acked Read 1 Done Sent Done false
23404 6778 Acked Write 1 Done Sent Done false
23405 6779 Acked Write 7 Done Sent Done false
23406 6780 Acked Flush 0 Done Sent Done false
23407 6781 Acked Write 10 Done Sent Done false
23408 6782 Acked Read 4 Done Sent Done false
23409 6783 Acked Write 8 Done Sent Done false
23410 6784 Acked Read 9 Done Sent Done false
23411 6785 Acked Read 7 Done Sent Done false
23412 6786 Acked Read 5 Done Sent Done false
23413 6787 Acked Write 6 Done Sent Done false
23414 6788 Acked Flush 0 Done Sent Done false
23415 6789 Acked Write 6 Done Sent Done false
23416 6790 Acked Flush 0 Done Sent Done false
23417 6791 Acked Read 9 Done Sent Done false
23418 6792 Acked Write 4 Done Sent Done false
23419 6793 Acked Write 9 Done Sent Done false
23420 6794 Acked Write 3 Done Sent Done false
23421 6795 Acked Write 7 Done Sent Done false
23422 6796 Acked Read 7 Done Sent Done false
23423 6797 Acked Write 7 Done Sent Done false
23424 6798 Acked Write 7 Done Sent Done false
23425 6799 Acked Write 2 Done Sent Done false
23426 6800 Acked Flush 0 Done Sent Done false
23427 6801 Acked Read 5 Done Sent Done false
23428 6802 Acked Write 3 Done Sent Done false
23429 6803 Acked Write 6 Done Sent Done false
23430 6804 Acked Write 9 Done Sent Done false
23431 6805 Acked Write 5 Done Sent Done false
23432 6806 Acked Write 4 Done Sent Done false
23433 6807 Acked Read 8 Done Sent Done false
23434 6808 Acked Read 6 Done Sent Done false
23435 6809 Acked Write 2 Done Sent Done false
23436 6810 Acked Write 1 Done Sent Done false
23437 6811 Acked Read 9 Done Sent Done false
23438 6812 Acked Read 4 Done Sent Done false
23439 6813 Acked Read 1 Done Sent Done false
23440 6814 Acked Read 2 Done Sent Done false
23441 6815 Acked Write 5 Done Sent Done false
23442 6816 Acked Write 7 Done Sent Done false
23443 6817 Acked Flush 0 Done Sent Done false
23444 6818 Acked Read 7 Done Sent Done false
23445 6819 Acked Write 6 Done Sent Done false
23446 6820 Acked Write 10 Done Sent Done false
23447 6821 Acked Write 4 Done Sent Done false
23448 6822 Acked Read 5 Done Sent Done false
23449 6823 Acked Write 4 Done Sent Done false
23450 6824 Acked Read 5 Done Sent Done false
23451 6825 Acked Write 3 Done Sent Done false
23452 6826 Acked Read 3 Done Sent Done false
23453 6827 Acked Read 8 Done Sent Done false
23454 6828 Acked Read 6 Done Sent Done false
23455 6829 Acked Write 8 Done Sent Done false
23456 6830 Acked Read 5 Done Sent Done false
23457 6831 Acked Write 3 Done Sent Done false
23458 6832 Acked Read 1 Done Sent Done false
23459 6833 Acked Write 5 Done Sent Done false
23460 6834 Acked Write 4 Done Sent Done false
23461 6835 Acked Write 4 Done Sent Done false
23462 6836 Acked Read 6 Done Sent Done false
23463 6837 Acked Read 4 Done Sent Done false
23464 6838 Acked Write 5 Done Sent Done false
23465 6839 Acked Read 10 Done Sent Done false
23466 6840 Acked Read 4 Done Sent Done false
23467 6841 Acked Write 5 Done Sent Done false
23468 6842 Acked Write 5 Done Sent Done false
23469 6843 Acked Write 8 Done Sent Done false
23470 6844 Acked Read 2 Done Sent Done false
23471 6845 Acked Read 10 Done Sent Done false
23472 6846 Acked Read 1 Done Sent Done false
23473 6847 Acked Flush 0 Done Sent Done false
23474 6848 Acked Read 3 Done Sent Done false
23475 6849 Acked Read 10 Done Sent Done false
23476 6850 Acked Write 5 Done Sent Done false
23477 6851 Acked Read 1 Done Sent Done false
23478 6852 Acked Read 5 Done Sent Done false
23479 6853 Acked Write 8 Done Sent Done false
23480 6854 Acked Read 2 Done Sent Done false
23481 6855 Acked Write 1 Done Sent Done false
23482 6856 Acked Flush 0 Done Sent Done false
23483 6857 Acked Read 1 Done Sent Done false
23484 6858 Acked Write 9 Done Sent Done false
23485 6859 Acked Read 10 Done Sent Done false
23486 6860 Acked Write 2 Done Sent Done false
23487 6861 Acked Read 7 Done Sent Done false
23488 6862 Acked Write 9 Done Sent Done false
23489 6863 Acked Write 4 Done Sent Done false
23490 6864 Acked Write 3 Done Sent Done false
23491 6865 Acked Read 3 Done Sent Done false
23492 6866 Acked Read 2 Done Sent Done false
23493 6867 Acked Write 5 Done Sent Done false
23494 6868 Acked Write 6 Done Sent Done false
23495 6869 Acked Read 2 Done Sent Done false
23496 6870 Acked Write 6 Done Sent Done false
23497 6871 Acked Read 2 Done Sent Done false
23498 6872 Acked Read 2 Done Sent Done false
23499 6873 Acked Write 3 Done Sent Done false
23500 6874 Acked Read 8 Done Sent Done false
23501 6875 Acked Read 9 Done Sent Done false
23502 6876 Acked Write 6 Done Sent Done false
23503 6877 Acked Read 4 Done Sent Done false
23504 6878 Acked Read 4 Done Sent Done false
23505 6879 Acked Write 5 Done Sent Done false
23506 6880 Acked Read 4 Done Sent Done false
23507 6881 Acked Read 4 Done Sent Done false
23508 6882 Acked Flush 0 Done Sent Done false
23509 6883 Acked Read 5 Done Sent Done false
23510 6884 Acked Read 9 Done Sent Done false
23511 6885 Acked Read 8 Done Sent Done false
23512 6886 Acked Write 8 Done Sent Done false
23513 6887 Acked Flush 0 Done Sent Done false
23514 6888 Acked Write 6 Done Sent Done false
23515 6889 Acked Read 10 Done Sent Done false
23516 6890 Acked Read 9 Done Sent Done false
23517 6891 Acked Read 9 Done Sent Done false
23518 6892 Acked Write 3 Done Sent Done false
23519 6893 Acked Read 3 Done Sent Done false
23520 6894 Acked Read 1 Done Sent Done false
23521 6895 Acked Write 3 Done Sent Done false
23522 6896 Acked Write 4 Done Sent Done false
23523 6897 Acked Write 6 Done Sent Done false
23524 6898 Acked Write 5 Done Sent Done false
23525 6899 Acked Write 7 Done Sent Done false
23526 6900 Acked Write 4 Done Sent Done false
23527 6901 Acked Write 1 Done Sent Done false
23528 6902 Acked Write 6 Done Sent Done false
23529 6903 Acked Write 8 Done Sent Done false
23530 6904 Acked Read 9 Done Sent Done false
23531 6905 Acked Write 5 Done Sent Done false
23532 6906 Acked Read 1 Done Sent Done false
23533 6907 Acked Write 6 Done Sent Done false
23534 6908 Acked Read 4 Done Sent Done false
23535 6909 Acked Read 2 Done Sent Done false
23536 6910 Acked Read 6 Done Sent Done false
23537 6911 Acked Read 6 Done Sent Done false
23538 6912 Acked Write 2 Done Sent Done false
23539 6913 Acked Flush 0 Done Sent Done false
23540 6914 Acked Read 4 Done Sent Done false
23541 6915 Acked Read 10 Done Sent Done false
23542 6916 Acked Read 4 Done Sent Done false
23543 6917 Acked Read 9 Done Sent Done false
23544 6918 Acked Read 6 Done Sent Done false
23545 6919 Acked Write 5 Done Sent Done false
23546 6920 Acked Write 10 Done Sent Done false
23547 6921 Acked Read 2 Done Sent Done false
23548 6922 Acked Read 5 Done Sent Done false
23549 6923 Acked Read 6 Done Sent Done false
23550 6924 Acked Write 1 Done Sent Done false
23551 6925 Acked Read 9 Done Sent Done false
23552 6926 Acked Write 8 Done Sent Done false
23553 6927 Acked Write 8 Done Sent Done false
23554 6928 Acked Read 4 Done Sent Done false
23555 6929 Acked Read 4 Done Sent Done false
23556 6930 Acked Write 4 Done Sent Done false
23557 6931 Acked Read 1 Done Sent Done false
23558 6932 Acked Write 8 Done Sent Done false
23559 6933 Acked Write 1 Done Sent Done false
23560 6934 Acked Read 1 Done Sent Done false
23561 6935 Acked Read 3 Done Sent Done false
23562 6936 Acked Read 2 Done Sent Done false
23563 6937 Acked Read 8 Done Sent Done false
23564 6938 Acked Write 8 Done Sent Done false
23565 6939 Acked Write 8 Done Sent Done false
23566 6940 Acked Read 7 Done Sent Done false
23567 6941 Acked Write 1 Done Sent Done false
23568 6942 Acked Read 3 Done Sent Done false
23569 6943 Acked Flush 0 Done Sent Done false
23570 6944 Acked Write 2 Done Sent Done false
23571 6945 Acked Read 1 Done Sent Done false
23572 6946 Acked Read 5 Done Sent Done false
23573 6947 Acked Read 9 Done Sent Done false
23574 6948 Acked Flush 0 Done Sent Done false
23575 6949 Acked Write 7 Done Sent Done false
23576 6950 Acked Flush 0 Done Sent Done false
23577 6951 Acked Write 8 Done Sent Done false
23578 6952 Acked Write 9 Done Sent Done false
23579 6953 Acked Write 8 Done Sent Done false
23580 6954 Acked Write 10 Done Sent Done false
23581 6955 Acked Write 5 Done Sent Done false
23582 6956 Acked Read 6 Done Sent Done false
23583 6957 Acked Read 3 Done Sent Done false
23584 6958 Acked Read 7 Done Sent Done false
23585 6959 Acked Write 2 Done Sent Done false
23586 6960 Acked Write 3 Done Sent Done false
23587 6961 Acked Read 6 Done Sent Done false
23588 6962 Acked Write 3 Done Sent Done false
23589 6963 Acked Write 9 Done Sent Done false
23590 6964 Acked Read 2 Done Sent Done false
23591 6965 Acked Read 6 Done Sent Done false
23592 6966 Acked Flush 0 Done Sent Done false
23593 6967 Acked Write 8 Done Sent Done false
23594 6968 Acked Read 6 Done Sent Done false
23595 6969 Acked Write 3 Done Sent Done false
23596 6970 Acked Read 9 Done Sent Done false
23597 6971 Acked Write 4 Done Sent Done false
23598 6972 Acked Read 7 Done Sent Done false
23599 6973 Acked Write 3 Done Sent Done false
23600 6974 Acked Read 4 Done Sent Done false
23601 6975 Acked Write 6 Done Sent Done false
23602 6976 Acked Read 10 Done Sent Done false
23603 6977 Acked Read 7 Done Sent Done false
23604 6978 Acked Read 2 Done Sent Done false
23605 6979 Acked Read 3 Done Sent Done false
23606 6980 Acked Read 5 Done Sent Done false
23607 6981 Acked Flush 0 Done Sent Done false
23608 6982 Acked Read 9 Done Sent Done false
23609 6983 Acked Write 3 Done Sent Done false
23610 6984 Acked Read 6 Done Sent Done false
23611 6985 Acked Read 2 Done Sent Done false
23612 6986 Acked Write 1 Done Sent Done false
23613 6987 Acked Read 3 Done Sent Done false
23614 6988 Acked Read 4 Done Sent Done false
23615 6989 Acked Read 5 Done Sent Done false
23616 6990 Acked Flush 0 Done Sent Done false
23617 6991 Acked Write 8 Done Sent Done false
23618 6992 Acked Write 6 Done Sent Done false
23619 6993 Acked Read 1 Done Sent Done false
23620 6994 Acked Read 2 Done Sent Done false
23621 6995 Acked Write 8 Done Sent Done false
23622 6996 Acked Write 9 Done Sent Done false
23623 6997 Acked Read 2 Done Sent Done false
23624 6998 Acked Flush 0 Done Sent Done false
23625 6999 Acked Write 6 Done Sent Done false
23626 7000 Acked Read 3 Done Sent Done false
23627 7001 Acked Read 4 Done Sent Done false
23628 7002 Acked Read 4 Done Sent Done false
23629 7003 Acked Write 5 Done Sent Done false
23630 7004 Acked Read 8 Done Sent Done false
23631 7005 Acked Write 2 Done Sent Done false
23632 7006 Acked Write 4 Done Sent Done false
23633 7007 Acked Write 2 Done Sent Done false
23634 7008 Acked Read 3 Done Sent Done false
23635 7009 Acked Read 3 Done Sent Done false
23636 7010 Acked Write 8 Done Sent Done false
23637 7011 Acked Read 5 Done Sent Done false
23638 7012 Acked Read 1 Done Sent Done false
23639 7013 Acked Read 4 Done Sent Done false
23640 7014 Acked Write 7 Done Sent Done false
23641 7015 Acked Read 3 Done Sent Done false
23642 7016 Acked Read 1 Done Sent Done false
23643 7017 Acked Write 2 Done Sent Done false
23644 7018 Acked Write 4 Done Sent Done false
23645 7019 Acked Read 3 Done Sent Done false
23646 7020 Acked Write 9 Done Sent Done false
23647 7021 Acked Write 7 Done Sent Done false
23648 7022 Acked Read 2 Done Sent Done false
23649 7023 Acked Flush 0 Done Sent Done false
23650 7024 Acked Read 4 Done Sent Done false
23651 7025 Acked Write 8 Done Sent Done false
23652 7026 Acked Read 2 Done Sent Done false
23653 7027 Acked Write 4 Done Sent Done false
23654 7028 Acked Write 7 Done Sent Done false
23655 7029 Acked Read 3 Done Sent Done false
23656 7030 Acked Write 5 Done Sent Done false
23657 7031 Acked Read 3 Done Sent Done false
23658 7032 Acked Write 4 Done Sent Done false
23659 7033 Acked Read 1 Done Sent Done false
23660 7034 Acked Write 4 Done Sent Done false
23661 7035 Acked Read 6 Done Sent Done false
23662 7036 Acked Flush 0 Done Sent Done false
23663 7037 Acked Write 4 Done Sent Done false
23664 STATES DS:0 DS:1 DS:2 TOTAL
23665 Sent 0 301 0 301
23666 Done 301 0 301 602
23667 Skipped 0 0 0 0
23668 Error 0 0 0 0
23669 Last Flush: Some(JobId(7036)) Some(JobId(6736)) Some(JobId(7036))
23670 Downstairs last five completed: 6736 6735 6734 6733 6732
23671 Upstairs last five completed: 7037 7036 7035 7034 7033
23672 CLIENT: Up:0 ds:301 act:2
236732025-03-17T22:39:47.185ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236742025-03-17T22:39:47.680ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236752025-03-17T22:39:47.680ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46625) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236762025-03-17T22:39:47.681ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236772025-03-17T22:39:47.681ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236782025-03-17T22:39:47.681ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236792025-03-17T22:39:47.681ZINFOcrucible: send last flush ID to this DS: Some(JobId(6736)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236802025-03-17T22:39:47.681ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236812025-03-17T22:39:47.681ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6736)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236822025-03-17T22:39:47.681ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6736)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
23683 ----------------------------------------------------------------
23684 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23685 Downstairs last five completed: 7038 7037 7036 7035 7034
23686 Upstairs last five completed: 7038 7037 7036 7035 7034
23687 CLIENT: Up:0 ds:0 act:3
23688 Replay: All jobs finished
23689 Main task has work to do, go find it
23690 got dsc Stop(2)
23691 stop 2
23692 Mar 17 22:39:49.923 INFO request completed, latency_us: 352, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a7e8c8e0-605d-4ee2-9d8b-d9c8cf3ee395, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
23693 [8830] Got stop action so:false kr:false
236942025-03-17T22:39:49.924ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236952025-03-17T22:39:49.924ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
236962025-03-17T22:39:49.924ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
23697 Mar 17 22:39:49.924 INFO request completed, latency_us: 174, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1de0df84-33b9-47a8-bd78-ae09943639b7, remote_addr: 127.0.0.1:35180{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:39:49.924513663Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
23698 , local_addr: 127.0.0.1:9998
23699 [8830] Exited with: ExitStatus(unix_wait_status(9))
23700 [8830][2] reports Exit
23701 Mar 17 22:39:53.925 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: a4d0046b-de53-48d9-95d5-4b79c96cccc1, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
23702 001/300 Read block 3738 len 40960
23703 002/300 Write block 3837 len 20480 data: 6 5 8 7 6
23704 003/300 Write block 4569 len 28672 data: 5 3 4 3 3 3 6
23705 004/300 Read block 4048 len 24576
23706 005/300 Read block 4299 len 32768
23707 006/300 Write block 2581 len 28672 data: 7 4 4 2 2 2 2
23708 007/300 Read block 2161 len 36864
23709 008/300 Read block 2765 len 36864
23710 009/300 Write block 1798 len 28672 data: 2 2 2 3 3 4 5
23711 010/300 Flush
23712 011/300 Read block 2778 len 36864
23713 012/300 Read block 2896 len 24576
23714 013/300 Read block 2707 len 36864
23715 014/300 Write block 2081 len 36864 data: 3 3 4 4 4 6 5 5 5
23716 015/300 Write block 283 len 12288 data: 2 4 3
23717 016/300 Read block 3124 len 28672
23718 017/300 Read block 3383 len 28672
23719 018/300 Read block 4308 len 20480
23720 019/300 Read block 1514 len 24576
23721 020/300 Write block 4026 len 8192 data: 7 6
23722 021/300 Read block 4088 len 40960
23723 022/300 Read block 4005 len 28672
23724 023/300 Write block 2091 len 8192 data: 4 3
23725 024/300 Write block 4204 len 20480 data: 6 6 7 7 8
23726 025/300 Write block 3934 len 16384 data: 5 5 4 5
23727 026/300 Write block 2700 len 28672 data: 4 4 3 3 3 3 3
23728 027/300 Read block 1120 len 32768
23729 028/300 Read block 1371 len 4096
23730 029/300 Write block 2445 len 4096 data: 3
23731 030/300 Read block 4230 len 20480
23732 031/300 Write block 387 len 28672 data: 8 8 7 6 5 5 5
23733 032/300 Read block 4288 len 12288
23734 033/300 Write block 1575 len 8192 data: 4 2
23735 034/300 Write block 2977 len 20480 data: 7 9 10 9 7
23736 035/300 Read block 4644 len 28672
23737 036/300 Write block 1910 len 4096 data: 5
23738 037/300 Write block 4216 len 32768 data: 6 6 6 5 5 5 8 6
23739 038/300 Read block 3744 len 4096
23740 039/300 Read block 2445 len 4096
23741 040/300 Read block 3005 len 36864
23742 041/300 Flush
23743 042/300 Read block 4436 len 28672
23744 043/300 Read block 590 len 12288
23745 044/300 Read block 290 len 8192
23746 045/300 Write block 1729 len 20480 data: 5 6 7 7 7
23747 046/300 Read block 3880 len 12288
23748 047/300 Read block 2133 len 40960
23749 048/300 Read block 583 len 28672
23750 049/300 Flush
23751 050/300 Write block 2417 len 12288 data: 3 3 3
23752 051/300 Write block 1837 len 32768 data: 6 5 6 5 5 5 5 6
23753 052/300 Write block 2008 len 8192 data: 3 4
23754 053/300 Flush
23755 054/300 Read block 991 len 20480
23756 055/300 Read block 3013 len 28672
23757 056/300 Write block 3396 len 24576 data: 5 6 6 6 7 6
23758 057/300 Flush
23759 058/300 Write block 1155 len 20480 data: 4 5 5 5 6
23760 059/300 Read block 2000 len 40960
23761 060/300 Read block 733 len 20480
23762 061/300 Write block 3306 len 16384 data: 3 2 3 4
23763 062/300 Read block 2776 len 20480
23764 063/300 Read block 146 len 20480
23765 064/300 Read block 4392 len 8192
23766 065/300 Read block 4207 len 28672
23767 066/300 Write block 3239 len 28672 data: 5 5 5 4 4 3 4
23768 067/300 Flush
23769 068/300 Flush
23770 069/300 Read block 891 len 20480
23771 070/300 Write block 4018 len 4096 data: 5
23772 071/300 Read block 728 len 16384
23773 072/300 Flush
23774 073/300 Write block 855 len 8192 data: 7 6
23775 074/300 Read block 4099 len 16384
23776 075/300 Read block 1565 len 40960
23777 076/300 Write block 2456 len 4096 data: 5
23778 077/300 Read block 233 len 8192
23779 078/300 Read block 3408 len 12288
23780 079/300 Write block 1320 len 8192 data: 6 5
23781 080/300 Flush
23782 081/300 Read block 634 len 12288
23783 082/300 Write block 1170 len 20480 data: 3 3 3 3 3
23784 083/300 Write block 1982 len 20480 data: 8 10 9 9 7
23785 084/300 Write block 4191 len 12288 data: 2 3 3
23786 085/300 Write block 1672 len 24576 data: 4 4 4 3 3 4
23787 086/300 Flush
23788 087/300 Flush
23789 088/300 Read block 534 len 24576
23790 089/300 Read block 3477 len 32768
23791 090/300 Write block 3502 len 32768 data: 3 3 4 3 4 4 5 5
23792 091/300 Read block 2870 len 36864
23793 092/300 Read block 4684 len 36864
23794 093/300 Write block 1981 len 36864 data: 4 9 11 10 10 8 9 8 6
23795 094/300 Read block 2195 len 40960
23796 095/300 Read block 3242 len 8192
23797 096/300 Read block 2367 len 4096
23798 097/300 Flush
23799 098/300 Read block 2966 len 28672
23800 099/300 Read block 151 len 28672
23801 100/300 Write block 936 len 24576 data: 5 5 6 5 6 7
23802 101/300 Write block 3989 len 20480 data: 5 5 5 5 4
23803 102/300 Read block 4781 len 36864
23804 103/300 Write block 4274 len 32768 data: 4 4 4 3 3 2 2 4
23805 104/300 Flush
23806 105/300 Flush
23807 106/300 Write block 2628 len 36864 data: 2 2 4 4 5 5 4 5 3
23808 107/300 Read block 2502 len 32768
23809 108/300 Read block 558 len 28672
23810 109/300 Read block 4176 len 24576
23811 110/300 Write block 4256 len 20480 data: 5 4 6 5 5
23812 111/300 Write block 2120 len 12288 data: 5 4 3
23813 112/300 Read block 2799 len 16384
23814 113/300 Read block 587 len 16384
23815 114/300 Read block 4076 len 4096
23816 115/300 Flush
23817 116/300 Read block 1914 len 12288
23818 117/300 Read block 1401 len 28672
23819 118/300 Read block 4439 len 4096
23820 119/300 Write block 1245 len 12288 data: 4 4 3
23821 120/300 Write block 4573 len 28672 data: 4 4 7 6 4 5 4
23822 121/300 Read block 1467 len 32768
23823 122/300 Write block 2171 len 28672 data: 3 3 3 4 4 4 5
23824 123/300 Write block 1858 len 36864 data: 4 3 3 4 4 4 4 4 5
23825 124/300 Write block 4700 len 16384 data: 7 7 7 6
23826 125/300 Write block 548 len 16384 data: 4 5 6 7
23827 126/300 Read block 1475 len 20480
23828 127/300 Read block 2667 len 40960
23829 128/300 Write block 4565 len 8192 data: 5 4
23830 129/300 Read block 2864 len 8192
23831 130/300 Write block 2342 len 40960 data: 4 4 4 3 3 3 3 3 4 5
23832 131/300 Flush
23833 132/300 Read block 2623 len 12288
23834 133/300 Write block 3985 len 20480 data: 3 4 7 6 6
23835 134/300 Read block 1190 len 12288
23836 135/300 Write block 1290 len 24576 data: 3 4 6 7 7 6
23837 136/300 Read block 323 len 28672
23838 137/300 Read block 746 len 12288
23839 138/300 Write block 3260 len 16384 data: 3 4 4 4
23840 139/300 Flush
23841 140/300 Write block 33 len 20480 data: 5 5 6 5 5
23842 141/300 Read block 4084 len 4096
23843 142/300 Read block 2190 len 28672
23844 143/300 Read block 4750 len 20480
23845 144/300 Write block 830 len 8192 data: 3 3
23846 145/300 Read block 2629 len 8192
23847 146/300 Write block 2356 len 12288 data: 5 4 5
23848 147/300 Write block 37 len 36864 data: 6 4 5 4 4 5 6 6 6
23849 148/300 Write block 2563 len 4096 data: 5
23850 149/300 Write block 1016 len 12288 data: 2 3 4
23851 150/300 Write block 324 len 16384 data: 3 3 3 2
23852 151/300 Write block 1283 len 28672 data: 4 3 2 2 2 3 4
23853 152/300 Read block 1663 len 16384
23854 153/300 Write block 1806 len 16384 data: 4 4 4 4
23855 154/300 Write block 402 len 16384 data: 4 4 5 6
23856 155/300 Read block 3249 len 16384
23857 156/300 Write block 1702 len 8192 data: 3 3
23858 157/300 Read block 3953 len 16384
23859 158/300 Read block 1036 len 36864
23860 159/300 Write block 561 len 16384 data: 4 5 5 4
23861 160/300 Write block 2011 len 12288 data: 4 4 4
23862 161/300 Write block 1803 len 4096 data: 5
23863 162/300 Read block 135 len 12288
23864 163/300 Write block 823 len 8192 data: 3 3
23865 164/300 Write block 3122 len 36864 data: 3 4 3 3 3 4 6 5 6
23866 165/300 Flush
23867 166/300 Write block 1607 len 8192 data: 5 4
23868 167/300 Read block 4444 len 16384
23869 168/300 Read block 4401 len 28672
23870 169/300 Read block 2619 len 4096
23871 170/300 Read block 1079 len 36864
23872 171/300 Read block 3239 len 36864
23873 172/300 Write block 3981 len 40960 data: 2 2 2 2 4 5 8 7 7 6
23874 173/300 Read block 1275 len 24576
23875 174/300 Read block 1746 len 12288
23876 175/300 Read block 2787 len 36864
23877 176/300 Flush
23878 177/300 Write block 78 len 36864 data: 3 3 4 4 4 8 6 6 4
23879 178/300 Write block 947 len 28672 data: 6 5 4 3 3 3 3
23880 179/300 Read block 4942 len 28672
23881 180/300 Write block 2667 len 4096 data: 5
23882 181/300 Read block 4362 len 4096
23883 182/300 Read block 2422 len 12288
23884 183/300 Write block 4636 len 4096 data: 8
23885 184/300 Read block 4817 len 16384
23886 185/300 Write block 3583 len 8192 data: 2 2
23887 186/300 Write block 4460 len 4096 data: 5
23888 187/300 Read block 2611 len 28672
23889 188/300 Read block 2809 len 16384
23890 189/300 Write block 1180 len 16384 data: 3 3 2 2
23891 190/300 Read block 217 len 24576
23892 191/300 Flush
23893 192/300 Read block 4912 len 40960
23894 193/300 Write block 1284 len 36864 data: 4 3 3 3 4 5 4 5 7
23895 194/300 Write block 3714 len 32768 data: 6 6 7 5 6 8 7 8
23896 195/300 Read block 2788 len 16384
23897 196/300 Write block 3375 len 40960 data: 3 3 2 3 3 4 4 5 5 7
23898 197/300 Write block 2416 len 20480 data: 3 4 4 4 3
23899 198/300 Flush
23900 199/300 Write block 1552 len 36864 data: 5 4 5 4 4 5 4 3 4
23901 200/300 Flush
23902 201/300 Flush
23903 202/300 Write block 2884 len 40960 data: 4 4 3 4 4 4 4 5 4 4
23904 203/300 Read block 4718 len 36864
23905 204/300 Write block 1 len 32768 data: 3 2 2 2 2 2 3 4
23906 205/300 Read block 865 len 36864
23907 206/300 Write block 61 len 8192 data: 3 3
23908 207/300 Read block 1666 len 36864
23909 208/300 Read block 4397 len 24576
23910 209/300 Write block 3450 len 8192 data: 3 3
23911 210/300 Read block 254 len 28672
23912 211/300 Write block 4225 len 4096 data: 5
23913 212/300 Read block 4062 len 20480
23914 213/300 Read block 4323 len 4096
23915 214/300 Read block 2634 len 32768
23916 215/300 Read block 3830 len 12288
23917 216/300 Write block 4188 len 40960 data: 3 3 3 3 4 4 3 2 2 2
23918 217/300 Read block 177 len 12288
23919 218/300 Write block 4672 len 4096 data: 6
23920 219/300 Write block 2978 len 20480 data: 10 11 10 8 7
23921 220/300 Read block 4438 len 16384
23922 221/300 Read block 2816 len 12288
23923 222/300 Write block 3734 len 8192 data: 6 6
23924 223/300 Read block 2115 len 36864
23925 224/300 Read block 2135 len 8192
23926 225/300 Flush
23927 226/300 Write block 4415 len 32768 data: 4 3 2 3 3 4 4 5
23928 227/300 Flush
23929 228/300 Read block 3918 len 8192
23930 229/300 Write block 2985 len 8192 data: 7 8
23931 230/300 Read block 2709 len 32768
23932 231/300 Flush
23933 232/300 Read block 1616 len 20480
23934 233/300 Flush
23935 234/300 Read block 1094 len 24576
23936 235/300 Read block 3102 len 12288
23937 236/300 Write block 2663 len 12288 data: 4 4 4
23938 237/300 Read block 1801 len 28672
23939 238/300 Write block 3522 len 32768 data: 5 3 2 2 4 4 4 4
23940 239/300 Flush
23941 240/300 Write block 3165 len 28672 data: 4 4 3 3 3 2 2
23942 241/300 Read block 4145 len 12288
23943 242/300 Read block 4772 len 32768
23944 243/300 Write block 1530 len 36864 data: 3 4 8 9 7 6 6 3 2
23945 244/300 Read block 4652 len 12288
23946 245/300 Flush
23947 246/300 Write block 3997 len 16384 data: 3 3 4 4
23948 247/300 Flush
23949 248/300 Read block 2101 len 16384
23950 249/300 Read block 3998 len 36864
23951 250/300 Read block 2464 len 20480
23952 251/300 Read block 1019 len 24576
23953 252/300 Write block 4659 len 40960 data: 6 6 6 5 5 4 4 4 5 5
23954 253/300 Write block 3291 len 16384 data: 7 6 6 6
23955 254/300 Read block 34 len 36864
23956 255/300 Read block 1782 len 12288
23957 256/300 Write block 1303 len 20480 data: 3 4 3 4 4
23958 257/300 Read block 3679 len 20480
23959 258/300 Read block 4524 len 20480
23960 259/300 Write block 2865 len 32768 data: 6 6 5 4 5 5 3 4
23961 260/300 Write block 4786 len 36864 data: 6 5 5 5 5 5 5 5 5
23962 261/300 Write block 1983 len 24576 data: 12 11 11 9 10 9
23963 262/300 Read block 4856 len 40960
23964 263/300 Write block 3988 len 28672 data: 8 8 7 6 6 5 4
23965 264/300 Read block 2697 len 36864
23966 265/300 Write block 2090 len 16384 data: 5 5 4 3
23967 266/300 Flush
23968 267/300 Write block 3233 len 32768 data: 5 5 5 4 5 5 6 6
23969 268/300 Write block 2223 len 28672 data: 2 3 2 2 2 3 3
23970 269/300 Read block 4726 len 28672
23971 270/300 Write block 727 len 4096 data: 7
23972 271/300 Read block 2998 len 32768
23973 272/300 Write block 2436 len 36864 data: 6 6 5 5 6 7 6 5 5
23974 273/300 Flush
23975 274/300 Write block 3395 len 24576 data: 6 6 7 7 7 8
23976 275/300 Write block 3419 len 36864 data: 6 8 9 8 7 7 7 6 6
23977 276/300 Read block 68 len 8192
23978 277/300 Read block 3760 len 40960
23979 278/300 Read block 489 len 4096
23980 279/300 Read block 2146 len 28672
23981 280/300 Flush
23982 281/300 Read block 3091 len 12288
23983 282/300 Write block 2515 len 8192 data: 4 5
23984 283/300 Write block 2037 len 20480 data: 8 8 8 9 8
23985 284/300 Read block 1425 len 20480
23986 285/300 Read block 655 len 8192
23987 286/300 Read block 1354 len 8192
23988 287/300 Read block 729 len 16384
23989 288/300 Flush
23990 289/300 Write block 1516 len 20480 data: 5 5 4 4 3
23991 290/300 Write block 3033 len 4096 data: 7
23992 291/300 Read block 3054 len 24576
23993 292/300 Write block 1035 len 8192 data: 2 3
23994 293/300 Read block 4039 len 36864
23995 294/300 Read block 3288 len 32768
23996 295/300 Write block 1741 len 16384 data: 4 4 5 6
23997 296/300 Write block 3444 len 8192 data: 3 3
23998 297/300 Read block 582 len 32768
23999 298/300 Write block 652 len 20480 data: 4 4 5 5 5
24000 299/300 Write block 512 len 12288 data: 4 6 5
24001 300/300 Read block 1000 len 4096
24002 301/300 Read block 4812 len 40960
24003 Main task has work to do, go find it
24004 got dsc Start(2)
24005 start 2
24006 Mar 17 22:39:54.165 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/2, method: POST, req_id: 862550eb-da76-4210-b749-c756012b80e7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
24007 Replay: started 2, returned:Ok(())
24008 ----------------------------------------------------------------
24009 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24010 States: Active Active Offline
24011 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24012 7039 Acked Read 10 Done Done Sent false
24013 7040 Acked Write 5 Done Done Sent false
24014 7041 Acked Write 7 Done Done Sent false
24015 7042 Acked Read 6 Done Done Sent false
24016 7043 Acked Read 8 Done Done Sent false
24017 7044 Acked Write 7 Done Done Sent false
24018 7045 Acked Read 9 Done Done Sent false
24019 7046 Acked Read 9 Done Done Sent false
24020 7047 Acked Write 7 Done Done Sent false
24021 7048 Acked Flush 0 Done Done Sent false
24022 7049 Acked Read 9 Done Done Sent false
24023 7050 Acked Read 6 Done Done Sent false
24024 7051 Acked Read 9 Done Done Sent false
24025 7052 Acked Write 9 Done Done Sent false
24026 7053 Acked Write 3 Done Done Sent false
24027 7054 Acked Read 7 Done Done Sent false
24028 7055 Acked Read 7 Done Done Sent false
24029 7056 Acked Read 5 Done Done Sent false
24030 7057 Acked Read 6 Done Done Sent false
24031 7058 Acked Write 2 Done Done Sent false
24032 7059 Acked Read 10 Done Done Sent false
24033 7060 Acked Read 7 Done Done Sent false
24034 7061 Acked Write 2 Done Done Sent false
24035 7062 Acked Write 5 Done Done Sent false
24036 7063 Acked Write 4 Done Done Sent false
24037 7064 Acked Write 7 Done Done Sent false
24038 7065 Acked Read 8 Done Done Sent false
24039 7066 Acked Read 1 Done Done Sent false
24040 7067 Acked Write 1 Done Done Sent false
24041 7068 Acked Read 5 Done Done Sent false
24042 7069 Acked Write 7 Done Done Sent false
24043 7070 Acked Read 3 Done Done Sent false
24044 7071 Acked Write 2 Done Done Sent false
24045 7072 Acked Write 5 Done Done Sent false
24046 7073 Acked Read 7 Done Done Sent false
24047 7074 Acked Write 1 Done Done Sent false
24048 7075 Acked Write 8 Done Done Sent false
24049 7076 Acked Read 1 Done Done Sent false
24050 7077 Acked Read 1 Done Done Sent false
24051 7078 Acked Read 9 Done Done Sent false
24052 7079 Acked Flush 0 Done Done Sent false
24053 7080 Acked Read 7 Done Done Sent false
24054 7081 Acked Read 3 Done Done Sent false
24055 7082 Acked Read 2 Done Done Sent false
24056 7083 Acked Write 5 Done Done Sent false
24057 7084 Acked Read 3 Done Done Sent false
24058 7085 Acked Read 10 Done Done Sent false
24059 7086 Acked Read 7 Done Done Sent false
24060 7087 Acked Flush 0 Done Done Sent false
24061 7088 Acked Write 3 Done Done Sent false
24062 7089 Acked Write 8 Done Done Sent false
24063 7090 Acked Write 2 Done Done Sent false
24064 7091 Acked Flush 0 Done Done Sent false
24065 7092 Acked Read 5 Done Done Sent false
24066 7093 Acked Read 7 Done Done Sent false
24067 7094 Acked Write 6 Done Done Sent false
24068 7095 Acked Flush 0 Done Done Sent false
24069 7096 Acked Write 5 Done Done Sent false
24070 7097 Acked Read 10 Done Done Sent false
24071 7098 Acked Read 5 Done Done Sent false
24072 7099 Acked Write 4 Done Done Sent false
24073 7100 Acked Read 5 Done Done Sent false
24074 7101 Acked Read 5 Done Done Sent false
24075 7102 Acked Read 2 Done Done Sent false
24076 7103 Acked Read 7 Done Done Sent false
24077 7104 Acked Write 7 Done Done Sent false
24078 7105 Acked Flush 0 Done Done Sent false
24079 7106 Acked Flush 0 Done Done Sent false
24080 7107 Acked Read 5 Done Done Sent false
24081 7108 Acked Write 1 Done Done Sent false
24082 7109 Acked Read 4 Done Done Sent false
24083 7110 Acked Flush 0 Done Done Sent false
24084 7111 Acked Write 2 Done Done Sent false
24085 7112 Acked Read 4 Done Done Sent false
24086 7113 Acked Read 10 Done Done Sent false
24087 7114 Acked Write 1 Done Done Sent false
24088 7115 Acked Read 2 Done Done Sent false
24089 7116 Acked Read 3 Done Done Sent false
24090 7117 Acked Write 2 Done Done Sent false
24091 7118 Acked Flush 0 Done Done Sent false
24092 7119 Acked Read 3 Done Done Sent false
24093 7120 Acked Write 5 Done Done Sent false
24094 7121 Acked Write 5 Done Done Sent false
24095 7122 Acked Write 3 Done Done Sent false
24096 7123 Acked Write 6 Done Done Sent false
24097 7124 Acked Flush 0 Done Done Sent false
24098 7125 Acked Flush 0 Done Done Sent false
24099 7126 Acked Read 6 Done Done Sent false
24100 7127 Acked Read 8 Done Done Sent false
24101 7128 Acked Write 8 Done Done Sent false
24102 7129 Acked Read 9 Done Done Sent false
24103 7130 Acked Read 9 Done Done Sent false
24104 7131 Acked Write 9 Done Done Sent false
24105 7132 Acked Read 10 Done Done Sent false
24106 7133 Acked Read 2 Done Done Sent false
24107 7134 Acked Read 1 Done Done Sent false
24108 7135 Acked Flush 0 Done Done Sent false
24109 7136 Acked Read 7 Done Done Sent false
24110 7137 Acked Read 7 Done Done Sent false
24111 7138 Acked Write 6 Done Done Sent false
24112 7139 Acked Write 5 Done Done Sent false
24113 7140 Acked Read 9 Done Done Sent false
24114 7141 Acked Write 8 Done Done Sent false
24115 7142 Acked Flush 0 Done Done Sent false
24116 7143 Acked Flush 0 Done Done Sent false
24117 7144 Acked Write 9 Done Done Sent false
24118 7145 Acked Read 8 Done Done Sent false
24119 7146 Acked Read 7 Done Done Sent false
24120 7147 Acked Read 6 Done Done Sent false
24121 7148 Acked Write 5 Done Done Sent false
24122 7149 Acked Write 3 Done Done Sent false
24123 7150 Acked Read 4 Done Done Sent false
24124 7151 Acked Read 4 Done Done Sent false
24125 7152 Acked Read 1 Done Done Sent false
24126 7153 Acked Flush 0 Done Done Sent false
24127 7154 Acked Read 3 Done Done Sent false
24128 7155 Acked Read 7 Done Done Sent false
24129 7156 Acked Read 1 Done Done Sent false
24130 7157 Acked Write 3 Done Done Sent false
24131 7158 Acked Write 7 Done Done Sent false
24132 7159 Acked Read 8 Done Done Sent false
24133 7160 Acked Write 7 Done Done Sent false
24134 7161 Acked Write 9 Done Done Sent false
24135 7162 Acked Write 4 Done Done Sent false
24136 7163 Acked Write 4 Done Done Sent false
24137 7164 Acked Read 5 Done Done Sent false
24138 7165 Acked Read 10 Done Done Sent false
24139 7166 Acked Write 2 Done Done Sent false
24140 7167 Acked Read 2 Done Done Sent false
24141 7168 Acked Write 10 Done Done Sent false
24142 7169 Acked Flush 0 Done Done Sent false
24143 7170 Acked Read 3 Done Done Sent false
24144 7171 Acked Write 5 Done Done Sent false
24145 7172 Acked Read 3 Done Done Sent false
24146 7173 Acked Write 6 Done Done Sent false
24147 7174 Acked Read 7 Done Done Sent false
24148 7175 Acked Read 3 Done Done Sent false
24149 7176 Acked Write 4 Done Done Sent false
24150 7177 Acked Flush 0 Done Done Sent false
24151 7178 Acked Write 5 Done Done Sent false
24152 7179 Acked Read 1 Done Done Sent false
24153 7180 Acked Read 7 Done Done Sent false
24154 7181 Acked Read 5 Done Done Sent false
24155 7182 Acked Write 2 Done Done Sent false
24156 7183 Acked Read 2 Done Done Sent false
24157 7184 Acked Write 3 Done Done Sent false
24158 7185 Acked Write 9 Done Done Sent false
24159 7186 Acked Write 1 Done Done Sent false
24160 7187 Acked Write 3 Done Done Sent false
24161 7188 Acked Write 4 Done Done Sent false
24162 7189 Acked Write 7 Done Done Sent false
24163 7190 Acked Read 4 Done Done Sent false
24164 7191 Acked Write 4 Done Done Sent false
24165 7192 Acked Write 4 Done Done Sent false
24166 7193 Acked Read 4 Done Done Sent false
24167 7194 Acked Write 2 Done Done Sent false
24168 7195 Acked Read 4 Done Done Sent false
24169 7196 Acked Read 9 Done Done Sent false
24170 7197 Acked Write 4 Done Done Sent false
24171 7198 Acked Write 3 Done Done Sent false
24172 7199 Acked Write 1 Done Done Sent false
24173 7200 Acked Read 3 Done Done Sent false
24174 7201 Acked Write 2 Done Done Sent false
24175 7202 Acked Write 9 Done Done Sent false
24176 7203 Acked Flush 0 Done Done Sent false
24177 7204 Acked Write 2 Done Done Sent false
24178 7205 Acked Read 4 Done Done Sent false
24179 7206 Acked Read 7 Done Done Sent false
24180 7207 Acked Read 1 Done Done Sent false
24181 7208 Acked Read 9 Done Done Sent false
24182 7209 Acked Read 9 Done Done Sent false
24183 7210 Acked Write 10 Done Done Sent false
24184 7211 Acked Read 6 Done Done Sent false
24185 7212 Acked Read 3 Done Done Sent false
24186 7213 Acked Read 9 Done Done Sent false
24187 7214 Acked Flush 0 Done Done Sent false
24188 7215 Acked Write 9 Done Done Sent false
24189 7216 Acked Write 7 Done Done Sent false
24190 7217 Acked Read 7 Done Done Sent false
24191 7218 Acked Write 1 Done Done Sent false
24192 7219 Acked Read 1 Done Done Sent false
24193 7220 Acked Read 3 Done Done Sent false
24194 7221 Acked Write 1 Done Done Sent false
24195 7222 Acked Read 4 Done Done Sent false
24196 7223 Acked Write 2 Done Done Sent false
24197 7224 Acked Write 1 Done Done Sent false
24198 7225 Acked Read 7 Done Done Sent false
24199 7226 Acked Read 4 Done Done Sent false
24200 7227 Acked Write 4 Done Done Sent false
24201 7228 Acked Read 6 Done Done Sent false
24202 7229 Acked Flush 0 Done Done Sent false
24203 7230 Acked Read 10 Done Done Sent false
24204 7231 Acked Write 9 Done Done Sent false
24205 7232 Acked Write 8 Done Done Sent false
24206 7233 Acked Read 4 Done Done Sent false
24207 7234 Acked Write 10 Done Done Sent false
24208 7235 Acked Write 5 Done Done Sent false
24209 7236 Acked Flush 0 Done Done Sent false
24210 7237 Acked Write 9 Done Done Sent false
24211 7238 Acked Flush 0 Done Done Sent false
24212 7239 Acked Flush 0 Done Done Sent false
24213 7240 Acked Write 10 Done Done Sent false
24214 7241 Acked Read 9 Done Done Sent false
24215 7242 Acked Write 8 Done Done Sent false
24216 7243 Acked Read 9 Done Done Sent false
24217 7244 Acked Write 2 Done Done Sent false
24218 7245 Acked Read 9 Done Done Sent false
24219 7246 Acked Read 6 Done Done Sent false
24220 7247 Acked Write 2 Done Done Sent false
24221 7248 Acked Read 7 Done Done Sent false
24222 7249 Acked Write 1 Done Done Sent false
24223 7250 Acked Read 5 Done Done Sent false
24224 7251 Acked Read 1 Done Done Sent false
24225 7252 Acked Read 8 Done Done Sent false
24226 7253 Acked Read 3 Done Done Sent false
24227 7254 Acked Write 10 Done Done Sent false
24228 7255 Acked Read 3 Done Done Sent false
24229 7256 Acked Write 1 Done Done Sent false
24230 7257 Acked Write 5 Done Done Sent false
24231 7258 Acked Read 4 Done Done Sent false
24232 7259 Acked Read 3 Done Done Sent false
24233 7260 Acked Write 2 Done Done Sent false
24234 7261 Acked Read 9 Done Done Sent false
24235 7262 Acked Read 2 Done Done Sent false
24236 7263 Acked Flush 0 Done Done Sent false
24237 7264 Acked Write 8 Done Done Sent false
24238 7265 Acked Flush 0 Done Done Sent false
24239 7266 Acked Read 2 Done Done Sent false
24240 7267 Acked Write 2 Done Done Sent false
24241 7268 Acked Read 8 Done Done Sent false
24242 7269 Acked Flush 0 Done Done Sent false
24243 7270 Acked Read 5 Done Done Sent false
24244 7271 Acked Flush 0 Done Done Sent false
24245 7272 Acked Read 6 Done Done Sent false
24246 7273 Acked Read 3 Done Done Sent false
24247 7274 Acked Write 3 Done Done Sent false
24248 7275 Acked Read 7 Done Done Sent false
24249 7276 Acked Write 8 Done Done Sent false
24250 7277 Acked Flush 0 Done Done Sent false
24251 7278 Acked Write 7 Done Done Sent false
24252 7279 Acked Read 3 Done Done Sent false
24253 7280 Acked Read 8 Done Done Sent false
24254 7281 Acked Write 9 Done Done Sent false
24255 7282 Acked Read 3 Done Done Sent false
24256 7283 Acked Flush 0 Done Done Sent false
24257 7284 Acked Write 4 Done Done Sent false
24258 7285 Acked Flush 0 Done Done Sent false
24259 7286 Acked Read 4 Done Done Sent false
24260 7287 Acked Read 9 Done Done Sent false
24261 7288 Acked Read 5 Done Done Sent false
24262 7289 Acked Read 6 Done Done Sent false
24263 7290 Acked Write 10 Done Done Sent false
24264 7291 Acked Write 4 Done Done Sent false
24265 7292 Acked Read 9 Done Done Sent false
24266 7293 Acked Read 3 Done Done Sent false
24267 7294 Acked Write 5 Done Done Sent false
24268 7295 Acked Read 5 Done Done Sent false
24269 7296 Acked Read 5 Done Done Sent false
24270 7297 Acked Write 8 Done Done Sent false
24271 7298 Acked Write 9 Done Done Sent false
24272 7299 Acked Write 6 Done Done Sent false
24273 7300 Acked Read 10 Done Done Sent false
24274 7301 Acked Write 7 Done Done Sent false
24275 7302 Acked Read 9 Done Done Sent false
24276 7303 Acked Write 4 Done Done Sent false
24277 7304 Acked Flush 0 Done Done Sent false
24278 7305 Acked Write 8 Done Done Sent false
24279 7306 Acked Write 7 Done Done Sent false
24280 7307 Acked Read 7 Done Done Sent false
24281 7308 Acked Write 1 Done Done Sent false
24282 7309 Acked Read 8 Done Done Sent false
24283 7310 Acked Write 9 Done Done Sent false
24284 7311 Acked Flush 0 Done Done Sent false
24285 7312 Acked Write 6 Done Done Sent false
24286 7313 Acked Write 9 Done Done Sent false
24287 7314 Acked Read 2 Done Done Sent false
24288 7315 Acked Read 10 Done Done Sent false
24289 7316 Acked Read 1 Done Done Sent false
24290 7317 Acked Read 7 Done Done Sent false
24291 7318 Acked Flush 0 Done Done Sent false
24292 7319 Acked Read 3 Done Done Sent false
24293 7320 Acked Write 2 Done Done Sent false
24294 7321 Acked Write 5 Done Done Sent false
24295 7322 Acked Read 5 Done Done Sent false
24296 7323 Acked Read 2 Done Done Sent false
24297 7324 Acked Read 2 Done Done Sent false
24298 7325 Acked Read 4 Done Done Sent false
24299 7326 Acked Flush 0 Done Done Sent false
24300 7327 Acked Write 5 Done Done Sent false
24301 7328 Acked Write 1 Done Done Sent false
24302 7329 Acked Read 6 Done Done Sent false
24303 7330 Acked Write 2 Done Done Sent false
24304 7331 Acked Read 9 Done Done Sent false
24305 7332 Acked Read 8 Done Done Sent false
24306 7333 Acked Write 4 Done Done Sent false
24307 7334 Acked Write 2 Done Done Sent false
24308 7335 Acked Read 8 Done Done Sent false
24309 7336 Acked Write 5 Done Done Sent false
24310 7337 Acked Write 3 Done Done Sent false
24311 7338 Acked Read 1 Done Done Sent false
24312 7339 Acked Read 10 Done Done Sent false
24313 STATES DS:0 DS:1 DS:2 TOTAL
24314 Sent 0 0 301 301
24315 Done 301 301 0 602
24316 Skipped 0 0 0 0
24317 Error 0 0 0 0
24318 Last Flush: Some(JobId(7326)) Some(JobId(7326)) Some(JobId(7038))
24319 Downstairs last five completed: 7038 7037 7036 7035 7034
24320 Upstairs last five completed: 7339 7338 7337 7336 7335
24321 CLIENT: Up:0 ds:301 act:2
243222025-03-17T22:39:54.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
243232025-03-17T22:39:55.816ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24324 [8830] got start action
24325 [8830] I am going to restart
24326 Starting downstairs at port 8830
24327 Make output file at "/tmp/dsc/downstairs-8830.txt"
24328 [8830][2] reports Starting
24329 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
24330 [8830][2] reports Running
243312025-03-17T22:39:57.187ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24332 ----------------------------------------------------------------
24333 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24334 States: Active Active Offline
24335 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24336 7039 Acked Read 10 Done Done Sent false
24337 7040 Acked Write 5 Done Done Sent false
24338 7041 Acked Write 7 Done Done Sent false
24339 7042 Acked Read 6 Done Done Sent false
24340 7043 Acked Read 8 Done Done Sent false
24341 7044 Acked Write 7 Done Done Sent false
24342 7045 Acked Read 9 Done Done Sent false
24343 7046 Acked Read 9 Done Done Sent false
24344 7047 Acked Write 7 Done Done Sent false
24345 7048 Acked Flush 0 Done Done Sent false
24346 7049 Acked Read 9 Done Done Sent false
24347 7050 Acked Read 6 Done Done Sent false
24348 7051 Acked Read 9 Done Done Sent false
24349 7052 Acked Write 9 Done Done Sent false
24350 7053 Acked Write 3 Done Done Sent false
24351 7054 Acked Read 7 Done Done Sent false
24352 7055 Acked Read 7 Done Done Sent false
24353 7056 Acked Read 5 Done Done Sent false
24354 7057 Acked Read 6 Done Done Sent false
24355 7058 Acked Write 2 Done Done Sent false
24356 7059 Acked Read 10 Done Done Sent false
24357 7060 Acked Read 7 Done Done Sent false
24358 7061 Acked Write 2 Done Done Sent false
24359 7062 Acked Write 5 Done Done Sent false
24360 7063 Acked Write 4 Done Done Sent false
24361 7064 Acked Write 7 Done Done Sent false
24362 7065 Acked Read 8 Done Done Sent false
24363 7066 Acked Read 1 Done Done Sent false
24364 7067 Acked Write 1 Done Done Sent false
24365 7068 Acked Read 5 Done Done Sent false
24366 7069 Acked Write 7 Done Done Sent false
24367 7070 Acked Read 3 Done Done Sent false
24368 7071 Acked Write 2 Done Done Sent false
24369 7072 Acked Write 5 Done Done Sent false
24370 7073 Acked Read 7 Done Done Sent false
24371 7074 Acked Write 1 Done Done Sent false
24372 7075 Acked Write 8 Done Done Sent false
24373 7076 Acked Read 1 Done Done Sent false
24374 7077 Acked Read 1 Done Done Sent false
24375 7078 Acked Read 9 Done Done Sent false
24376 7079 Acked Flush 0 Done Done Sent false
24377 7080 Acked Read 7 Done Done Sent false
24378 7081 Acked Read 3 Done Done Sent false
24379 7082 Acked Read 2 Done Done Sent false
24380 7083 Acked Write 5 Done Done Sent false
24381 7084 Acked Read 3 Done Done Sent false
24382 7085 Acked Read 10 Done Done Sent false
24383 7086 Acked Read 7 Done Done Sent false
24384 7087 Acked Flush 0 Done Done Sent false
24385 7088 Acked Write 3 Done Done Sent false
24386 7089 Acked Write 8 Done Done Sent false
24387 7090 Acked Write 2 Done Done Sent false
24388 7091 Acked Flush 0 Done Done Sent false
24389 7092 Acked Read 5 Done Done Sent false
24390 7093 Acked Read 7 Done Done Sent false
24391 7094 Acked Write 6 Done Done Sent false
24392 7095 Acked Flush 0 Done Done Sent false
24393 7096 Acked Write 5 Done Done Sent false
24394 7097 Acked Read 10 Done Done Sent false
24395 7098 Acked Read 5 Done Done Sent false
24396 7099 Acked Write 4 Done Done Sent false
24397 7100 Acked Read 5 Done Done Sent false
24398 7101 Acked Read 5 Done Done Sent false
24399 7102 Acked Read 2 Done Done Sent false
24400 7103 Acked Read 7 Done Done Sent false
24401 7104 Acked Write 7 Done Done Sent false
24402 7105 Acked Flush 0 Done Done Sent false
24403 7106 Acked Flush 0 Done Done Sent false
24404 7107 Acked Read 5 Done Done Sent false
24405 7108 Acked Write 1 Done Done Sent false
24406 7109 Acked Read 4 Done Done Sent false
24407 7110 Acked Flush 0 Done Done Sent false
24408 7111 Acked Write 2 Done Done Sent false
24409 7112 Acked Read 4 Done Done Sent false
24410 7113 Acked Read 10 Done Done Sent false
24411 7114 Acked Write 1 Done Done Sent false
24412 7115 Acked Read 2 Done Done Sent false
24413 7116 Acked Read 3 Done Done Sent false
24414 7117 Acked Write 2 Done Done Sent false
24415 7118 Acked Flush 0 Done Done Sent false
24416 7119 Acked Read 3 Done Done Sent false
24417 7120 Acked Write 5 Done Done Sent false
24418 7121 Acked Write 5 Done Done Sent false
24419 7122 Acked Write 3 Done Done Sent false
24420 7123 Acked Write 6 Done Done Sent false
24421 7124 Acked Flush 0 Done Done Sent false
24422 7125 Acked Flush 0 Done Done Sent false
24423 7126 Acked Read 6 Done Done Sent false
24424 7127 Acked Read 8 Done Done Sent false
24425 7128 Acked Write 8 Done Done Sent false
24426 7129 Acked Read 9 Done Done Sent false
24427 7130 Acked Read 9 Done Done Sent false
24428 7131 Acked Write 9 Done Done Sent false
24429 7132 Acked Read 10 Done Done Sent false
24430 7133 Acked Read 2 Done Done Sent false
24431 7134 Acked Read 1 Done Done Sent false
24432 7135 Acked Flush 0 Done Done Sent false
24433 7136 Acked Read 7 Done Done Sent false
24434 7137 Acked Read 7 Done Done Sent false
24435 7138 Acked Write 6 Done Done Sent false
24436 7139 Acked Write 5 Done Done Sent false
24437 7140 Acked Read 9 Done Done Sent false
24438 7141 Acked Write 8 Done Done Sent false
24439 7142 Acked Flush 0 Done Done Sent false
24440 7143 Acked Flush 0 Done Done Sent false
24441 7144 Acked Write 9 Done Done Sent false
24442 7145 Acked Read 8 Done Done Sent false
24443 7146 Acked Read 7 Done Done Sent false
24444 7147 Acked Read 6 Done Done Sent false
24445 7148 Acked Write 5 Done Done Sent false
24446 7149 Acked Write 3 Done Done Sent false
24447 7150 Acked Read 4 Done Done Sent false
24448 7151 Acked Read 4 Done Done Sent false
24449 7152 Acked Read 1 Done Done Sent false
24450 7153 Acked Flush 0 Done Done Sent false
24451 7154 Acked Read 3 Done Done Sent false
24452 7155 Acked Read 7 Done Done Sent false
24453 7156 Acked Read 1 Done Done Sent false
24454 7157 Acked Write 3 Done Done Sent false
24455 7158 Acked Write 7 Done Done Sent false
24456 7159 Acked Read 8 Done Done Sent false
24457 7160 Acked Write 7 Done Done Sent false
24458 7161 Acked Write 9 Done Done Sent false
24459 7162 Acked Write 4 Done Done Sent false
24460 7163 Acked Write 4 Done Done Sent false
24461 7164 Acked Read 5 Done Done Sent false
24462 7165 Acked Read 10 Done Done Sent false
24463 7166 Acked Write 2 Done Done Sent false
24464 7167 Acked Read 2 Done Done Sent false
24465 7168 Acked Write 10 Done Done Sent false
24466 7169 Acked Flush 0 Done Done Sent false
24467 7170 Acked Read 3 Done Done Sent false
24468 7171 Acked Write 5 Done Done Sent false
24469 7172 Acked Read 3 Done Done Sent false
24470 7173 Acked Write 6 Done Done Sent false
24471 7174 Acked Read 7 Done Done Sent false
24472 7175 Acked Read 3 Done Done Sent false
24473 7176 Acked Write 4 Done Done Sent false
24474 7177 Acked Flush 0 Done Done Sent false
24475 7178 Acked Write 5 Done Done Sent false
24476 7179 Acked Read 1 Done Done Sent false
24477 7180 Acked Read 7 Done Done Sent false
24478 7181 Acked Read 5 Done Done Sent false
24479 7182 Acked Write 2 Done Done Sent false
24480 7183 Acked Read 2 Done Done Sent false
24481 7184 Acked Write 3 Done Done Sent false
24482 7185 Acked Write 9 Done Done Sent false
24483 7186 Acked Write 1 Done Done Sent false
24484 7187 Acked Write 3 Done Done Sent false
24485 7188 Acked Write 4 Done Done Sent false
24486 7189 Acked Write 7 Done Done Sent false
24487 7190 Acked Read 4 Done Done Sent false
24488 7191 Acked Write 4 Done Done Sent false
24489 7192 Acked Write 4 Done Done Sent false
24490 7193 Acked Read 4 Done Done Sent false
24491 7194 Acked Write 2 Done Done Sent false
24492 7195 Acked Read 4 Done Done Sent false
24493 7196 Acked Read 9 Done Done Sent false
24494 7197 Acked Write 4 Done Done Sent false
24495 7198 Acked Write 3 Done Done Sent false
24496 7199 Acked Write 1 Done Done Sent false
24497 7200 Acked Read 3 Done Done Sent false
24498 7201 Acked Write 2 Done Done Sent false
24499 7202 Acked Write 9 Done Done Sent false
24500 7203 Acked Flush 0 Done Done Sent false
24501 7204 Acked Write 2 Done Done Sent false
24502 7205 Acked Read 4 Done Done Sent false
24503 7206 Acked Read 7 Done Done Sent false
24504 7207 Acked Read 1 Done Done Sent false
24505 7208 Acked Read 9 Done Done Sent false
24506 7209 Acked Read 9 Done Done Sent false
24507 7210 Acked Write 10 Done Done Sent false
24508 7211 Acked Read 6 Done Done Sent false
24509 7212 Acked Read 3 Done Done Sent false
24510 7213 Acked Read 9 Done Done Sent false
24511 7214 Acked Flush 0 Done Done Sent false
24512 7215 Acked Write 9 Done Done Sent false
24513 7216 Acked Write 7 Done Done Sent false
24514 7217 Acked Read 7 Done Done Sent false
24515 7218 Acked Write 1 Done Done Sent false
24516 7219 Acked Read 1 Done Done Sent false
24517 7220 Acked Read 3 Done Done Sent false
24518 7221 Acked Write 1 Done Done Sent false
24519 7222 Acked Read 4 Done Done Sent false
24520 7223 Acked Write 2 Done Done Sent false
24521 7224 Acked Write 1 Done Done Sent false
24522 7225 Acked Read 7 Done Done Sent false
24523 7226 Acked Read 4 Done Done Sent false
24524 7227 Acked Write 4 Done Done Sent false
24525 7228 Acked Read 6 Done Done Sent false
24526 7229 Acked Flush 0 Done Done Sent false
24527 7230 Acked Read 10 Done Done Sent false
24528 7231 Acked Write 9 Done Done Sent false
24529 7232 Acked Write 8 Done Done Sent false
24530 7233 Acked Read 4 Done Done Sent false
24531 7234 Acked Write 10 Done Done Sent false
24532 7235 Acked Write 5 Done Done Sent false
24533 7236 Acked Flush 0 Done Done Sent false
24534 7237 Acked Write 9 Done Done Sent false
24535 7238 Acked Flush 0 Done Done Sent false
24536 7239 Acked Flush 0 Done Done Sent false
24537 7240 Acked Write 10 Done Done Sent false
24538 7241 Acked Read 9 Done Done Sent false
24539 7242 Acked Write 8 Done Done Sent false
24540 7243 Acked Read 9 Done Done Sent false
24541 7244 Acked Write 2 Done Done Sent false
24542 7245 Acked Read 9 Done Done Sent false
24543 7246 Acked Read 6 Done Done Sent false
24544 7247 Acked Write 2 Done Done Sent false
24545 7248 Acked Read 7 Done Done Sent false
24546 7249 Acked Write 1 Done Done Sent false
24547 7250 Acked Read 5 Done Done Sent false
24548 7251 Acked Read 1 Done Done Sent false
24549 7252 Acked Read 8 Done Done Sent false
24550 7253 Acked Read 3 Done Done Sent false
24551 7254 Acked Write 10 Done Done Sent false
24552 7255 Acked Read 3 Done Done Sent false
24553 7256 Acked Write 1 Done Done Sent false
24554 7257 Acked Write 5 Done Done Sent false
24555 7258 Acked Read 4 Done Done Sent false
24556 7259 Acked Read 3 Done Done Sent false
24557 7260 Acked Write 2 Done Done Sent false
24558 7261 Acked Read 9 Done Done Sent false
24559 7262 Acked Read 2 Done Done Sent false
24560 7263 Acked Flush 0 Done Done Sent false
24561 7264 Acked Write 8 Done Done Sent false
24562 7265 Acked Flush 0 Done Done Sent false
24563 7266 Acked Read 2 Done Done Sent false
24564 7267 Acked Write 2 Done Done Sent false
24565 7268 Acked Read 8 Done Done Sent false
24566 7269 Acked Flush 0 Done Done Sent false
24567 7270 Acked Read 5 Done Done Sent false
24568 7271 Acked Flush 0 Done Done Sent false
24569 7272 Acked Read 6 Done Done Sent false
24570 7273 Acked Read 3 Done Done Sent false
24571 7274 Acked Write 3 Done Done Sent false
24572 7275 Acked Read 7 Done Done Sent false
24573 7276 Acked Write 8 Done Done Sent false
24574 7277 Acked Flush 0 Done Done Sent false
24575 7278 Acked Write 7 Done Done Sent false
24576 7279 Acked Read 3 Done Done Sent false
24577 7280 Acked Read 8 Done Done Sent false
24578 7281 Acked Write 9 Done Done Sent false
24579 7282 Acked Read 3 Done Done Sent false
24580 7283 Acked Flush 0 Done Done Sent false
24581 7284 Acked Write 4 Done Done Sent false
24582 7285 Acked Flush 0 Done Done Sent false
24583 7286 Acked Read 4 Done Done Sent false
24584 7287 Acked Read 9 Done Done Sent false
24585 7288 Acked Read 5 Done Done Sent false
24586 7289 Acked Read 6 Done Done Sent false
24587 7290 Acked Write 10 Done Done Sent false
24588 7291 Acked Write 4 Done Done Sent false
24589 7292 Acked Read 9 Done Done Sent false
24590 7293 Acked Read 3 Done Done Sent false
24591 7294 Acked Write 5 Done Done Sent false
24592 7295 Acked Read 5 Done Done Sent false
24593 7296 Acked Read 5 Done Done Sent false
24594 7297 Acked Write 8 Done Done Sent false
24595 7298 Acked Write 9 Done Done Sent false
24596 7299 Acked Write 6 Done Done Sent false
24597 7300 Acked Read 10 Done Done Sent false
24598 7301 Acked Write 7 Done Done Sent false
24599 7302 Acked Read 9 Done Done Sent false
24600 7303 Acked Write 4 Done Done Sent false
24601 7304 Acked Flush 0 Done Done Sent false
24602 7305 Acked Write 8 Done Done Sent false
24603 7306 Acked Write 7 Done Done Sent false
24604 7307 Acked Read 7 Done Done Sent false
24605 7308 Acked Write 1 Done Done Sent false
24606 7309 Acked Read 8 Done Done Sent false
24607 7310 Acked Write 9 Done Done Sent false
24608 7311 Acked Flush 0 Done Done Sent false
24609 7312 Acked Write 6 Done Done Sent false
24610 7313 Acked Write 9 Done Done Sent false
24611 7314 Acked Read 2 Done Done Sent false
24612 7315 Acked Read 10 Done Done Sent false
24613 7316 Acked Read 1 Done Done Sent false
24614 7317 Acked Read 7 Done Done Sent false
24615 7318 Acked Flush 0 Done Done Sent false
24616 7319 Acked Read 3 Done Done Sent false
24617 7320 Acked Write 2 Done Done Sent false
24618 7321 Acked Write 5 Done Done Sent false
24619 7322 Acked Read 5 Done Done Sent false
24620 7323 Acked Read 2 Done Done Sent false
24621 7324 Acked Read 2 Done Done Sent false
24622 7325 Acked Read 4 Done Done Sent false
24623 7326 Acked Flush 0 Done Done Sent false
24624 7327 Acked Write 5 Done Done Sent false
24625 7328 Acked Write 1 Done Done Sent false
24626 7329 Acked Read 6 Done Done Sent false
24627 7330 Acked Write 2 Done Done Sent false
24628 7331 Acked Read 9 Done Done Sent false
24629 7332 Acked Read 8 Done Done Sent false
24630 7333 Acked Write 4 Done Done Sent false
24631 7334 Acked Write 2 Done Done Sent false
24632 7335 Acked Read 8 Done Done Sent false
24633 7336 Acked Write 5 Done Done Sent false
24634 7337 Acked Write 3 Done Done Sent false
24635 7338 Acked Read 1 Done Done Sent false
24636 7339 Acked Read 10 Done Done Sent false
24637 STATES DS:0 DS:1 DS:2 TOTAL
24638 Sent 0 0 301 301
24639 Done 301 301 0 602
24640 Skipped 0 0 0 0
24641 Error 0 0 0 0
24642 Last Flush: Some(JobId(7326)) Some(JobId(7326)) Some(JobId(7038))
24643 Downstairs last five completed: 7038 7037 7036 7035 7034
24644 Upstairs last five completed: 7339 7338 7337 7336 7335
24645 CLIENT: Up:0 ds:301 act:2
246462025-03-17T22:39:58.820ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246472025-03-17T22:39:59.821ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246482025-03-17T22:39:59.925ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246492025-03-17T22:39:59.926ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57840) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246502025-03-17T22:39:59.927ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246512025-03-17T22:39:59.927ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246522025-03-17T22:39:59.927ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246532025-03-17T22:39:59.927ZINFOcrucible: send last flush ID to this DS: Some(JobId(7038)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246542025-03-17T22:39:59.927ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246552025-03-17T22:39:59.927ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7038)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246562025-03-17T22:39:59.927ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7038)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24657 ----------------------------------------------------------------
24658 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24659 Downstairs last five completed: 7340 7339 7338 7337 7336
24660 Upstairs last five completed: 7340 7339 7338 7337 7336
24661 CLIENT: Up:0 ds:0 act:3
24662 Replay: All jobs finished
24663 Main task has work to do, go find it
24664 got dsc Stop(1)
24665 stop 1
24666 Mar 17 22:40:02.171 INFO request completed[8820] Got stop action so:false kr:false
24667 , latency_us: 317, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 808582e3-1230-4ed8-9154-bdc484d4f753, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
246682025-03-17T22:40:02.172ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246692025-03-17T22:40:02.172ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
246702025-03-17T22:40:02.172ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24671 Mar 17 22:40:02.172 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/1, method: GET, req_id: a2220871-53d4-49cb-a8b7-ca7c6a925994, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
246722025-03-17T22:40:02.172ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
24673 [8820] Exited with: ExitStatus(unix_wait_status(9))
24674 [8820][1] reports Exit
24675 Mar 17 22:40:06.174 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/1, method: GET, req_id: 49d686f3-0a54-4d77-b713-5619f3a5be71, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
24676 001/300 Write block 2885 len 8192 data: 5 4
24677 002/300 Write block 250 len 40960 data: 5 5 4 6 8 7 7 6 6 4
24678 003/300 Write block 4605 len 20480 data: 8 7 6 4 4
24679 004/300 Flush
24680 005/300 Read block 4159 len 32768
24681 006/300 Read block 2445 len 40960
24682 007/300 Read block 1765 len 12288
24683 008/300 Flush
24684 009/300 Read block 3615 len 8192
24685 010/300 Read block 2231 len 32768
24686 011/300 Read block 304 len 16384
24687 012/300 Flush
24688 013/300 Write block 4978 len 40960 data: 2 2 3 4 4 4 5 4 4 3
24689 014/300 Read block 771 len 24576
24690 015/300 Read block 2395 len 32768
24691 016/300 Write block 4798 len 4096 data: 6
24692 017/300 Read block 2299 len 12288
24693 018/300 Read block 3918 len 32768
24694 019/300 Read block 3540 len 4096
24695 020/300 Read block 4987 len 20480
24696 021/300 Flush
24697 022/300 Read block 1455 len 4096
24698 023/300 Read block 1939 len 8192
24699 024/300 Read block 1085 len 4096
24700 025/300 Write block 2143 len 24576 data: 5 6 6 7 7 6
24701 026/300 Read block 4926 len 24576
24702 027/300 Flush
24703 028/300 Flush
24704 029/300 Write block 2148 len 8192 data: 7 6
24705 030/300 Read block 2680 len 28672
24706 031/300 Read block 4139 len 4096
24707 032/300 Write block 761 len 24576 data: 4 4 4 5 5 7
24708 033/300 Write block 2602 len 36864 data: 5 5 3 3 2 2 4 5 4
24709 034/300 Read block 2317 len 32768
24710 035/300 Read block 2080 len 4096
24711 036/300 Read block 1173 len 4096
24712 037/300 Write block 3863 len 40960 data: 3 3 3 3 3 3 3 4 5 5
24713 038/300 Flush
24714 039/300 Write block 3559 len 32768 data: 5 5 5 6 7 4 5 5
24715 040/300 Read block 3311 len 4096
24716 041/300 Write block 4901 len 4096 data: 3
24717 042/300 Write block 3967 len 40960 data: 4 4 4 4 4 3 2 2 3 4
24718 043/300 Write block 4646 len 12288 data: 5 5 4
24719 044/300 Write block 2089 len 28672 data: 6 6 6 5 4 3 4
24720 045/300 Write block 385 len 12288 data: 8 7 9
24721 046/300 Read block 783 len 24576
24722 047/300 Read block 4198 len 28672
24723 048/300 Write block 4602 len 16384 data: 6 6 7 9
24724 049/300 Flush
24725 050/300 Flush
24726 051/300 Write block 2698 len 8192 data: 4 4
24727 052/300 Read block 3868 len 12288
24728 053/300 Write block 2710 len 4096 data: 5
24729 054/300 Write block 3660 len 40960 data: 3 5 6 4 3 4 4 3 4 5
24730 055/300 Flush
24731 056/300 Write block 4899 len 16384 data: 2 2 4 3
24732 057/300 Read block 4991 len 8192
24733 058/300 Read block 995 len 20480
24734 059/300 Write block 472 len 12288 data: 2 2 2
24735 060/300 Write block 287 len 8192 data: 5 5
24736 061/300 Flush
24737 062/300 Flush
24738 063/300 Write block 709 len 8192 data: 6 5
24739 064/300 Write block 4492 len 4096 data: 3
24740 065/300 Write block 2945 len 12288 data: 6 6 6
24741 066/300 Read block 1208 len 28672
24742 067/300 Read block 979 len 16384
24743 068/300 Read block 4850 len 4096
24744 069/300 Read block 490 len 32768
24745 070/300 Write block 997 len 24576 data: 4 4 5 5 4 5
24746 071/300 Read block 641 len 20480
24747 072/300 Write block 3850 len 28672 data: 3 2 2 2 3 3 4
24748 073/300 Read block 1779 len 24576
24749 074/300 Write block 3516 len 20480 data: 4 3 2 3 5
24750 075/300 Read block 4489 len 24576
24751 076/300 Write block 944 len 20480 data: 8 7 7 7 6
24752 077/300 Read block 225 len 20480
24753 078/300 Read block 4150 len 12288
24754 079/300 Flush
24755 080/300 Flush
24756 081/300 Write block 3333 len 40960 data: 3 4 5 5 5 6 6 6 6 6
24757 082/300 Read block 3574 len 40960
24758 083/300 Write block 165 len 20480 data: 5 5 5 4 3
24759 084/300 Read block 2167 len 20480
24760 085/300 Read block 701 len 32768
24761 086/300 Read block 575 len 8192
24762 087/300 Flush
24763 088/300 Flush
24764 089/300 Read block 129 len 24576
24765 090/300 Flush
24766 091/300 Read block 1904 len 20480
24767 092/300 Flush
24768 093/300 Read block 2782 len 12288
24769 094/300 Read block 1180 len 32768
24770 095/300 Write block 3684 len 12288 data: 4 3 3
24771 096/300 Read block 4610 len 36864
24772 097/300 Write block 3449 len 20480 data: 3 4 4 3 4
24773 098/300 Read block 1251 len 8192
24774 099/300 Read block 3752 len 28672
24775 100/300 Write block 949 len 8192 data: 5 4
24776 101/300 Write block 3999 len 8192 data: 5 5
24777 102/300 Read block 3858 len 36864
24778 103/300 Write block 2767 len 20480 data: 5 6 6 6 5
24779 104/300 Flush
24780 105/300 Read block 2989 len 16384
24781 106/300 Flush
24782 107/300 Write block 2692 len 20480 data: 2 2 2 3 4
24783 108/300 Write block 297 len 16384 data: 8 5 5 5
24784 109/300 Write block 3603 len 12288 data: 3 3 3
24785 110/300 Read block 4099 len 36864
24786 111/300 Write block 4530 len 8192 data: 6 6
24787 112/300 Flush
24788 113/300 Write block 3490 len 8192 data: 2 2
24789 114/300 Write block 3397 len 36864 data: 8 8 8 9 7 7 6 5 6
24790 115/300 Write block 587 len 32768 data: 3 4 3 2 2 2 2 2
24791 116/300 Read block 1930 len 24576
24792 117/300 Write block 2109 len 32768 data: 3 3 2 3 3 4 5 5
24793 118/300 Write block 4964 len 16384 data: 4 5 6 7
24794 119/300 Flush
24795 120/300 Read block 4632 len 36864
24796 121/300 Read block 3592 len 24576
24797 122/300 Read block 1922 len 12288
24798 123/300 Read block 3239 len 36864
24799 124/300 Read block 2738 len 16384
24800 125/300 Read block 2077 len 20480
24801 126/300 Write block 2099 len 24576 data: 5 5 5 5 4 4
24802 127/300 Read block 4699 len 12288
24803 128/300 Write block 1327 len 12288 data: 7 6 4
24804 129/300 Flush
24805 130/300 Read block 3552 len 4096
24806 131/300 Read block 1794 len 24576
24807 132/300 Write block 855 len 24576 data: 8 7 5 5 3 2
24808 133/300 Read block 2920 len 28672
24809 134/300 Write block 4286 len 36864 data: 5 5 4 4 4 6 7 8 8
24810 135/300 Read block 2876 len 20480
24811 136/300 Write block 123 len 16384 data: 7 7 8 8
24812 137/300 Read block 927 len 16384
24813 138/300 Write block 1353 len 8192 data: 4 4
24814 139/300 Read block 1246 len 32768
24815 140/300 Read block 1787 len 12288
24816 141/300 Read block 2957 len 40960
24817 142/300 Flush
24818 143/300 Write block 3645 len 36864 data: 2 2 2 2 3 3 3 5 6
24819 144/300 Read block 4932 len 16384
24820 145/300 Read block 951 len 32768
24821 146/300 Read block 1924 len 8192
24822 147/300 Write block 1110 len 4096 data: 3
24823 148/300 Read block 2958 len 4096
24824 149/300 Read block 3011 len 20480
24825 150/300 Write block 3320 len 40960 data: 8 7 8 8 7 7 5 5 3 4
24826 151/300 Read block 3824 len 16384
24827 152/300 Read block 2902 len 28672
24828 153/300 Write block 2694 len 28672 data: 3 4 5 4 5 5 5
24829 154/300 Read block 2201 len 32768
24830 155/300 Flush
24831 156/300 Write block 4802 len 40960 data: 6 7 7 6 6 6 5 6 5 5
24832 157/300 Write block 4908 len 8192 data: 4 4
24833 158/300 Read block 2498 len 36864
24834 159/300 Read block 3865 len 24576
24835 160/300 Read block 414 len 24576
24836 161/300 Read block 4050 len 24576
24837 162/300 Read block 2006 len 32768
24838 163/300 Write block 1459 len 20480 data: 4 4 4 4 4
24839 164/300 Write block 3592 len 16384 data: 4 6 7 7
24840 165/300 Read block 793 len 32768
24841 166/300 Write block 4350 len 4096 data: 3
24842 167/300 Write block 2879 len 8192 data: 6 4
24843 168/300 Flush
24844 169/300 Read block 1074 len 16384
24845 170/300 Read block 4600 len 24576
24846 171/300 Write block 4140 len 16384 data: 5 5 7 7
24847 172/300 Write block 3909 len 12288 data: 6 7 7
24848 173/300 Read block 1613 len 32768
24849 174/300 Read block 10 len 32768
24850 175/300 Write block 1700 len 40960 data: 3 3 4 4 2 2 3 3 3 3
24851 176/300 Write block 1892 len 12288 data: 2 3 4
24852 177/300 Flush
24853 178/300 Read block 2276 len 8192
24854 179/300 Write block 641 len 32768 data: 6 5 5 5 6 5 3 3
24855 180/300 Flush
24856 181/300 Read block 1823 len 8192
24857 182/300 Read block 182 len 28672
24858 183/300 Write block 4487 len 40960 data: 4 5 4 4 3 4 6 6 5 5
24859 184/300 Read block 3307 len 16384
24860 185/300 Read block 1805 len 24576
24861 186/300 Read block 4176 len 12288
24862 187/300 Read block 3181 len 16384
24863 188/300 Write block 1738 len 16384 data: 5 5 5 5
24864 189/300 Read block 4022 len 32768
24865 190/300 Read block 3743 len 40960
24866 191/300 Write block 381 len 20480 data: 4 3 4 7 9
24867 192/300 Write block 2832 len 40960 data: 6 6 7 7 6 6 6 5 4 3
24868 193/300 Write block 257 len 28672 data: 7 7 5 4 4 5 7
24869 194/300 Flush
24870 195/300 Write block 2872 len 8192 data: 5 4
24871 196/300 Read block 4439 len 4096
24872 197/300 Read block 3314 len 8192
24873 198/300 Read block 3542 len 16384
24874 199/300 Read block 2307 len 20480
24875 200/300 Write block 2001 len 12288 data: 6 6 6
24876 201/300 Read block 2334 len 8192
24877 202/300 Read block 1927 len 4096
24878 203/300 Read block 2284 len 28672
24879 204/300 Write block 4799 len 8192 data: 7 6
24880 205/300 Write block 2903 len 20480 data: 4 4 4 4 3
24881 206/300 Flush
24882 207/300 Read block 3014 len 4096
24883 208/300 Write block 1872 len 36864 data: 5 4 4 3 4 5 6 5 6
24884 209/300 Flush
24885 210/300 Read block 3880 len 4096
24886 211/300 Flush
24887 212/300 Read block 117 len 12288
24888 213/300 Read block 384 len 8192
24889 214/300 Read block 2042 len 4096
24890 215/300 Write block 2088 len 36864 data: 6 7 7 7 6 5 4 5 5
24891 216/300 Write block 4755 len 32768 data: 2 2 3 3 4 4 4 4
24892 217/300 Write block 3459 len 12288 data: 3 3 3
24893 218/300 Read block 292 len 16384
24894 219/300 Flush
24895 220/300 Read block 1896 len 20480
24896 221/300 Read block 3597 len 8192
24897 222/300 Write block 4292 len 4096 data: 8
24898 223/300 Read block 4968 len 36864
24899 224/300 Read block 1486 len 8192
24900 225/300 Write block 2272 len 28672 data: 4 4 4 4 4 3 3
24901 226/300 Read block 2603 len 28672
24902 227/300 Write block 2572 len 8192 data: 7 7
24903 228/300 Flush
24904 229/300 Read block 2961 len 36864
24905 230/300 Read block 3334 len 4096
24906 231/300 Read block 4270 len 36864
24907 232/300 Write block 2775 len 36864 data: 3 3 4 4 3 3 3 3 4
24908 233/300 Write block 3363 len 28672 data: 7 3 3 4 5 5 5
24909 234/300 Read block 3759 len 16384
24910 235/300 Read block 2227 len 4096
24911 236/300 Read block 4103 len 40960
24912 237/300 Read block 2219 len 28672
24913 238/300 Read block 586 len 40960
24914 239/300 Write block 4939 len 40960 data: 3 4 3 3 3 3 3 3 3 4
24915 240/300 Read block 4132 len 8192
24916 241/300 Write block 4932 len 12288 data: 4 6 8
24917 242/300 Read block 2305 len 40960
24918 243/300 Write block 2362 len 36864 data: 4 3 2 2 2 2 2 3 3
24919 244/300 Read block 4207 len 4096
24920 245/300 Read block 1940 len 24576
24921 246/300 Write block 3623 len 40960 data: 6 6 6 7 8 8 8 6 6 7
24922 247/300 Write block 2296 len 28672 data: 6 4 4 4 5 3 4
24923 248/300 Flush
24924 249/300 Read block 1665 len 4096
24925 250/300 Write block 3233 len 24576 data: 6 6 6 5 6 6
24926 251/300 Read block 2840 len 24576
24927 252/300 Write block 4563 len 12288 data: 6 6 6
24928 253/300 Read block 1572 len 20480
24929 254/300 Write block 4174 len 40960 data: 5 4 5 5 5 6 6 6 6 6
24930 255/300 Write block 2659 len 36864 data: 5 5 5 6 5 5 5 5 6
24931 256/300 Read block 3055 len 36864
24932 257/300 Write block 1332 len 40960 data: 4 3 3 2 4 4 4 4 5 5
24933 258/300 Read block 1274 len 8192
24934 259/300 Read block 1781 len 16384
24935 260/300 Read block 2993 len 36864
24936 261/300 Flush
24937 262/300 Write block 4592 len 12288 data: 2 2 2
24938 263/300 Read block 3483 len 24576
24939 264/300 Read block 665 len 20480
24940 265/300 Write block 3855 len 16384 data: 4 5 4 3
24941 266/300 Write block 2625 len 28672 data: 3 2 2 3 3 5 5
24942 267/300 Write block 3061 len 40960 data: 6 5 5 3 3 4 4 5 6 6
24943 268/300 Read block 2922 len 4096
24944 269/300 Read block 2778 len 12288
24945 270/300 Write block 754 len 28672 data: 4 3 4 3 3 3 4
24946 271/300 Write block 788 len 28672 data: 4 4 4 4 4 6 4
24947 272/300 Read block 2265 len 36864
24948 273/300 Read block 4329 len 12288
24949 274/300 Read block 4602 len 24576
24950 275/300 Write block 4190 len 24576 data: 4 4 5 5 4 3
24951 276/300 Read block 1480 len 36864
24952 277/300 Read block 3763 len 28672
24953 278/300 Read block 836 len 8192
24954 279/300 Write block 4286 len 12288 data: 6 6 5
24955 280/300 Write block 861 len 16384 data: 2 2 2 3
24956 281/300 Read block 1384 len 16384
24957 282/300 Read block 2666 len 40960
24958 283/300 Write block 4819 len 28672 data: 6 5 4 5 3 3 4
24959 284/300 Write block 122 len 16384 data: 6 8 8 9
24960 285/300 Read block 3051 len 12288
24961 286/300 Write block 2223 len 24576 data: 3 4 3 3 3 4
24962 287/300 Write block 4554 len 12288 data: 5 4 4
24963 288/300 Write block 4401 len 20480 data: 3 3 3 3 3
24964 289/300 Read block 928 len 16384
24965 290/300 Read block 1103 len 36864
24966 291/300 Read block 4550 len 32768
24967 292/300 Read block 2725 len 20480
24968 293/300 Write block 3151 len 40960 data: 7 7 6 7 5 5 4 5 4 4
24969 294/300 Read block 4764 len 8192
24970 295/300 Read block 4776 len 28672
24971 296/300 Write block 1295 len 40960 data: 7 6 6 4 3 2 4 5 4 5
24972 297/300 Read block 2194 len 12288
24973 298/300 Read block 4014 len 36864
24974 299/300 Read block 2674 len 12288
24975 300/300 Read block 3454 len 24576
24976 301/300 Flush
24977 Main task has work to do, go find it
24978 got dsc Start(1)
24979 start 1
24980 Mar 17 22:40:06.416 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: 75a29242-71e4-43ca-ba96-4d0f421991a9, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
24981 Replay: started 1, returned:Ok(())
24982 ----------------------------------------------------------------
24983 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24984 States: Active Offline Active
24985 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24986 7341 Acked Write 2 Done Sent Done false
24987 7342 Acked Write 10 Done Sent Done false
24988 7343 Acked Write 5 Done Sent Done false
24989 7344 Acked Flush 0 Done Sent Done false
24990 7345 Acked Read 8 Done Sent Done false
24991 7346 Acked Read 10 Done Sent Done false
24992 7347 Acked Read 3 Done Sent Done false
24993 7348 Acked Flush 0 Done Sent Done false
24994 7349 Acked Read 2 Done Sent Done false
24995 7350 Acked Read 8 Done Sent Done false
24996 7351 Acked Read 4 Done Sent Done false
24997 7352 Acked Flush 0 Done Sent Done false
24998 7353 Acked Write 10 Done Sent Done false
24999 7354 Acked Read 6 Done Sent Done false
25000 7355 Acked Read 8 Done Sent Done false
25001 7356 Acked Write 1 Done Sent Done false
25002 7357 Acked Read 3 Done Sent Done false
25003 7358 Acked Read 8 Done Sent Done false
25004 7359 Acked Read 1 Done Sent Done false
25005 7360 Acked Read 5 Done Sent Done false
25006 7361 Acked Flush 0 Done Sent Done false
25007 7362 Acked Read 1 Done Sent Done false
25008 7363 Acked Read 2 Done Sent Done false
25009 7364 Acked Read 1 Done Sent Done false
25010 7365 Acked Write 6 Done Sent Done false
25011 7366 Acked Read 6 Done Sent Done false
25012 7367 Acked Flush 0 Done Sent Done false
25013 7368 Acked Flush 0 Done Sent Done false
25014 7369 Acked Write 2 Done Sent Done false
25015 7370 Acked Read 7 Done Sent Done false
25016 7371 Acked Read 1 Done Sent Done false
25017 7372 Acked Write 6 Done Sent Done false
25018 7373 Acked Write 9 Done Sent Done false
25019 7374 Acked Read 8 Done Sent Done false
25020 7375 Acked Read 1 Done Sent Done false
25021 7376 Acked Read 1 Done Sent Done false
25022 7377 Acked Write 10 Done Sent Done false
25023 7378 Acked Flush 0 Done Sent Done false
25024 7379 Acked Write 8 Done Sent Done false
25025 7380 Acked Read 1 Done Sent Done false
25026 7381 Acked Write 1 Done Sent Done false
25027 7382 Acked Write 10 Done Sent Done false
25028 7383 Acked Write 3 Done Sent Done false
25029 7384 Acked Write 7 Done Sent Done false
25030 7385 Acked Write 3 Done Sent Done false
25031 7386 Acked Read 6 Done Sent Done false
25032 7387 Acked Read 7 Done Sent Done false
25033 7388 Acked Write 4 Done Sent Done false
25034 7389 Acked Flush 0 Done Sent Done false
25035 7390 Acked Flush 0 Done Sent Done false
25036 7391 Acked Write 2 Done Sent Done false
25037 7392 Acked Read 3 Done Sent Done false
25038 7393 Acked Write 1 Done Sent Done false
25039 7394 Acked Write 10 Done Sent Done false
25040 7395 Acked Flush 0 Done Sent Done false
25041 7396 Acked Write 4 Done Sent Done false
25042 7397 Acked Read 2 Done Sent Done false
25043 7398 Acked Read 5 Done Sent Done false
25044 7399 Acked Write 3 Done Sent Done false
25045 7400 Acked Write 2 Done Sent Done false
25046 7401 Acked Flush 0 Done Sent Done false
25047 7402 Acked Flush 0 Done Sent Done false
25048 7403 Acked Write 2 Done Sent Done false
25049 7404 Acked Write 1 Done Sent Done false
25050 7405 Acked Write 3 Done Sent Done false
25051 7406 Acked Read 7 Done Sent Done false
25052 7407 Acked Read 4 Done Sent Done false
25053 7408 Acked Read 1 Done Sent Done false
25054 7409 Acked Read 8 Done Sent Done false
25055 7410 Acked Write 6 Done Sent Done false
25056 7411 Acked Read 5 Done Sent Done false
25057 7412 Acked Write 7 Done Sent Done false
25058 7413 Acked Read 6 Done Sent Done false
25059 7414 Acked Write 5 Done Sent Done false
25060 7415 Acked Read 6 Done Sent Done false
25061 7416 Acked Write 5 Done Sent Done false
25062 7417 Acked Read 5 Done Sent Done false
25063 7418 Acked Read 3 Done Sent Done false
25064 7419 Acked Flush 0 Done Sent Done false
25065 7420 Acked Flush 0 Done Sent Done false
25066 7421 Acked Write 10 Done Sent Done false
25067 7422 Acked Read 10 Done Sent Done false
25068 7423 Acked Write 5 Done Sent Done false
25069 7424 Acked Read 5 Done Sent Done false
25070 7425 Acked Read 8 Done Sent Done false
25071 7426 Acked Read 2 Done Sent Done false
25072 7427 Acked Flush 0 Done Sent Done false
25073 7428 Acked Flush 0 Done Sent Done false
25074 7429 Acked Read 6 Done Sent Done false
25075 7430 Acked Flush 0 Done Sent Done false
25076 7431 Acked Read 5 Done Sent Done false
25077 7432 Acked Flush 0 Done Sent Done false
25078 7433 Acked Read 3 Done Sent Done false
25079 7434 Acked Read 8 Done Sent Done false
25080 7435 Acked Write 3 Done Sent Done false
25081 7436 Acked Read 9 Done Sent Done false
25082 7437 Acked Write 5 Done Sent Done false
25083 7438 Acked Read 2 Done Sent Done false
25084 7439 Acked Read 7 Done Sent Done false
25085 7440 Acked Write 2 Done Sent Done false
25086 7441 Acked Write 2 Done Sent Done false
25087 7442 Acked Read 9 Done Sent Done false
25088 7443 Acked Write 5 Done Sent Done false
25089 7444 Acked Flush 0 Done Sent Done false
25090 7445 Acked Read 4 Done Sent Done false
25091 7446 Acked Flush 0 Done Sent Done false
25092 7447 Acked Write 5 Done Sent Done false
25093 7448 Acked Write 4 Done Sent Done false
25094 7449 Acked Write 3 Done Sent Done false
25095 7450 Acked Read 9 Done Sent Done false
25096 7451 Acked Write 2 Done Sent Done false
25097 7452 Acked Flush 0 Done Sent Done false
25098 7453 Acked Write 2 Done Sent Done false
25099 7454 Acked Write 9 Done Sent Done false
25100 7455 Acked Write 8 Done Sent Done false
25101 7456 Acked Read 6 Done Sent Done false
25102 7457 Acked Write 8 Done Sent Done false
25103 7458 Acked Write 4 Done Sent Done false
25104 7459 Acked Flush 0 Done Sent Done false
25105 7460 Acked Read 9 Done Sent Done false
25106 7461 Acked Read 6 Done Sent Done false
25107 7462 Acked Read 3 Done Sent Done false
25108 7463 Acked Read 9 Done Sent Done false
25109 7464 Acked Read 4 Done Sent Done false
25110 7465 Acked Read 5 Done Sent Done false
25111 7466 Acked Write 6 Done Sent Done false
25112 7467 Acked Read 3 Done Sent Done false
25113 7468 Acked Write 3 Done Sent Done false
25114 7469 Acked Flush 0 Done Sent Done false
25115 7470 Acked Read 1 Done Sent Done false
25116 7471 Acked Read 6 Done Sent Done false
25117 7472 Acked Write 6 Done Sent Done false
25118 7473 Acked Read 7 Done Sent Done false
25119 7474 Acked Write 9 Done Sent Done false
25120 7475 Acked Read 5 Done Sent Done false
25121 7476 Acked Write 4 Done Sent Done false
25122 7477 Acked Read 4 Done Sent Done false
25123 7478 Acked Write 2 Done Sent Done false
25124 7479 Acked Read 8 Done Sent Done false
25125 7480 Acked Read 3 Done Sent Done false
25126 7481 Acked Read 10 Done Sent Done false
25127 7482 Acked Flush 0 Done Sent Done false
25128 7483 Acked Write 9 Done Sent Done false
25129 7484 Acked Read 4 Done Sent Done false
25130 7485 Acked Read 8 Done Sent Done false
25131 7486 Acked Read 2 Done Sent Done false
25132 7487 Acked Write 1 Done Sent Done false
25133 7488 Acked Read 1 Done Sent Done false
25134 7489 Acked Read 5 Done Sent Done false
25135 7490 Acked Write 10 Done Sent Done false
25136 7491 Acked Read 4 Done Sent Done false
25137 7492 Acked Read 7 Done Sent Done false
25138 7493 Acked Write 7 Done Sent Done false
25139 7494 Acked Read 8 Done Sent Done false
25140 7495 Acked Flush 0 Done Sent Done false
25141 7496 Acked Write 10 Done Sent Done false
25142 7497 Acked Write 2 Done Sent Done false
25143 7498 Acked Read 9 Done Sent Done false
25144 7499 Acked Read 6 Done Sent Done false
25145 7500 Acked Read 6 Done Sent Done false
25146 7501 Acked Read 6 Done Sent Done false
25147 7502 Acked Read 8 Done Sent Done false
25148 7503 Acked Write 5 Done Sent Done false
25149 7504 Acked Write 4 Done Sent Done false
25150 7505 Acked Read 8 Done Sent Done false
25151 7506 Acked Write 1 Done Sent Done false
25152 7507 Acked Write 2 Done Sent Done false
25153 7508 Acked Flush 0 Done Sent Done false
25154 7509 Acked Read 4 Done Sent Done false
25155 7510 Acked Read 6 Done Sent Done false
25156 7511 Acked Write 4 Done Sent Done false
25157 7512 Acked Write 3 Done Sent Done false
25158 7513 Acked Read 8 Done Sent Done false
25159 7514 Acked Read 8 Done Sent Done false
25160 7515 Acked Write 10 Done Sent Done false
25161 7516 Acked Write 3 Done Sent Done false
25162 7517 Acked Flush 0 Done Sent Done false
25163 7518 Acked Read 2 Done Sent Done false
25164 7519 Acked Write 8 Done Sent Done false
25165 7520 Acked Flush 0 Done Sent Done false
25166 7521 Acked Read 2 Done Sent Done false
25167 7522 Acked Read 7 Done Sent Done false
25168 7523 Acked Write 10 Done Sent Done false
25169 7524 Acked Read 4 Done Sent Done false
25170 7525 Acked Read 6 Done Sent Done false
25171 7526 Acked Read 3 Done Sent Done false
25172 7527 Acked Read 4 Done Sent Done false
25173 7528 Acked Write 4 Done Sent Done false
25174 7529 Acked Read 8 Done Sent Done false
25175 7530 Acked Read 10 Done Sent Done false
25176 7531 Acked Write 5 Done Sent Done false
25177 7532 Acked Write 10 Done Sent Done false
25178 7533 Acked Write 7 Done Sent Done false
25179 7534 Acked Flush 0 Done Sent Done false
25180 7535 Acked Write 2 Done Sent Done false
25181 7536 Acked Read 1 Done Sent Done false
25182 7537 Acked Read 2 Done Sent Done false
25183 7538 Acked Read 4 Done Sent Done false
25184 7539 Acked Read 5 Done Sent Done false
25185 7540 Acked Write 3 Done Sent Done false
25186 7541 Acked Read 2 Done Sent Done false
25187 7542 Acked Read 1 Done Sent Done false
25188 7543 Acked Read 7 Done Sent Done false
25189 7544 Acked Write 2 Done Sent Done false
25190 7545 Acked Write 5 Done Sent Done false
25191 7546 Acked Flush 0 Done Sent Done false
25192 7547 Acked Read 1 Done Sent Done false
25193 7548 Acked Write 9 Done Sent Done false
25194 7549 Acked Flush 0 Done Sent Done false
25195 7550 Acked Read 1 Done Sent Done false
25196 7551 Acked Flush 0 Done Sent Done false
25197 7552 Acked Read 3 Done Sent Done false
25198 7553 Acked Read 2 Done Sent Done false
25199 7554 Acked Read 1 Done Sent Done false
25200 7555 Acked Write 9 Done Sent Done false
25201 7556 Acked Write 8 Done Sent Done false
25202 7557 Acked Write 3 Done Sent Done false
25203 7558 Acked Read 4 Done Sent Done false
25204 7559 Acked Flush 0 Done Sent Done false
25205 7560 Acked Read 5 Done Sent Done false
25206 7561 Acked Read 2 Done Sent Done false
25207 7562 Acked Write 1 Done Sent Done false
25208 7563 Acked Read 9 Done Sent Done false
25209 7564 Acked Read 2 Done Sent Done false
25210 7565 Acked Write 7 Done Sent Done false
25211 7566 Acked Read 7 Done Sent Done false
25212 7567 Acked Write 2 Done Sent Done false
25213 7568 Acked Flush 0 Done Sent Done false
25214 7569 Acked Read 9 Done Sent Done false
25215 7570 Acked Read 1 Done Sent Done false
25216 7571 Acked Read 9 Done Sent Done false
25217 7572 Acked Write 9 Done Sent Done false
25218 7573 Acked Write 7 Done Sent Done false
25219 7574 Acked Read 4 Done Sent Done false
25220 7575 Acked Read 1 Done Sent Done false
25221 7576 Acked Read 10 Done Sent Done false
25222 7577 Acked Read 7 Done Sent Done false
25223 7578 Acked Read 10 Done Sent Done false
25224 7579 Acked Write 10 Done Sent Done false
25225 7580 Acked Read 2 Done Sent Done false
25226 7581 Acked Write 3 Done Sent Done false
25227 7582 Acked Read 10 Done Sent Done false
25228 7583 Acked Write 9 Done Sent Done false
25229 7584 Acked Read 1 Done Sent Done false
25230 7585 Acked Read 6 Done Sent Done false
25231 7586 Acked Write 10 Done Sent Done false
25232 7587 Acked Write 7 Done Sent Done false
25233 7588 Acked Flush 0 Done Sent Done false
25234 7589 Acked Read 1 Done Sent Done false
25235 7590 Acked Write 6 Done Sent Done false
25236 7591 Acked Read 6 Done Sent Done false
25237 7592 Acked Write 3 Done Sent Done false
25238 7593 Acked Read 5 Done Sent Done false
25239 7594 Acked Write 10 Done Sent Done false
25240 7595 Acked Write 9 Done Sent Done false
25241 7596 Acked Read 9 Done Sent Done false
25242 7597 Acked Write 10 Done Sent Done false
25243 7598 Acked Read 2 Done Sent Done false
25244 7599 Acked Read 4 Done Sent Done false
25245 7600 Acked Read 9 Done Sent Done false
25246 7601 Acked Flush 0 Done Sent Done false
25247 7602 Acked Write 3 Done Sent Done false
25248 7603 Acked Read 6 Done Sent Done false
25249 7604 Acked Read 5 Done Sent Done false
25250 7605 Acked Write 4 Done Sent Done false
25251 7606 Acked Write 7 Done Sent Done false
25252 7607 Acked Write 10 Done Sent Done false
25253 7608 Acked Read 1 Done Sent Done false
25254 7609 Acked Read 3 Done Sent Done false
25255 7610 Acked Write 7 Done Sent Done false
25256 7611 Acked Write 7 Done Sent Done false
25257 7612 Acked Read 9 Done Sent Done false
25258 7613 Acked Read 3 Done Sent Done false
25259 7614 Acked Read 6 Done Sent Done false
25260 7615 Acked Write 6 Done Sent Done false
25261 7616 Acked Read 9 Done Sent Done false
25262 7617 Acked Read 7 Done Sent Done false
25263 7618 Acked Read 2 Done Sent Done false
25264 7619 Acked Write 3 Done Sent Done false
25265 7620 Acked Write 4 Done Sent Done false
25266 7621 Acked Read 4 Done Sent Done false
25267 7622 Acked Read 10 Done Sent Done false
25268 7623 Acked Write 7 Done Sent Done false
25269 7624 Acked Write 4 Done Sent Done false
25270 7625 Acked Read 3 Done Sent Done false
25271 7626 Acked Write 6 Done Sent Done false
25272 7627 Acked Write 3 Done Sent Done false
25273 7628 Acked Write 5 Done Sent Done false
25274 7629 Acked Read 4 Done Sent Done false
25275 7630 Acked Read 9 Done Sent Done false
25276 7631 Acked Read 8 Done Sent Done false
25277 7632 Acked Read 5 Done Sent Done false
25278 7633 Acked Write 10 Done Sent Done false
25279 7634 Acked Read 2 Done Sent Done false
25280 7635 Acked Read 7 Done Sent Done false
25281 7636 Acked Write 10 Done Sent Done false
25282 7637 Acked Read 3 Done Sent Done false
25283 7638 Acked Read 9 Done Sent Done false
25284 7639 Acked Read 3 Done Sent Done false
25285 7640 Acked Read 6 Done Sent Done false
25286 7641 Acked Flush 0 Done Sent Done false
25287 STATES DS:0 DS:1 DS:2 TOTAL
25288 Sent 0 301 0 301
25289 Done 301 0 301 602
25290 Skipped 0 0 0 0
25291 Error 0 0 0 0
25292 Last Flush: Some(JobId(7641)) Some(JobId(7340)) Some(JobId(7641))
25293 Downstairs last five completed: 7340 7339 7338 7337 7336
25294 Upstairs last five completed: 7641 7640 7639 7638 7637
25295 CLIENT: Up:0 ds:301 act:2
252962025-03-17T22:40:07.189ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
252972025-03-17T22:40:07.834ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
25298 [8820] got start action
25299 [8820] I am going to restart
25300 Starting downstairs at port 8820
25301 Make output file at "/tmp/dsc/downstairs-8820.txt"
25302 [8820][1] reports Starting
25303 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1230)
25304 [8820][1] reports Running
253052025-03-17T22:40:08.836ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
253062025-03-17T22:40:09.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
25307 ----------------------------------------------------------------
25308 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25309 States: Active Offline Active
25310 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25311 7341 Acked Write 2 Done Sent Done false
25312 7342 Acked Write 10 Done Sent Done false
25313 7343 Acked Write 5 Done Sent Done false
25314 7344 Acked Flush 0 Done Sent Done false
25315 7345 Acked Read 8 Done Sent Done false
25316 7346 Acked Read 10 Done Sent Done false
25317 7347 Acked Read 3 Done Sent Done false
25318 7348 Acked Flush 0 Done Sent Done false
25319 7349 Acked Read 2 Done Sent Done false
25320 7350 Acked Read 8 Done Sent Done false
25321 7351 Acked Read 4 Done Sent Done false
25322 7352 Acked Flush 0 Done Sent Done false
25323 7353 Acked Write 10 Done Sent Done false
25324 7354 Acked Read 6 Done Sent Done false
25325 7355 Acked Read 8 Done Sent Done false
25326 7356 Acked Write 1 Done Sent Done false
25327 7357 Acked Read 3 Done Sent Done false
25328 7358 Acked Read 8 Done Sent Done false
25329 7359 Acked Read 1 Done Sent Done false
25330 7360 Acked Read 5 Done Sent Done false
25331 7361 Acked Flush 0 Done Sent Done false
25332 7362 Acked Read 1 Done Sent Done false
25333 7363 Acked Read 2 Done Sent Done false
25334 7364 Acked Read 1 Done Sent Done false
25335 7365 Acked Write 6 Done Sent Done false
25336 7366 Acked Read 6 Done Sent Done false
25337 7367 Acked Flush 0 Done Sent Done false
25338 7368 Acked Flush 0 Done Sent Done false
25339 7369 Acked Write 2 Done Sent Done false
25340 7370 Acked Read 7 Done Sent Done false
25341 7371 Acked Read 1 Done Sent Done false
25342 7372 Acked Write 6 Done Sent Done false
25343 7373 Acked Write 9 Done Sent Done false
25344 7374 Acked Read 8 Done Sent Done false
25345 7375 Acked Read 1 Done Sent Done false
25346 7376 Acked Read 1 Done Sent Done false
25347 7377 Acked Write 10 Done Sent Done false
25348 7378 Acked Flush 0 Done Sent Done false
25349 7379 Acked Write 8 Done Sent Done false
25350 7380 Acked Read 1 Done Sent Done false
25351 7381 Acked Write 1 Done Sent Done false
25352 7382 Acked Write 10 Done Sent Done false
25353 7383 Acked Write 3 Done Sent Done false
25354 7384 Acked Write 7 Done Sent Done false
25355 7385 Acked Write 3 Done Sent Done false
25356 7386 Acked Read 6 Done Sent Done false
25357 7387 Acked Read 7 Done Sent Done false
25358 7388 Acked Write 4 Done Sent Done false
25359 7389 Acked Flush 0 Done Sent Done false
25360 7390 Acked Flush 0 Done Sent Done false
25361 7391 Acked Write 2 Done Sent Done false
25362 7392 Acked Read 3 Done Sent Done false
25363 7393 Acked Write 1 Done Sent Done false
25364 7394 Acked Write 10 Done Sent Done false
25365 7395 Acked Flush 0 Done Sent Done false
25366 7396 Acked Write 4 Done Sent Done false
25367 7397 Acked Read 2 Done Sent Done false
25368 7398 Acked Read 5 Done Sent Done false
25369 7399 Acked Write 3 Done Sent Done false
25370 7400 Acked Write 2 Done Sent Done false
25371 7401 Acked Flush 0 Done Sent Done false
25372 7402 Acked Flush 0 Done Sent Done false
25373 7403 Acked Write 2 Done Sent Done false
25374 7404 Acked Write 1 Done Sent Done false
25375 7405 Acked Write 3 Done Sent Done false
25376 7406 Acked Read 7 Done Sent Done false
25377 7407 Acked Read 4 Done Sent Done false
25378 7408 Acked Read 1 Done Sent Done false
25379 7409 Acked Read 8 Done Sent Done false
25380 7410 Acked Write 6 Done Sent Done false
25381 7411 Acked Read 5 Done Sent Done false
25382 7412 Acked Write 7 Done Sent Done false
25383 7413 Acked Read 6 Done Sent Done false
25384 7414 Acked Write 5 Done Sent Done false
25385 7415 Acked Read 6 Done Sent Done false
25386 7416 Acked Write 5 Done Sent Done false
25387 7417 Acked Read 5 Done Sent Done false
25388 7418 Acked Read 3 Done Sent Done false
25389 7419 Acked Flush 0 Done Sent Done false
25390 7420 Acked Flush 0 Done Sent Done false
25391 7421 Acked Write 10 Done Sent Done false
25392 7422 Acked Read 10 Done Sent Done false
25393 7423 Acked Write 5 Done Sent Done false
25394 7424 Acked Read 5 Done Sent Done false
25395 7425 Acked Read 8 Done Sent Done false
25396 7426 Acked Read 2 Done Sent Done false
25397 7427 Acked Flush 0 Done Sent Done false
25398 7428 Acked Flush 0 Done Sent Done false
25399 7429 Acked Read 6 Done Sent Done false
25400 7430 Acked Flush 0 Done Sent Done false
25401 7431 Acked Read 5 Done Sent Done false
25402 7432 Acked Flush 0 Done Sent Done false
25403 7433 Acked Read 3 Done Sent Done false
25404 7434 Acked Read 8 Done Sent Done false
25405 7435 Acked Write 3 Done Sent Done false
25406 7436 Acked Read 9 Done Sent Done false
25407 7437 Acked Write 5 Done Sent Done false
25408 7438 Acked Read 2 Done Sent Done false
25409 7439 Acked Read 7 Done Sent Done false
25410 7440 Acked Write 2 Done Sent Done false
25411 7441 Acked Write 2 Done Sent Done false
25412 7442 Acked Read 9 Done Sent Done false
25413 7443 Acked Write 5 Done Sent Done false
25414 7444 Acked Flush 0 Done Sent Done false
25415 7445 Acked Read 4 Done Sent Done false
25416 7446 Acked Flush 0 Done Sent Done false
25417 7447 Acked Write 5 Done Sent Done false
25418 7448 Acked Write 4 Done Sent Done false
25419 7449 Acked Write 3 Done Sent Done false
25420 7450 Acked Read 9 Done Sent Done false
25421 7451 Acked Write 2 Done Sent Done false
25422 7452 Acked Flush 0 Done Sent Done false
25423 7453 Acked Write 2 Done Sent Done false
25424 7454 Acked Write 9 Done Sent Done false
25425 7455 Acked Write 8 Done Sent Done false
25426 7456 Acked Read 6 Done Sent Done false
25427 7457 Acked Write 8 Done Sent Done false
25428 7458 Acked Write 4 Done Sent Done false
25429 7459 Acked Flush 0 Done Sent Done false
25430 7460 Acked Read 9 Done Sent Done false
25431 7461 Acked Read 6 Done Sent Done false
25432 7462 Acked Read 3 Done Sent Done false
25433 7463 Acked Read 9 Done Sent Done false
25434 7464 Acked Read 4 Done Sent Done false
25435 7465 Acked Read 5 Done Sent Done false
25436 7466 Acked Write 6 Done Sent Done false
25437 7467 Acked Read 3 Done Sent Done false
25438 7468 Acked Write 3 Done Sent Done false
25439 7469 Acked Flush 0 Done Sent Done false
25440 7470 Acked Read 1 Done Sent Done false
25441 7471 Acked Read 6 Done Sent Done false
25442 7472 Acked Write 6 Done Sent Done false
25443 7473 Acked Read 7 Done Sent Done false
25444 7474 Acked Write 9 Done Sent Done false
25445 7475 Acked Read 5 Done Sent Done false
25446 7476 Acked Write 4 Done Sent Done false
25447 7477 Acked Read 4 Done Sent Done false
25448 7478 Acked Write 2 Done Sent Done false
25449 7479 Acked Read 8 Done Sent Done false
25450 7480 Acked Read 3 Done Sent Done false
25451 7481 Acked Read 10 Done Sent Done false
25452 7482 Acked Flush 0 Done Sent Done false
25453 7483 Acked Write 9 Done Sent Done false
25454 7484 Acked Read 4 Done Sent Done false
25455 7485 Acked Read 8 Done Sent Done false
25456 7486 Acked Read 2 Done Sent Done false
25457 7487 Acked Write 1 Done Sent Done false
25458 7488 Acked Read 1 Done Sent Done false
25459 7489 Acked Read 5 Done Sent Done false
25460 7490 Acked Write 10 Done Sent Done false
25461 7491 Acked Read 4 Done Sent Done false
25462 7492 Acked Read 7 Done Sent Done false
25463 7493 Acked Write 7 Done Sent Done false
25464 7494 Acked Read 8 Done Sent Done false
25465 7495 Acked Flush 0 Done Sent Done false
25466 7496 Acked Write 10 Done Sent Done false
25467 7497 Acked Write 2 Done Sent Done false
25468 7498 Acked Read 9 Done Sent Done false
25469 7499 Acked Read 6 Done Sent Done false
25470 7500 Acked Read 6 Done Sent Done false
25471 7501 Acked Read 6 Done Sent Done false
25472 7502 Acked Read 8 Done Sent Done false
25473 7503 Acked Write 5 Done Sent Done false
25474 7504 Acked Write 4 Done Sent Done false
25475 7505 Acked Read 8 Done Sent Done false
25476 7506 Acked Write 1 Done Sent Done false
25477 7507 Acked Write 2 Done Sent Done false
25478 7508 Acked Flush 0 Done Sent Done false
25479 7509 Acked Read 4 Done Sent Done false
25480 7510 Acked Read 6 Done Sent Done false
25481 7511 Acked Write 4 Done Sent Done false
25482 7512 Acked Write 3 Done Sent Done false
25483 7513 Acked Read 8 Done Sent Done false
25484 7514 Acked Read 8 Done Sent Done false
25485 7515 Acked Write 10 Done Sent Done false
25486 7516 Acked Write 3 Done Sent Done false
25487 7517 Acked Flush 0 Done Sent Done false
25488 7518 Acked Read 2 Done Sent Done false
25489 7519 Acked Write 8 Done Sent Done false
25490 7520 Acked Flush 0 Done Sent Done false
25491 7521 Acked Read 2 Done Sent Done false
25492 7522 Acked Read 7 Done Sent Done false
25493 7523 Acked Write 10 Done Sent Done false
25494 7524 Acked Read 4 Done Sent Done false
25495 7525 Acked Read 6 Done Sent Done false
25496 7526 Acked Read 3 Done Sent Done false
25497 7527 Acked Read 4 Done Sent Done false
25498 7528 Acked Write 4 Done Sent Done false
25499 7529 Acked Read 8 Done Sent Done false
25500 7530 Acked Read 10 Done Sent Done false
25501 7531 Acked Write 5 Done Sent Done false
25502 7532 Acked Write 10 Done Sent Done false
25503 7533 Acked Write 7 Done Sent Done false
25504 7534 Acked Flush 0 Done Sent Done false
25505 7535 Acked Write 2 Done Sent Done false
25506 7536 Acked Read 1 Done Sent Done false
25507 7537 Acked Read 2 Done Sent Done false
25508 7538 Acked Read 4 Done Sent Done false
25509 7539 Acked Read 5 Done Sent Done false
25510 7540 Acked Write 3 Done Sent Done false
25511 7541 Acked Read 2 Done Sent Done false
25512 7542 Acked Read 1 Done Sent Done false
25513 7543 Acked Read 7 Done Sent Done false
25514 7544 Acked Write 2 Done Sent Done false
25515 7545 Acked Write 5 Done Sent Done false
25516 7546 Acked Flush 0 Done Sent Done false
25517 7547 Acked Read 1 Done Sent Done false
25518 7548 Acked Write 9 Done Sent Done false
25519 7549 Acked Flush 0 Done Sent Done false
25520 7550 Acked Read 1 Done Sent Done false
25521 7551 Acked Flush 0 Done Sent Done false
25522 7552 Acked Read 3 Done Sent Done false
25523 7553 Acked Read 2 Done Sent Done false
25524 7554 Acked Read 1 Done Sent Done false
25525 7555 Acked Write 9 Done Sent Done false
25526 7556 Acked Write 8 Done Sent Done false
25527 7557 Acked Write 3 Done Sent Done false
25528 7558 Acked Read 4 Done Sent Done false
25529 7559 Acked Flush 0 Done Sent Done false
25530 7560 Acked Read 5 Done Sent Done false
25531 7561 Acked Read 2 Done Sent Done false
25532 7562 Acked Write 1 Done Sent Done false
25533 7563 Acked Read 9 Done Sent Done false
25534 7564 Acked Read 2 Done Sent Done false
25535 7565 Acked Write 7 Done Sent Done false
25536 7566 Acked Read 7 Done Sent Done false
25537 7567 Acked Write 2 Done Sent Done false
25538 7568 Acked Flush 0 Done Sent Done false
25539 7569 Acked Read 9 Done Sent Done false
25540 7570 Acked Read 1 Done Sent Done false
25541 7571 Acked Read 9 Done Sent Done false
25542 7572 Acked Write 9 Done Sent Done false
25543 7573 Acked Write 7 Done Sent Done false
25544 7574 Acked Read 4 Done Sent Done false
25545 7575 Acked Read 1 Done Sent Done false
25546 7576 Acked Read 10 Done Sent Done false
25547 7577 Acked Read 7 Done Sent Done false
25548 7578 Acked Read 10 Done Sent Done false
25549 7579 Acked Write 10 Done Sent Done false
25550 7580 Acked Read 2 Done Sent Done false
25551 7581 Acked Write 3 Done Sent Done false
25552 7582 Acked Read 10 Done Sent Done false
25553 7583 Acked Write 9 Done Sent Done false
25554 7584 Acked Read 1 Done Sent Done false
25555 7585 Acked Read 6 Done Sent Done false
25556 7586 Acked Write 10 Done Sent Done false
25557 7587 Acked Write 7 Done Sent Done false
25558 7588 Acked Flush 0 Done Sent Done false
25559 7589 Acked Read 1 Done Sent Done false
25560 7590 Acked Write 6 Done Sent Done false
25561 7591 Acked Read 6 Done Sent Done false
25562 7592 Acked Write 3 Done Sent Done false
25563 7593 Acked Read 5 Done Sent Done false
25564 7594 Acked Write 10 Done Sent Done false
25565 7595 Acked Write 9 Done Sent Done false
25566 7596 Acked Read 9 Done Sent Done false
25567 7597 Acked Write 10 Done Sent Done false
25568 7598 Acked Read 2 Done Sent Done false
25569 7599 Acked Read 4 Done Sent Done false
25570 7600 Acked Read 9 Done Sent Done false
25571 7601 Acked Flush 0 Done Sent Done false
25572 7602 Acked Write 3 Done Sent Done false
25573 7603 Acked Read 6 Done Sent Done false
25574 7604 Acked Read 5 Done Sent Done false
25575 7605 Acked Write 4 Done Sent Done false
25576 7606 Acked Write 7 Done Sent Done false
25577 7607 Acked Write 10 Done Sent Done false
25578 7608 Acked Read 1 Done Sent Done false
25579 7609 Acked Read 3 Done Sent Done false
25580 7610 Acked Write 7 Done Sent Done false
25581 7611 Acked Write 7 Done Sent Done false
25582 7612 Acked Read 9 Done Sent Done false
25583 7613 Acked Read 3 Done Sent Done false
25584 7614 Acked Read 6 Done Sent Done false
25585 7615 Acked Write 6 Done Sent Done false
25586 7616 Acked Read 9 Done Sent Done false
25587 7617 Acked Read 7 Done Sent Done false
25588 7618 Acked Read 2 Done Sent Done false
25589 7619 Acked Write 3 Done Sent Done false
25590 7620 Acked Write 4 Done Sent Done false
25591 7621 Acked Read 4 Done Sent Done false
25592 7622 Acked Read 10 Done Sent Done false
25593 7623 Acked Write 7 Done Sent Done false
25594 7624 Acked Write 4 Done Sent Done false
25595 7625 Acked Read 3 Done Sent Done false
25596 7626 Acked Write 6 Done Sent Done false
25597 7627 Acked Write 3 Done Sent Done false
25598 7628 Acked Write 5 Done Sent Done false
25599 7629 Acked Read 4 Done Sent Done false
25600 7630 Acked Read 9 Done Sent Done false
25601 7631 Acked Read 8 Done Sent Done false
25602 7632 Acked Read 5 Done Sent Done false
25603 7633 Acked Write 10 Done Sent Done false
25604 7634 Acked Read 2 Done Sent Done false
25605 7635 Acked Read 7 Done Sent Done false
25606 7636 Acked Write 10 Done Sent Done false
25607 7637 Acked Read 3 Done Sent Done false
25608 7638 Acked Read 9 Done Sent Done false
25609 7639 Acked Read 3 Done Sent Done false
25610 7640 Acked Read 6 Done Sent Done false
25611 7641 Acked Flush 0 Done Sent Done false
25612 STATES DS:0 DS:1 DS:2 TOTAL
25613 Sent 0 301 0 301
25614 Done 301 0 301 602
25615 Skipped 0 0 0 0
25616 Error 0 0 0 0
25617 Last Flush: Some(JobId(7641)) Some(JobId(7340)) Some(JobId(7641))
25618 Downstairs last five completed: 7340 7339 7338 7337 7336
25619 Upstairs last five completed: 7641 7640 7639 7638 7637
25620 CLIENT: Up:0 ds:301 act:2
256212025-03-17T22:40:10.920ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256222025-03-17T22:40:12.173ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256232025-03-17T22:40:12.173ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47335) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256242025-03-17T22:40:12.174ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256252025-03-17T22:40:12.175ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256262025-03-17T22:40:12.175ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256272025-03-17T22:40:12.175ZINFOcrucible: send last flush ID to this DS: Some(JobId(7340)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256282025-03-17T22:40:12.175ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256292025-03-17T22:40:12.175ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7340)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256302025-03-17T22:40:12.175ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7340)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
25631 ----------------------------------------------------------------
25632 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25633 Downstairs last five completed: 7641 7640 7639 7638 7637
25634 Upstairs last five completed: 7641 7640 7639 7638 7637
25635 CLIENT: Up:0 ds:0 act:3
25636 Replay: All jobs finished
25637 Main task has work to do, go find it
25638 got dsc Stop(1)
25639 stop 1
25640 Mar 17 22:40:14.421 INFO request completed, latency_us: 344, response_code: 204, uri: /stop/cid/1, method: POST, req_id: aa013aeb-5f23-4787-8a92-cb16707191c2, remote_addr: 127.0.0[8820] Got stop action so:false kr:false
25641 .1:35180, local_addr: 127.0.0.1:9998
25642 Mar 17 22:40:14.422 INFO request completed, latency_us: 111, response_code: 200, uri: /state/cid/1, method: GET, req_id: 339b3ad2-e6e7-4513-8f18-373e2c3a2cf5, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
256432025-03-17T22:40:14.422ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256442025-03-17T22:40:14.422ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256452025-03-17T22:40:14.422ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
256462025-03-17T22:40:14.422ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
25647 [8820] Exited with: ExitStatus(unix_wait_status(9))
25648 [8820][1] reports Exit
25649 Mar 17 22:40:18.423 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/1, method: GET, req_id: 99e1463d-149e-4748-8ca0-b14bedf803ac, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
25650 001/300 Write block 2073 len 12288 data: 6 7 7
25651 002/300 Read block 2846 len 32768
25652 003/300 Write block 2029 len 24576 data: 7 8 8 8 8 10
25653 004/300 Read block 2946 len 20480
25654 005/300 Read block 3339 len 8192
25655 006/300 Write block 1460 len 16384 data: 5 5 5 5
25656 007/300 Flush
25657 008/300 Flush
25658 009/300 Write block 1770 len 20480 data: 4 6 7 7 5
25659 010/300 Read block 1020 len 16384
25660 011/300 Read block 4735 len 16384
25661 012/300 Write block 178 len 24576 data: 7 5 4 3 3 4
25662 013/300 Read block 2417 len 40960
25663 014/300 Write block 1879 len 24576 data: 6 7 6 6 4 4
25664 015/300 Write block 2052 len 16384 data: 5 5 6 4
25665 016/300 Read block 2939 len 12288
25666 017/300 Read block 2804 len 36864
25667 018/300 Flush
25668 019/300 Read block 1047 len 12288
25669 020/300 Write block 1020 len 28672 data: 3 2 3 4 4 3 3
25670 021/300 Write block 1185 len 24576 data: 3 3 4 4 5 6
25671 022/300 Write block 562 len 36864 data: 6 6 5 3 3 4 4 5 4
25672 023/300 Read block 1819 len 4096
25673 024/300 Write block 274 len 32768 data: 6 7 6 5 5 4 4 4
25674 025/300 Read block 1465 len 40960
25675 026/300 Write block 3730 len 4096 data: 5
25676 027/300 Read block 1856 len 28672
25677 028/300 Write block 3994 len 12288 data: 5 4 5
25678 029/300 Flush
25679 030/300 Read block 2077 len 16384
25680 031/300 Write block 4019 len 8192 data: 5 5
25681 032/300 Write block 3449 len 28672 data: 4 5 5 4 5 3 4
25682 033/300 Flush
25683 034/300 Read block 3664 len 20480
25684 035/300 Write block 1277 len 40960 data: 6 5 6 7 5 4 5 5 4 4
25685 036/300 Write block 3844 len 8192 data: 5 6
25686 037/300 Read block 3170 len 4096
25687 038/300 Read block 3749 len 40960
25688 039/300 Write block 1701 len 36864 data: 4 5 5 3 3 4 4 4 4
25689 040/300 Read block 380 len 16384
25690 041/300 Flush
25691 042/300 Read block 3968 len 12288
25692 043/300 Flush
25693 044/300 Read block 563 len 24576
25694 045/300 Write block 1412 len 28672 data: 5 5 6 7 7 7 7
25695 046/300 Flush
25696 047/300 Flush
25697 048/300 Write block 1581 len 8192 data: 4 3
25698 049/300 Write block 2974 len 32768 data: 4 3 7 8 11 12 11 9
25699 050/300 Read block 4848 len 36864
25700 051/300 Read block 3599 len 12288
25701 052/300 Read block 2235 len 40960
25702 053/300 Read block 1592 len 12288
25703 054/300 Flush
25704 055/300 Write block 2776 len 16384 data: 4 5 5 4
25705 056/300 Read block 4669 len 20480
25706 057/300 Read block 4544 len 36864
25707 058/300 Write block 1032 len 12288 data: 3 3 3
25708 059/300 Write block 281 len 4096 data: 5
25709 060/300 Write block 1962 len 4096 data: 7
25710 061/300 Read block 944 len 24576
25711 062/300 Read block 2691 len 24576
25712 063/300 Read block 2018 len 40960
25713 064/300 Write block 3262 len 40960 data: 5 5 5 4 4 4 4 4 2 3
25714 065/300 Read block 4912 len 20480
25715 066/300 Write block 744 len 36864 data: 4 6 6 7 11 9 9 5 3
25716 067/300 Read block 3713 len 4096
25717 068/300 Read block 1507 len 40960
25718 069/300 Read block 2787 len 4096
25719 070/300 Write block 1480 len 8192 data: 9 9
25720 071/300 Write block 1719 len 20480 data: 4 6 8 7 7
25721 072/300 Read block 4708 len 40960
25722 073/300 Write block 120 len 16384 data: 5 5 7 9
25723 074/300 Write block 1360 len 24576 data: 5 6 7 7 7 5
25724 075/300 Read block 4121 len 32768
25725 076/300 Write block 3340 len 12288 data: 7 7 7
25726 077/300 Flush
25727 078/300 Write block 1176 len 8192 data: 5 5
25728 079/300 Write block 3656 len 12288 data: 5 4 4
25729 080/300 Write block 2986 len 32768 data: 9 6 6 5 7 6 7 6
25730 081/300 Read block 2092 len 12288
25731 082/300 Read block 1265 len 8192
25732 083/300 Read block 4500 len 8192
25733 084/300 Read block 3717 len 36864
25734 085/300 Flush
25735 086/300 Write block 2584 len 8192 data: 3 3
25736 087/300 Read block 475 len 20480
25737 088/300 Read block 3403 len 20480
25738 089/300 Write block 2955 len 12288 data: 5 5 5
25739 090/300 Write block 455 len 32768 data: 4 4 4 4 4 5 3 3
25740 091/300 Flush
25741 092/300 Read block 57 len 32768
25742 093/300 Read block 4931 len 40960
25743 094/300 Read block 3635 len 32768
25744 095/300 Flush
25745 096/300 Read block 2532 len 32768
25746 097/300 Write block 2654 len 12288 data: 3 4 6
25747 098/300 Write block 3144 len 4096 data: 4
25748 099/300 Write block 3324 len 12288 data: 8 8 6
25749 100/300 Write block 962 len 4096 data: 4
25750 101/300 Write block 4626 len 28672 data: 3 4 4 4 6 6 7
25751 102/300 Write block 3649 len 32768 data: 4 4 4 6 7 6 6 6
25752 103/300 Read block 4666 len 8192
25753 104/300 Write block 1191 len 40960 data: 6 6 7 6 6 5 5 4 4 4
25754 105/300 Read block 652 len 16384
25755 106/300 Read block 3655 len 12288
25756 107/300 Write block 543 len 8192 data: 4 5
25757 108/300 Write block 3480 len 32768 data: 3 3 3 4 3 4 4 4
25758 109/300 Read block 4841 len 20480
25759 110/300 Read block 4629 len 40960
25760 111/300 Write block 2401 len 20480 data: 5 6 6 6 5
25761 112/300 Read block 2460 len 12288
25762 113/300 Write block 3323 len 4096 data: 9
25763 114/300 Read block 1490 len 24576
25764 115/300 Read block 654 len 24576
25765 116/300 Write block 1305 len 28672 data: 4 5 5 5 5 4 4
25766 117/300 Read block 2750 len 8192
25767 118/300 Read block 1206 len 20480
25768 119/300 Flush
25769 120/300 Read block 1972 len 20480
25770 121/300 Write block 4684 len 40960 data: 4 4 3 3 5 5 6 7 6 6
25771 122/300 Read block 2897 len 12288
25772 123/300 Read block 1205 len 4096
25773 124/300 Write block 4006 len 40960 data: 6 4 5 6 6 6 8 9 8 6
25774 125/300 Read block 2104 len 16384
25775 126/300 Read block 3588 len 24576
25776 127/300 Write block 4552 len 32768 data: 5 5 6 5 5 2 2 2
25777 128/300 Write block 3903 len 24576 data: 4 2 4 4 6 6
25778 129/300 Write block 774 len 28672 data: 4 5 5 5 6 7 7
25779 130/300 Write block 4128 len 36864 data: 3 3 4 5 5 4 5 6 5
25780 131/300 Read block 966 len 8192
25781 132/300 Write block 3084 len 8192 data: 4 4
25782 133/300 Read block 3276 len 16384
25783 134/300 Read block 3479 len 24576
25784 135/300 Write block 2555 len 36864 data: 5 6 5 4 2 3 4 5 6
25785 136/300 Write block 3534 len 20480 data: 3 3 3 3 3
25786 137/300 Read block 3826 len 40960
25787 138/300 Read block 910 len 16384
25788 139/300 Write block 731 len 40960 data: 7 6 7 6 6 6 4 4 4 4
25789 140/300 Write block 623 len 4096 data: 2
25790 141/300 Read block 3556 len 36864
25791 142/300 Read block 1783 len 24576
25792 143/300 Read block 4052 len 24576
25793 144/300 Read block 3248 len 12288
25794 145/300 Flush
25795 146/300 Write block 1674 len 4096 data: 5
25796 147/300 Write block 440 len 32768 data: 7 6 5 6 7 6 6 5
25797 148/300 Write block 4740 len 20480 data: 2 2 4 5 7
25798 149/300 Read block 870 len 4096
25799 150/300 Write block 3786 len 12288 data: 5 5 5
25800 151/300 Read block 853 len 36864
25801 152/300 Write block 3666 len 32768 data: 5 4 5 6 4 4 3 4
25802 153/300 Flush
25803 154/300 Read block 3172 len 8192
25804 155/300 Read block 3036 len 24576
25805 156/300 Read block 111 len 4096
25806 157/300 Read block 2318 len 12288
25807 158/300 Read block 150 len 8192
25808 159/300 Read block 3752 len 16384
25809 160/300 Write block 3614 len 20480 data: 3 3 4 3 2
25810 161/300 Write block 4522 len 20480 data: 4 3 3 4 3
25811 162/300 Flush
25812 163/300 Read block 3076 len 8192
25813 164/300 Read block 163 len 8192
25814 165/300 Write block 3917 len 12288 data: 5 5 4
25815 166/300 Write block 4579 len 32768 data: 5 5 4 3 4 5 5 5
25816 167/300 Write block 4700 len 16384 data: 8 8 8 7
25817 168/300 Write block 3208 len 36864 data: 7 8 9 9 11 10 10 9 6
25818 169/300 Flush
25819 170/300 Flush
25820 171/300 Read block 280 len 40960
25821 172/300 Read block 857 len 4096
25822 173/300 Read block 3013 len 12288
25823 174/300 Flush
25824 175/300 Read block 513 len 24576
25825 176/300 Flush
25826 177/300 Write block 1949 len 32768 data: 6 6 3 3 3 3 4 5
25827 178/300 Flush
25828 179/300 Flush
25829 180/300 Read block 4176 len 24576
25830 181/300 Read block 4339 len 8192
25831 182/300 Write block 2552 len 28672 data: 6 6 6 6 7 6 5
25832 183/300 Write block 3278 len 40960 data: 5 4 6 5 4 4 5 5 5 5
25833 184/300 Read block 4930 len 24576
25834 185/300 Read block 3161 len 28672
25835 186/300 Read block 4255 len 4096
25836 187/300 Read block 3597 len 12288
25837 188/300 Flush
25838 189/300 Read block 3918 len 24576
25839 190/300 Read block 2900 len 8192
25840 191/300 Read block 4600 len 20480
25841 192/300 Write block 4378 len 24576 data: 4 3 3 5 3 2
25842 193/300 Read block 1569 len 4096
25843 194/300 Write block 1194 len 16384 data: 7 7 6 6
25844 195/300 Read block 3761 len 12288
25845 196/300 Read block 2844 len 12288
25846 197/300 Write block 1226 len 8192 data: 3 3
25847 198/300 Write block 1401 len 20480 data: 5 3 3 5 6
25848 199/300 Read block 1279 len 4096
25849 200/300 Write block 423 len 12288 data: 6 5 6
25850 201/300 Write block 4201 len 12288 data: 4 4 5
25851 202/300 Read block 2642 len 40960
25852 203/300 Write block 2709 len 20480 data: 4 6 4 4 3
25853 204/300 Flush
25854 205/300 Read block 2760 len 4096
25855 206/300 Write block 4079 len 24576 data: 3 4 4 3 2 3
25856 207/300 Read block 1256 len 8192
25857 208/300 Write block 1524 len 16384 data: 4 6 5 4
25858 209/300 Write block 2139 len 4096 data: 4
25859 210/300 Read block 4273 len 36864
25860 211/300 Write block 1446 len 4096 data: 3
25861 212/300 Read block 3544 len 8192
25862 213/300 Flush
25863 214/300 Read block 923 len 28672
25864 215/300 Write block 271 len 36864 data: 6 6 6 7 8 7 6 6 5
25865 216/300 Write block 3293 len 40960 data: 7 7 6 5 5 4 4 4 5 5
25866 217/300 Write block 1269 len 28672 data: 3 3 4 4 5 5 5
25867 218/300 Read block 959 len 20480
25868 219/300 Read block 3930 len 32768
25869 220/300 Read block 2775 len 32768
25870 221/300 Read block 2994 len 8192
25871 222/300 Read block 2146 len 12288
25872 223/300 Read block 3814 len 4096
25873 224/300 Read block 387 len 8192
25874 225/300 Read block 1917 len 4096
25875 226/300 Write block 4405 len 8192 data: 4 4
25876 227/300 Write block 3120 len 12288 data: 4 3 4
25877 228/300 Read block 3490 len 24576
25878 229/300 Read block 552 len 32768
25879 230/300 Write block 1 len 20480 data: 4 3 3 3 3
25880 231/300 Flush
25881 232/300 Flush
25882 233/300 Write block 3908 len 16384 data: 7 7 8 8
25883 234/300 Read block 1748 len 36864
25884 235/300 Read block 447 len 32768
25885 236/300 Read block 2082 len 28672
25886 237/300 Read block 381 len 4096
25887 238/300 Write block 1704 len 8192 data: 4 4
25888 239/300 Write block 3628 len 40960 data: 9 9 7 7 8 7 6 5 5 2
25889 240/300 Read block 2601 len 12288
25890 241/300 Read block 929 len 36864
25891 242/300 Write block 1398 len 12288 data: 6 5 5
25892 243/300 Read block 1379 len 40960
25893 244/300 Read block 2768 len 32768
25894 245/300 Read block 4736 len 36864
25895 246/300 Read block 3482 len 8192
25896 247/300 Read block 4112 len 36864
25897 248/300 Read block 1793 len 36864
25898 249/300 Read block 4958 len 20480
25899 250/300 Read block 4560 len 16384
25900 251/300 Read block 4627 len 24576
25901 252/300 Flush
25902 253/300 Flush
25903 254/300 Read block 2047 len 36864
25904 255/300 Read block 3515 len 12288
25905 256/300 Read block 1204 len 4096
25906 257/300 Read block 4271 len 36864
25907 258/300 Write block 777 len 20480 data: 6 7 8 8 7
25908 259/300 Flush
25909 260/300 Write block 2940 len 36864 data: 4 4 5 6 7 7 7 7 5
25910 261/300 Write block 3918 len 24576 data: 6 5 3 4 4 3
25911 262/300 Flush
25912 263/300 Read block 2271 len 36864
25913 264/300 Read block 4039 len 28672
25914 265/300 Read block 4134 len 32768
25915 266/300 Read block 4089 len 16384
25916 267/300 Read block 977 len 28672
25917 268/300 Read block 2841 len 24576
25918 269/300 Read block 3745 len 8192
25919 270/300 Write block 2551 len 40960 data: 8 7 7 7 7 8 7 6 3 4
25920 271/300 Read block 3416 len 36864
25921 272/300 Write block 3969 len 8192 data: 5 5
25922 273/300 Read block 4658 len 32768
25923 274/300 Write block 2853 len 8192 data: 3 3
25924 275/300 Read block 3004 len 40960
25925 276/300 Read block 3832 len 32768
25926 277/300 Write block 4017 len 36864 data: 7 6 6 6 6 7 8 7 8
25927 278/300 Read block 99 len 32768
25928 279/300 Write block 2197 len 4096 data: 4
25929 280/300 Write block 2260 len 32768 data: 5 4 5 4 4 5 4 5
25930 281/300 Write block 3319 len 36864 data: 6 9 8 9 10 9 9 7 6
25931 282/300 Write block 3927 len 40960 data: 6 5 5 6 6 7 6 6 6 5
25932 283/300 Read block 4537 len 40960
25933 284/300 Read block 3324 len 16384
25934 285/300 Read block 23 len 20480
25935 286/300 Read block 3651 len 32768
25936 287/300 Write block 4557 len 32768 data: 3 3 3 5 6 7 7 7
25937 288/300 Write block 1027 len 8192 data: 4 4
25938 289/300 Read block 3615 len 36864
25939 290/300 Read block 4110 len 16384
25940 291/300 Write block 4519 len 28672 data: 6 6 6 5 4 4 5
25941 292/300 Write block 420 len 40960 data: 7 7 7 7 6 7 3 2 4 4
25942 293/300 Read block 3611 len 12288
25943 294/300 Flush
25944 295/300 Flush
25945 296/300 Flush
25946 297/300 Flush
25947 298/300 Write block 81 len 20480 data: 5 5 9 7 7
25948 299/300 Write block 2822 len 28672 data: 2 2 2 3 3 3 3
25949 300/300 Read block 1372 len 24576
25950 301/300 Write block 1597 len 28672 data: 9 9 7 6 6 4 3
25951 Main task has work to do, go find it
25952 got dsc Start(1)
25953 start 1
25954 Mar 17 22:40:18.663 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: f205f2db-81d1-4d6c-8468-ba392f496c7d, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
25955 Replay: started 1, returned:Ok(())
25956 ----------------------------------------------------------------
25957 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25958 States: Active Offline Active
25959 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25960 7642 Acked Write 3 Done Sent Done false
25961 7643 Acked Read 8 Done Sent Done false
25962 7644 Acked Write 6 Done Sent Done false
25963 7645 Acked Read 5 Done Sent Done false
25964 7646 Acked Read 2 Done Sent Done false
25965 7647 Acked Write 4 Done Sent Done false
25966 7648 Acked Flush 0 Done Sent Done false
25967 7649 Acked Flush 0 Done Sent Done false
25968 7650 Acked Write 5 Done Sent Done false
25969 7651 Acked Read 4 Done Sent Done false
25970 7652 Acked Read 4 Done Sent Done false
25971 7653 Acked Write 6 Done Sent Done false
25972 7654 Acked Read 10 Done Sent Done false
25973 7655 Acked Write 6 Done Sent Done false
25974 7656 Acked Write 4 Done Sent Done false
25975 7657 Acked Read 3 Done Sent Done false
25976 7658 Acked Read 9 Done Sent Done false
25977 7659 Acked Flush 0 Done Sent Done false
25978 7660 Acked Read 3 Done Sent Done false
25979 7661 Acked Write 7 Done Sent Done false
25980 7662 Acked Write 6 Done Sent Done false
25981 7663 Acked Write 9 Done Sent Done false
25982 7664 Acked Read 1 Done Sent Done false
25983 7665 Acked Write 8 Done Sent Done false
25984 7666 Acked Read 10 Done Sent Done false
25985 7667 Acked Write 1 Done Sent Done false
25986 7668 Acked Read 7 Done Sent Done false
25987 7669 Acked Write 3 Done Sent Done false
25988 7670 Acked Flush 0 Done Sent Done false
25989 7671 Acked Read 4 Done Sent Done false
25990 7672 Acked Write 2 Done Sent Done false
25991 7673 Acked Write 7 Done Sent Done false
25992 7674 Acked Flush 0 Done Sent Done false
25993 7675 Acked Read 5 Done Sent Done false
25994 7676 Acked Write 10 Done Sent Done false
25995 7677 Acked Write 2 Done Sent Done false
25996 7678 Acked Read 1 Done Sent Done false
25997 7679 Acked Read 10 Done Sent Done false
25998 7680 Acked Write 9 Done Sent Done false
25999 7681 Acked Read 4 Done Sent Done false
26000 7682 Acked Flush 0 Done Sent Done false
26001 7683 Acked Read 3 Done Sent Done false
26002 7684 Acked Flush 0 Done Sent Done false
26003 7685 Acked Read 6 Done Sent Done false
26004 7686 Acked Write 7 Done Sent Done false
26005 7687 Acked Flush 0 Done Sent Done false
26006 7688 Acked Flush 0 Done Sent Done false
26007 7689 Acked Write 2 Done Sent Done false
26008 7690 Acked Write 8 Done Sent Done false
26009 7691 Acked Read 9 Done Sent Done false
26010 7692 Acked Read 3 Done Sent Done false
26011 7693 Acked Read 10 Done Sent Done false
26012 7694 Acked Read 3 Done Sent Done false
26013 7695 Acked Flush 0 Done Sent Done false
26014 7696 Acked Write 4 Done Sent Done false
26015 7697 Acked Read 5 Done Sent Done false
26016 7698 Acked Read 9 Done Sent Done false
26017 7699 Acked Write 3 Done Sent Done false
26018 7700 Acked Write 1 Done Sent Done false
26019 7701 Acked Write 1 Done Sent Done false
26020 7702 Acked Read 6 Done Sent Done false
26021 7703 Acked Read 6 Done Sent Done false
26022 7704 Acked Read 10 Done Sent Done false
26023 7705 Acked Write 10 Done Sent Done false
26024 7706 Acked Read 5 Done Sent Done false
26025 7707 Acked Write 9 Done Sent Done false
26026 7708 Acked Read 1 Done Sent Done false
26027 7709 Acked Read 10 Done Sent Done false
26028 7710 Acked Read 1 Done Sent Done false
26029 7711 Acked Write 2 Done Sent Done false
26030 7712 Acked Write 5 Done Sent Done false
26031 7713 Acked Read 10 Done Sent Done false
26032 7714 Acked Write 4 Done Sent Done false
26033 7715 Acked Write 6 Done Sent Done false
26034 7716 Acked Read 8 Done Sent Done false
26035 7717 Acked Write 3 Done Sent Done false
26036 7718 Acked Flush 0 Done Sent Done false
26037 7719 Acked Write 2 Done Sent Done false
26038 7720 Acked Write 3 Done Sent Done false
26039 7721 Acked Write 8 Done Sent Done false
26040 7722 Acked Read 3 Done Sent Done false
26041 7723 Acked Read 2 Done Sent Done false
26042 7724 Acked Read 2 Done Sent Done false
26043 7725 Acked Read 9 Done Sent Done false
26044 7726 Acked Flush 0 Done Sent Done false
26045 7727 Acked Write 2 Done Sent Done false
26046 7728 Acked Read 5 Done Sent Done false
26047 7729 Acked Read 5 Done Sent Done false
26048 7730 Acked Write 3 Done Sent Done false
26049 7731 Acked Write 8 Done Sent Done false
26050 7732 Acked Flush 0 Done Sent Done false
26051 7733 Acked Read 8 Done Sent Done false
26052 7734 Acked Read 10 Done Sent Done false
26053 7735 Acked Read 8 Done Sent Done false
26054 7736 Acked Flush 0 Done Sent Done false
26055 7737 Acked Read 8 Done Sent Done false
26056 7738 Acked Write 3 Done Sent Done false
26057 7739 Acked Write 1 Done Sent Done false
26058 7740 Acked Write 3 Done Sent Done false
26059 7741 Acked Write 1 Done Sent Done false
26060 7742 Acked Write 7 Done Sent Done false
26061 7743 Acked Write 8 Done Sent Done false
26062 7744 Acked Read 2 Done Sent Done false
26063 7745 Acked Write 10 Done Sent Done false
26064 7746 Acked Read 4 Done Sent Done false
26065 7747 Acked Read 3 Done Sent Done false
26066 7748 Acked Write 2 Done Sent Done false
26067 7749 Acked Write 8 Done Sent Done false
26068 7750 Acked Read 5 Done Sent Done false
26069 7751 Acked Read 10 Done Sent Done false
26070 7752 Acked Write 5 Done Sent Done false
26071 7753 Acked Read 3 Done Sent Done false
26072 7754 Acked Write 1 Done Sent Done false
26073 7755 Acked Read 6 Done Sent Done false
26074 7756 Acked Read 6 Done Sent Done false
26075 7757 Acked Write 7 Done Sent Done false
26076 7758 Acked Read 2 Done Sent Done false
26077 7759 Acked Read 5 Done Sent Done false
26078 7760 Acked Flush 0 Done Sent Done false
26079 7761 Acked Read 5 Done Sent Done false
26080 7762 Acked Write 10 Done Sent Done false
26081 7763 Acked Read 3 Done Sent Done false
26082 7764 Acked Read 1 Done Sent Done false
26083 7765 Acked Write 10 Done Sent Done false
26084 7766 Acked Read 4 Done Sent Done false
26085 7767 Acked Read 6 Done Sent Done false
26086 7768 Acked Write 8 Done Sent Done false
26087 7769 Acked Write 6 Done Sent Done false
26088 7770 Acked Write 7 Done Sent Done false
26089 7771 Acked Write 9 Done Sent Done false
26090 7772 Acked Read 2 Done Sent Done false
26091 7773 Acked Write 2 Done Sent Done false
26092 7774 Acked Read 4 Done Sent Done false
26093 7775 Acked Read 6 Done Sent Done false
26094 7776 Acked Write 9 Done Sent Done false
26095 7777 Acked Write 5 Done Sent Done false
26096 7778 Acked Read 10 Done Sent Done false
26097 7779 Acked Read 4 Done Sent Done false
26098 7780 Acked Write 10 Done Sent Done false
26099 7781 Acked Write 1 Done Sent Done false
26100 7782 Acked Read 9 Done Sent Done false
26101 7783 Acked Read 6 Done Sent Done false
26102 7784 Acked Read 6 Done Sent Done false
26103 7785 Acked Read 3 Done Sent Done false
26104 7786 Acked Flush 0 Done Sent Done false
26105 7787 Acked Write 1 Done Sent Done false
26106 7788 Acked Write 8 Done Sent Done false
26107 7789 Acked Write 5 Done Sent Done false
26108 7790 Acked Read 1 Done Sent Done false
26109 7791 Acked Write 3 Done Sent Done false
26110 7792 Acked Read 9 Done Sent Done false
26111 7793 Acked Write 8 Done Sent Done false
26112 7794 Acked Flush 0 Done Sent Done false
26113 7795 Acked Read 2 Done Sent Done false
26114 7796 Acked Read 6 Done Sent Done false
26115 7797 Acked Read 1 Done Sent Done false
26116 7798 Acked Read 3 Done Sent Done false
26117 7799 Acked Read 2 Done Sent Done false
26118 7800 Acked Read 4 Done Sent Done false
26119 7801 Acked Write 5 Done Sent Done false
26120 7802 Acked Write 5 Done Sent Done false
26121 7803 Acked Flush 0 Done Sent Done false
26122 7804 Acked Read 2 Done Sent Done false
26123 7805 Acked Read 2 Done Sent Done false
26124 7806 Acked Write 3 Done Sent Done false
26125 7807 Acked Write 8 Done Sent Done false
26126 7808 Acked Write 4 Done Sent Done false
26127 7809 Acked Write 9 Done Sent Done false
26128 7810 Acked Flush 0 Done Sent Done false
26129 7811 Acked Flush 0 Done Sent Done false
26130 7812 Acked Read 10 Done Sent Done false
26131 7813 Acked Read 1 Done Sent Done false
26132 7814 Acked Read 3 Done Sent Done false
26133 7815 Acked Flush 0 Done Sent Done false
26134 7816 Acked Read 6 Done Sent Done false
26135 7817 Acked Flush 0 Done Sent Done false
26136 7818 Acked Write 8 Done Sent Done false
26137 7819 Acked Flush 0 Done Sent Done false
26138 7820 Acked Flush 0 Done Sent Done false
26139 7821 Acked Read 6 Done Sent Done false
26140 7822 Acked Read 2 Done Sent Done false
26141 7823 Acked Write 7 Done Sent Done false
26142 7824 Acked Write 10 Done Sent Done false
26143 7825 Acked Read 6 Done Sent Done false
26144 7826 Acked Read 7 Done Sent Done false
26145 7827 Acked Read 1 Done Sent Done false
26146 7828 Acked Read 3 Done Sent Done false
26147 7829 Acked Flush 0 Done Sent Done false
26148 7830 Acked Read 6 Done Sent Done false
26149 7831 Acked Read 2 Done Sent Done false
26150 7832 Acked Read 5 Done Sent Done false
26151 7833 Acked Write 6 Done Sent Done false
26152 7834 Acked Read 1 Done Sent Done false
26153 7835 Acked Write 4 Done Sent Done false
26154 7836 Acked Read 3 Done Sent Done false
26155 7837 Acked Read 3 Done Sent Done false
26156 7838 Acked Write 2 Done Sent Done false
26157 7839 Acked Write 5 Done Sent Done false
26158 7840 Acked Read 1 Done Sent Done false
26159 7841 Acked Write 3 Done Sent Done false
26160 7842 Acked Write 3 Done Sent Done false
26161 7843 Acked Read 10 Done Sent Done false
26162 7844 Acked Write 5 Done Sent Done false
26163 7845 Acked Flush 0 Done Sent Done false
26164 7846 Acked Read 1 Done Sent Done false
26165 7847 Acked Write 6 Done Sent Done false
26166 7848 Acked Read 2 Done Sent Done false
26167 7849 Acked Write 4 Done Sent Done false
26168 7850 Acked Write 1 Done Sent Done false
26169 7851 Acked Read 9 Done Sent Done false
26170 7852 Acked Write 1 Done Sent Done false
26171 7853 Acked Read 2 Done Sent Done false
26172 7854 Acked Flush 0 Done Sent Done false
26173 7855 Acked Read 7 Done Sent Done false
26174 7856 Acked Write 9 Done Sent Done false
26175 7857 Acked Write 10 Done Sent Done false
26176 7858 Acked Write 7 Done Sent Done false
26177 7859 Acked Read 5 Done Sent Done false
26178 7860 Acked Read 8 Done Sent Done false
26179 7861 Acked Read 8 Done Sent Done false
26180 7862 Acked Read 2 Done Sent Done false
26181 7863 Acked Read 3 Done Sent Done false
26182 7864 Acked Read 1 Done Sent Done false
26183 7865 Acked Read 2 Done Sent Done false
26184 7866 Acked Read 1 Done Sent Done false
26185 7867 Acked Write 2 Done Sent Done false
26186 7868 Acked Write 3 Done Sent Done false
26187 7869 Acked Read 6 Done Sent Done false
26188 7870 Acked Read 8 Done Sent Done false
26189 7871 Acked Write 5 Done Sent Done false
26190 7872 Acked Flush 0 Done Sent Done false
26191 7873 Acked Flush 0 Done Sent Done false
26192 7874 Acked Write 4 Done Sent Done false
26193 7875 Acked Read 9 Done Sent Done false
26194 7876 Acked Read 8 Done Sent Done false
26195 7877 Acked Read 7 Done Sent Done false
26196 7878 Acked Read 1 Done Sent Done false
26197 7879 Acked Write 2 Done Sent Done false
26198 7880 Acked Write 10 Done Sent Done false
26199 7881 Acked Read 3 Done Sent Done false
26200 7882 Acked Read 9 Done Sent Done false
26201 7883 Acked Write 3 Done Sent Done false
26202 7884 Acked Read 10 Done Sent Done false
26203 7885 Acked Read 8 Done Sent Done false
26204 7886 Acked Read 9 Done Sent Done false
26205 7887 Acked Read 2 Done Sent Done false
26206 7888 Acked Read 9 Done Sent Done false
26207 7889 Acked Read 9 Done Sent Done false
26208 7890 Acked Read 5 Done Sent Done false
26209 7891 Acked Read 4 Done Sent Done false
26210 7892 Acked Read 6 Done Sent Done false
26211 7893 Acked Flush 0 Done Sent Done false
26212 7894 Acked Flush 0 Done Sent Done false
26213 7895 Acked Read 9 Done Sent Done false
26214 7896 Acked Read 3 Done Sent Done false
26215 7897 Acked Read 1 Done Sent Done false
26216 7898 Acked Read 9 Done Sent Done false
26217 7899 Acked Write 5 Done Sent Done false
26218 7900 Acked Flush 0 Done Sent Done false
26219 7901 Acked Write 9 Done Sent Done false
26220 7902 Acked Write 6 Done Sent Done false
26221 7903 Acked Flush 0 Done Sent Done false
26222 7904 Acked Read 9 Done Sent Done false
26223 7905 Acked Read 7 Done Sent Done false
26224 7906 Acked Read 8 Done Sent Done false
26225 7907 Acked Read 4 Done Sent Done false
26226 7908 Acked Read 7 Done Sent Done false
26227 7909 Acked Read 6 Done Sent Done false
26228 7910 Acked Read 2 Done Sent Done false
26229 7911 Acked Write 10 Done Sent Done false
26230 7912 Acked Read 9 Done Sent Done false
26231 7913 Acked Write 2 Done Sent Done false
26232 7914 Acked Read 8 Done Sent Done false
26233 7915 Acked Write 2 Done Sent Done false
26234 7916 Acked Read 10 Done Sent Done false
26235 7917 Acked Read 8 Done Sent Done false
26236 7918 Acked Write 9 Done Sent Done false
26237 7919 Acked Read 8 Done Sent Done false
26238 7920 Acked Write 1 Done Sent Done false
26239 7921 Acked Write 8 Done Sent Done false
26240 7922 Acked Write 9 Done Sent Done false
26241 7923 Acked Write 10 Done Sent Done false
26242 7924 Acked Read 10 Done Sent Done false
26243 7925 Acked Read 4 Done Sent Done false
26244 7926 Acked Read 5 Done Sent Done false
26245 7927 Acked Read 8 Done Sent Done false
26246 7928 Acked Write 8 Done Sent Done false
26247 7929 Acked Write 2 Done Sent Done false
26248 7930 Acked Read 9 Done Sent Done false
26249 7931 Acked Read 4 Done Sent Done false
26250 7932 Acked Write 7 Done Sent Done false
26251 7933 Acked Write 10 Done Sent Done false
26252 7934 Acked Read 3 Done Sent Done false
26253 7935 Acked Flush 0 Done Sent Done false
26254 7936 Acked Flush 0 Done Sent Done false
26255 7937 Acked Flush 0 Done Sent Done false
26256 7938 Acked Flush 0 Done Sent Done false
26257 7939 Acked Write 5 Done Sent Done false
26258 7940 Acked Write 7 Done Sent Done false
26259 7941 Acked Read 6 Done Sent Done false
26260 7942 Acked Write 7 Done Sent Done false
26261 STATES DS:0 DS:1 DS:2 TOTAL
26262 Sent 0 301 0 301
26263 Done 301 0 301 602
26264 Skipped 0 0 0 0
26265 Error 0 0 0 0
26266 Last Flush: Some(JobId(7938)) Some(JobId(7641)) Some(JobId(7938))
26267 Downstairs last five completed: 7641 7640 7639 7638 7637
26268 Upstairs last five completed: 7942 7941 7940 7939 7938
26269 CLIENT: Up:0 ds:301 act:2
262702025-03-17T22:40:19.165ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
262712025-03-17T22:40:19.851ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
262722025-03-17T22:40:20.853ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
262732025-03-17T22:40:22.192ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
26274 ----------------------------------------------------------------
26275 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26276 States: Active Offline Active
26277 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26278 7642 Acked Write 3 Done Sent Done false
26279 7643 Acked Read 8 Done Sent Done false
26280 7644 Acked Write 6 Done Sent Done false
26281 7645 Acked Read 5 Done Sent Done false
26282 7646 Acked Read 2 Done Sent Done false
26283 7647 Acked Write 4 Done Sent Done false
26284 7648 Acked Flush 0 Done Sent Done false
26285 7649 Acked Flush 0 Done Sent Done false
26286 7650 Acked Write 5 Done Sent Done false
26287 7651 Acked Read 4 Done Sent Done false
26288 7652 Acked Read 4 Done Sent Done false
26289 7653 Acked Write 6 Done Sent Done false
26290 7654 Acked Read 10 Done Sent Done false
26291 7655 Acked Write 6 Done Sent Done false
26292 7656 Acked Write 4 Done Sent Done false
26293 7657 Acked Read 3 Done Sent Done false
26294 7658 Acked Read 9 Done Sent Done false
26295 7659 Acked Flush 0 Done Sent Done false
26296 7660 Acked Read 3 Done Sent Done false
26297 7661 Acked Write 7 Done Sent Done false
26298 7662 Acked Write 6 Done Sent Done false
26299 7663 Acked Write 9 Done Sent Done false
26300 7664 Acked Read 1 Done Sent Done false
26301 7665 Acked Write 8 Done Sent Done false
26302 7666 Acked Read 10 Done Sent Done false
26303 7667 Acked Write 1 Done Sent Done false
26304 7668 Acked Read 7 Done Sent Done false
26305 7669 Acked Write 3 Done Sent Done false
26306 7670 Acked Flush 0 Done Sent Done false
26307 7671 Acked Read 4 Done Sent Done false
26308 7672 Acked Write 2 Done Sent Done false
26309 7673 Acked Write 7 Done Sent Done false
26310 7674 Acked Flush 0 Done Sent Done false
26311 7675 Acked Read 5 Done Sent Done false
26312 7676 Acked Write 10 Done Sent Done false
26313 7677 Acked Write 2 Done Sent Done false
26314 7678 Acked Read 1 Done Sent Done false
26315 7679 Acked Read 10 Done Sent Done false
26316 7680 Acked Write 9 Done Sent Done false
26317 7681 Acked Read 4 Done Sent Done false
26318 7682 Acked Flush 0 Done Sent Done false
26319 7683 Acked Read 3 Done Sent Done false
26320 7684 Acked Flush 0 Done Sent Done false
26321 7685 Acked Read 6 Done Sent Done false
26322 7686 Acked Write 7 Done Sent Done false
26323 7687 Acked Flush 0 Done Sent Done false
26324 7688 Acked Flush 0 Done Sent Done false
26325 7689 Acked Write 2 Done Sent Done false
26326 7690 Acked Write 8 Done Sent Done false
26327 7691 Acked Read 9 Done Sent Done false
26328 7692 Acked Read 3 Done Sent Done false
26329 7693 Acked Read 10 Done Sent Done false
26330 7694 Acked Read 3 Done Sent Done false
26331 7695 Acked Flush 0 Done Sent Done false
26332 7696 Acked Write 4 Done Sent Done false
26333 7697 Acked Read 5 Done Sent Done false
26334 7698 Acked Read 9 Done Sent Done false
26335 7699 Acked Write 3 Done Sent Done false
26336 7700 Acked Write 1 Done Sent Done false
26337 7701 Acked Write 1 Done Sent Done false
26338 7702 Acked Read 6 Done Sent Done false
26339 7703 Acked Read 6 Done Sent Done false
26340 7704 Acked Read 10 Done Sent Done false
26341 7705 Acked Write 10 Done Sent Done false
26342 7706 Acked Read 5 Done Sent Done false
26343 7707 Acked Write 9 Done Sent Done false
26344 7708 Acked Read 1 Done Sent Done false
26345 7709 Acked Read 10 Done Sent Done false
26346 7710 Acked Read 1 Done Sent Done false
26347 7711 Acked Write 2 Done Sent Done false
26348 7712 Acked Write 5 Done Sent Done false
26349 7713 Acked Read 10 Done Sent Done false
26350 7714 Acked Write 4 Done Sent Done false
26351 7715 Acked Write 6 Done Sent Done false
26352 7716 Acked Read 8 Done Sent Done false
26353 7717 Acked Write 3 Done Sent Done false
26354 7718 Acked Flush 0 Done Sent Done false
26355 7719 Acked Write 2 Done Sent Done false
26356 7720 Acked Write 3 Done Sent Done false
26357 7721 Acked Write 8 Done Sent Done false
26358 7722 Acked Read 3 Done Sent Done false
26359 7723 Acked Read 2 Done Sent Done false
26360 7724 Acked Read 2 Done Sent Done false
26361 7725 Acked Read 9 Done Sent Done false
26362 7726 Acked Flush 0 Done Sent Done false
26363 7727 Acked Write 2 Done Sent Done false
26364 7728 Acked Read 5 Done Sent Done false
26365 7729 Acked Read 5 Done Sent Done false
26366 7730 Acked Write 3 Done Sent Done false
26367 7731 Acked Write 8 Done Sent Done false
26368 7732 Acked Flush 0 Done Sent Done false
26369 7733 Acked Read 8 Done Sent Done false
26370 7734 Acked Read 10 Done Sent Done false
26371 7735 Acked Read 8 Done Sent Done false
26372 7736 Acked Flush 0 Done Sent Done false
26373 7737 Acked Read 8 Done Sent Done false
26374 7738 Acked Write 3 Done Sent Done false
26375 7739 Acked Write 1 Done Sent Done false
26376 7740 Acked Write 3 Done Sent Done false
26377 7741 Acked Write 1 Done Sent Done false
26378 7742 Acked Write 7 Done Sent Done false
26379 7743 Acked Write 8 Done Sent Done false
26380 7744 Acked Read 2 Done Sent Done false
26381 7745 Acked Write 10 Done Sent Done false
26382 7746 Acked Read 4 Done Sent Done false
26383 7747 Acked Read 3 Done Sent Done false
26384 7748 Acked Write 2 Done Sent Done false
26385 7749 Acked Write 8 Done Sent Done false
26386 7750 Acked Read 5 Done Sent Done false
26387 7751 Acked Read 10 Done Sent Done false
26388 7752 Acked Write 5 Done Sent Done false
26389 7753 Acked Read 3 Done Sent Done false
26390 7754 Acked Write 1 Done Sent Done false
26391 7755 Acked Read 6 Done Sent Done false
26392 7756 Acked Read 6 Done Sent Done false
26393 7757 Acked Write 7 Done Sent Done false
26394 7758 Acked Read 2 Done Sent Done false
26395 7759 Acked Read 5 Done Sent Done false
26396 7760 Acked Flush 0 Done Sent Done false
26397 7761 Acked Read 5 Done Sent Done false
26398 7762 Acked Write 10 Done Sent Done false
26399 7763 Acked Read 3 Done Sent Done false
26400 7764 Acked Read 1 Done Sent Done false
26401 7765 Acked Write 10 Done Sent Done false
26402 7766 Acked Read 4 Done Sent Done false
26403 7767 Acked Read 6 Done Sent Done false
26404 7768 Acked Write 8 Done Sent Done false
26405 7769 Acked Write 6 Done Sent Done false
26406 7770 Acked Write 7 Done Sent Done false
26407 7771 Acked Write 9 Done Sent Done false
26408 7772 Acked Read 2 Done Sent Done false
26409 7773 Acked Write 2 Done Sent Done false
26410 7774 Acked Read 4 Done Sent Done false
26411 7775 Acked Read 6 Done Sent Done false
26412 7776 Acked Write 9 Done Sent Done false
26413 7777 Acked Write 5 Done Sent Done false
26414 7778 Acked Read 10 Done Sent Done false
26415 7779 Acked Read 4 Done Sent Done false
26416 7780 Acked Write 10 Done Sent Done false
26417 7781 Acked Write 1 Done Sent Done false
26418 7782 Acked Read 9 Done Sent Done false
26419 7783 Acked Read 6 Done Sent Done false
26420 7784 Acked Read 6 Done Sent Done false
26421 7785 Acked Read 3 Done Sent Done false
26422 7786 Acked Flush 0 Done Sent Done false
26423 7787 Acked Write 1 Done Sent Done false
26424 7788 Acked Write 8 Done Sent Done false
26425 7789 Acked Write 5 Done Sent Done false
26426 7790 Acked Read 1 Done Sent Done false
26427 7791 Acked Write 3 Done Sent Done false
26428 7792 Acked Read 9 Done Sent Done false
26429 7793 Acked Write 8 Done Sent Done false
26430 7794 Acked Flush 0 Done Sent Done false
26431 7795 Acked Read 2 Done Sent Done false
26432 7796 Acked Read 6 Done Sent Done false
26433 7797 Acked Read 1 Done Sent Done false
26434 7798 Acked Read 3 Done Sent Done false
26435 7799 Acked Read 2 Done Sent Done false
26436 7800 Acked Read 4 Done Sent Done false
26437 7801 Acked Write 5 Done Sent Done false
26438 7802 Acked Write 5 Done Sent Done false
26439 7803 Acked Flush 0 Done Sent Done false
26440 7804 Acked Read 2 Done Sent Done false
26441 7805 Acked Read 2 Done Sent Done false
26442 7806 Acked Write 3 Done Sent Done false
26443 7807 Acked Write 8 Done Sent Done false
26444 7808 Acked Write 4 Done Sent Done false
26445 7809 Acked Write 9 Done Sent Done false
26446 7810 Acked Flush 0 Done Sent Done false
26447 7811 Acked Flush 0 Done Sent Done false
26448 7812 Acked Read 10 Done Sent Done false
26449 7813 Acked Read 1 Done Sent Done false
26450 7814 Acked Read 3 Done Sent Done false
26451 7815 Acked Flush 0 Done Sent Done false
26452 7816 Acked Read 6 Done Sent Done false
26453 7817 Acked Flush 0 Done Sent Done false
26454 7818 Acked Write 8 Done Sent Done false
26455 7819 Acked Flush 0 Done Sent Done false
26456 7820 Acked Flush 0 Done Sent Done false
26457 7821 Acked Read 6 Done Sent Done false
26458 7822 Acked Read 2 Done Sent Done false
26459 7823 Acked Write 7 Done Sent Done false
26460 7824 Acked Write 10 Done Sent Done false
26461 7825 Acked Read 6 Done Sent Done false
26462 7826 Acked Read 7 Done Sent Done false
26463 7827 Acked Read 1 Done Sent Done false
26464 7828 Acked Read 3 Done Sent Done false
26465 7829 Acked Flush 0 Done Sent Done false
26466 7830 Acked Read 6 Done Sent Done false
26467 7831 Acked Read 2 Done Sent Done false
26468 7832 Acked Read 5 Done Sent Done false
26469 7833 Acked Write 6 Done Sent Done false
26470 7834 Acked Read 1 Done Sent Done false
26471 7835 Acked Write 4 Done Sent Done false
26472 7836 Acked Read 3 Done Sent Done false
26473 7837 Acked Read 3 Done Sent Done false
26474 7838 Acked Write 2 Done Sent Done false
26475 7839 Acked Write 5 Done Sent Done false
26476 7840 Acked Read 1 Done Sent Done false
26477 7841 Acked Write 3 Done Sent Done false
26478 7842 Acked Write 3 Done Sent Done false
26479 7843 Acked Read 10 Done Sent Done false
26480 7844 Acked Write 5 Done Sent Done false
26481 7845 Acked Flush 0 Done Sent Done false
26482 7846 Acked Read 1 Done Sent Done false
26483 7847 Acked Write 6 Done Sent Done false
26484 7848 Acked Read 2 Done Sent Done false
26485 7849 Acked Write 4 Done Sent Done false
26486 7850 Acked Write 1 Done Sent Done false
26487 7851 Acked Read 9 Done Sent Done false
26488 7852 Acked Write 1 Done Sent Done false
26489 7853 Acked Read 2 Done Sent Done false
26490 7854 Acked Flush 0 Done Sent Done false
26491 7855 Acked Read 7 Done Sent Done false
26492 7856 Acked Write 9 Done Sent Done false
26493 7857 Acked Write 10 Done Sent Done false
26494 7858 Acked Write 7 Done Sent Done false
26495 7859 Acked Read 5 Done Sent Done false
26496 7860 Acked Read 8 Done Sent Done false
26497 7861 Acked Read 8 Done Sent Done false
26498 7862 Acked Read 2 Done Sent Done false
26499 7863 Acked Read 3 Done Sent Done false
26500 7864 Acked Read 1 Done Sent Done false
26501 7865 Acked Read 2 Done Sent Done false
26502 7866 Acked Read 1 Done Sent Done false
26503 7867 Acked Write 2 Done Sent Done false
26504 7868 Acked Write 3 Done Sent Done false
26505 7869 Acked Read 6 Done Sent Done false
26506 7870 Acked Read 8 Done Sent Done false
26507 7871 Acked Write 5 Done Sent Done false
26508 7872 Acked Flush 0 Done Sent Done false
26509 7873 Acked Flush 0 Done Sent Done false
26510 7874 Acked Write 4 Done Sent Done false
26511 7875 Acked Read 9 Done Sent Done false
26512 7876 Acked Read 8 Done Sent Done false
26513 7877 Acked Read 7 Done Sent Done false
26514 7878 Acked Read 1 Done Sent Done false
26515 7879 Acked Write 2 Done Sent Done false
26516 7880 Acked Write 10 Done Sent Done false
26517 7881 Acked Read 3 Done Sent Done false
26518 7882 Acked Read 9 Done Sent Done false
26519 7883 Acked Write 3 Done Sent Done false
26520 7884 Acked Read 10 Done Sent Done false
26521 7885 Acked Read 8 Done Sent Done false
26522 7886 Acked Read 9 Done Sent Done false
26523 7887 Acked Read 2 Done Sent Done false
26524 7888 Acked Read 9 Done Sent Done false
26525 7889 Acked Read 9 Done Sent Done false
26526 7890 Acked Read 5 Done Sent Done false
26527 7891 Acked Read 4 Done Sent Done false
26528 7892 Acked Read 6 Done Sent Done false
26529 7893 Acked Flush 0 Done Sent Done false
26530 7894 Acked Flush 0 Done Sent Done false
26531 7895 Acked Read 9 Done Sent Done false
26532 7896 Acked Read 3 Done Sent Done false
26533 7897 Acked Read 1 Done Sent Done false
26534 7898 Acked Read 9 Done Sent Done false
26535 7899 Acked Write 5 Done Sent Done false
26536 7900 Acked Flush 0 Done Sent Done false
26537 7901 Acked Write 9 Done Sent Done false
26538 7902 Acked Write 6 Done Sent Done false
26539 7903 Acked Flush 0 Done Sent Done false
26540 7904 Acked Read 9 Done Sent Done false
26541 7905 Acked Read 7 Done Sent Done false
26542 7906 Acked Read 8 Done Sent Done false
26543 7907 Acked Read 4 Done Sent Done false
26544 7908 Acked Read 7 Done Sent Done false
26545 7909 Acked Read 6 Done Sent Done false
26546 7910 Acked Read 2 Done Sent Done false
26547 7911 Acked Write 10 Done Sent Done false
26548 7912 Acked Read 9 Done Sent Done false
26549 7913 Acked Write 2 Done Sent Done false
26550 7914 Acked Read 8 Done Sent Done false
26551 7915 Acked Write 2 Done Sent Done false
26552 7916 Acked Read 10 Done Sent Done false
26553 7917 Acked Read 8 Done Sent Done false
26554 7918 Acked Write 9 Done Sent Done false
26555 7919 Acked Read 8 Done Sent Done false
26556 7920 Acked Write 1 Done Sent Done false
26557 7921 Acked Write 8 Done Sent Done false
26558 7922 Acked Write 9 Done Sent Done false
26559 7923 Acked Write 10 Done Sent Done false
26560 7924 Acked Read 10 Done Sent Done false
26561 7925 Acked Read 4 Done Sent Done false
26562 7926 Acked Read 5 Done Sent Done false
26563 7927 Acked Read 8 Done Sent Done false
26564 7928 Acked Write 8 Done Sent Done false
26565 7929 Acked Write 2 Done Sent Done false
26566 7930 Acked Read 9 Done Sent Done false
26567 7931 Acked Read 4 Done Sent Done false
26568 7932 Acked Write 7 Done Sent Done false
26569 7933 Acked Write 10 Done Sent Done false
26570 7934 Acked Read 3 Done Sent Done false
26571 7935 Acked Flush 0 Done Sent Done false
26572 7936 Acked Flush 0 Done Sent Done false
26573 7937 Acked Flush 0 Done Sent Done false
26574 7938 Acked Flush 0 Done Sent Done false
26575 7939 Acked Write 5 Done Sent Done false
26576 7940 Acked Write 7 Done Sent Done false
26577 7941 Acked Read 6 Done Sent Done false
26578 7942 Acked Write 7 Done Sent Done false
26579 STATES DS:0 DS:1 DS:2 TOTAL
26580 Sent 0 301 0 301
26581 Done 301 0 301 602
26582 Skipped 0 0 0 0
26583 Error 0 0 0 0
26584 Last Flush: Some(JobId(7938)) Some(JobId(7641)) Some(JobId(7938))
26585 Downstairs last five completed: 7641 7640 7639 7638 7637
26586 Upstairs last five completed: 7942 7941 7940 7939 7938
26587 CLIENT: Up:0 ds:301 act:2
265882025-03-17T22:40:23.167ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265892025-03-17T22:40:23.857ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265902025-03-17T22:40:24.423ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265912025-03-17T22:40:24.423ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265922025-03-17T22:40:24.423ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265932025-03-17T22:40:24.423ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265942025-03-17T22:40:24.423ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265952025-03-17T22:40:24.924ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
265962025-03-17T22:40:25.860ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
26597 ----------------------------------------------------------------
26598 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26599 States: Active Offline Active
26600 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26601 7642 Acked Write 3 Done Sent Done false
26602 7643 Acked Read 8 Done Sent Done false
26603 7644 Acked Write 6 Done Sent Done false
26604 7645 Acked Read 5 Done Sent Done false
26605 7646 Acked Read 2 Done Sent Done false
26606 7647 Acked Write 4 Done Sent Done false
26607 7648 Acked Flush 0 Done Sent Done false
26608 7649 Acked Flush 0 Done Sent Done false
26609 7650 Acked Write 5 Done Sent Done false
26610 7651 Acked Read 4 Done Sent Done false
26611 7652 Acked Read 4 Done Sent Done false
26612 7653 Acked Write 6 Done Sent Done false
26613 7654 Acked Read 10 Done Sent Done false
26614 7655 Acked Write 6 Done Sent Done false
26615 7656 Acked Write 4 Done Sent Done false
26616 7657 Acked Read 3 Done Sent Done false
26617 7658 Acked Read 9 Done Sent Done false
26618 7659 Acked Flush 0 Done Sent Done false
26619 7660 Acked Read 3 Done Sent Done false
26620 7661 Acked Write 7 Done Sent Done false
26621 7662 Acked Write 6 Done Sent Done false
26622 7663 Acked Write 9 Done Sent Done false
26623 7664 Acked Read 1 Done Sent Done false
26624 7665 Acked Write 8 Done Sent Done false
26625 7666 Acked Read 10 Done Sent Done false
26626 7667 Acked Write 1 Done Sent Done false
26627 7668 Acked Read 7 Done Sent Done false
26628 7669 Acked Write 3 Done Sent Done false
26629 7670 Acked Flush 0 Done Sent Done false
26630 7671 Acked Read 4 Done Sent Done false
26631 7672 Acked Write 2 Done Sent Done false
26632 7673 Acked Write 7 Done Sent Done false
26633 7674 Acked Flush 0 Done Sent Done false
26634 7675 Acked Read 5 Done Sent Done false
26635 7676 Acked Write 10 Done Sent Done false
26636 7677 Acked Write 2 Done Sent Done false
26637 7678 Acked Read 1 Done Sent Done false
26638 7679 Acked Read 10 Done Sent Done false
26639 7680 Acked Write 9 Done Sent Done false
26640 7681 Acked Read 4 Done Sent Done false
26641 7682 Acked Flush 0 Done Sent Done false
26642 7683 Acked Read 3 Done Sent Done false
26643 7684 Acked Flush 0 Done Sent Done false
26644 7685 Acked Read 6 Done Sent Done false
26645 7686 Acked Write 7 Done Sent Done false
26646 7687 Acked Flush 0 Done Sent Done false
26647 7688 Acked Flush 0 Done Sent Done false
26648 7689 Acked Write 2 Done Sent Done false
26649 7690 Acked Write 8 Done Sent Done false
26650 7691 Acked Read 9 Done Sent Done false
26651 7692 Acked Read 3 Done Sent Done false
26652 7693 Acked Read 10 Done Sent Done false
26653 7694 Acked Read 3 Done Sent Done false
26654 7695 Acked Flush 0 Done Sent Done false
26655 7696 Acked Write 4 Done Sent Done false
26656 7697 Acked Read 5 Done Sent Done false
26657 7698 Acked Read 9 Done Sent Done false
26658 7699 Acked Write 3 Done Sent Done false
26659 7700 Acked Write 1 Done Sent Done false
26660 7701 Acked Write 1 Done Sent Done false
26661 7702 Acked Read 6 Done Sent Done false
26662 7703 Acked Read 6 Done Sent Done false
26663 7704 Acked Read 10 Done Sent Done false
26664 7705 Acked Write 10 Done Sent Done false
26665 7706 Acked Read 5 Done Sent Done false
26666 7707 Acked Write 9 Done Sent Done false
26667 7708 Acked Read 1 Done Sent Done false
26668 7709 Acked Read 10 Done Sent Done false
26669 7710 Acked Read 1 Done Sent Done false
26670 7711 Acked Write 2 Done Sent Done false
26671 7712 Acked Write 5 Done Sent Done false
26672 7713 Acked Read 10 Done Sent Done false
26673 7714 Acked Write 4 Done Sent Done false
26674 7715 Acked Write 6 Done Sent Done false
26675 7716 Acked Read 8 Done Sent Done false
26676 7717 Acked Write 3 Done Sent Done false
26677 7718 Acked Flush 0 Done Sent Done false
26678 7719 Acked Write 2 Done Sent Done false
26679 7720 Acked Write 3 Done Sent Done false
26680 7721 Acked Write 8 Done Sent Done false
26681 7722 Acked Read 3 Done Sent Done false
26682 7723 Acked Read 2 Done Sent Done false
26683 7724 Acked Read 2 Done Sent Done false
26684 7725 Acked Read 9 Done Sent Done false
26685 7726 Acked Flush 0 Done Sent Done false
26686 7727 Acked Write 2 Done Sent Done false
26687 7728 Acked Read 5 Done Sent Done false
26688 7729 Acked Read 5 Done Sent Done false
26689 7730 Acked Write 3 Done Sent Done false
26690 7731 Acked Write 8 Done Sent Done false
26691 7732 Acked Flush 0 Done Sent Done false
26692 7733 Acked Read 8 Done Sent Done false
26693 7734 Acked Read 10 Done Sent Done false
26694 7735 Acked Read 8 Done Sent Done false
26695 7736 Acked Flush 0 Done Sent Done false
26696 7737 Acked Read 8 Done Sent Done false
26697 7738 Acked Write 3 Done Sent Done false
26698 7739 Acked Write 1 Done Sent Done false
26699 7740 Acked Write 3 Done Sent Done false
26700 7741 Acked Write 1 Done Sent Done false
26701 7742 Acked Write 7 Done Sent Done false
26702 7743 Acked Write 8 Done Sent Done false
26703 7744 Acked Read 2 Done Sent Done false
26704 7745 Acked Write 10 Done Sent Done false
26705 7746 Acked Read 4 Done Sent Done false
26706 7747 Acked Read 3 Done Sent Done false
26707 7748 Acked Write 2 Done Sent Done false
26708 7749 Acked Write 8 Done Sent Done false
26709 7750 Acked Read 5 Done Sent Done false
26710 7751 Acked Read 10 Done Sent Done false
26711 7752 Acked Write 5 Done Sent Done false
26712 7753 Acked Read 3 Done Sent Done false
26713 7754 Acked Write 1 Done Sent Done false
26714 7755 Acked Read 6 Done Sent Done false
26715 7756 Acked Read 6 Done Sent Done false
26716 7757 Acked Write 7 Done Sent Done false
26717 7758 Acked Read 2 Done Sent Done false
26718 7759 Acked Read 5 Done Sent Done false
26719 7760 Acked Flush 0 Done Sent Done false
26720 7761 Acked Read 5 Done Sent Done false
26721 7762 Acked Write 10 Done Sent Done false
26722 7763 Acked Read 3 Done Sent Done false
26723 7764 Acked Read 1 Done Sent Done false
26724 7765 Acked Write 10 Done Sent Done false
26725 7766 Acked Read 4 Done Sent Done false
26726 7767 Acked Read 6 Done Sent Done false
26727 7768 Acked Write 8 Done Sent Done false
26728 7769 Acked Write 6 Done Sent Done false
26729 7770 Acked Write 7 Done Sent Done false
26730 7771 Acked Write 9 Done Sent Done false
26731 7772 Acked Read 2 Done Sent Done false
26732 7773 Acked Write 2 Done Sent Done false
26733 7774 Acked Read 4 Done Sent Done false
26734 7775 Acked Read 6 Done Sent Done false
26735 7776 Acked Write 9 Done Sent Done false
26736 7777 Acked Write 5 Done Sent Done false
26737 7778 Acked Read 10 Done Sent Done false
26738 7779 Acked Read 4 Done Sent Done false
26739 7780 Acked Write 10 Done Sent Done false
26740 7781 Acked Write 1 Done Sent Done false
26741 7782 Acked Read 9 Done Sent Done false
26742 7783 Acked Read 6 Done Sent Done false
26743 7784 Acked Read 6 Done Sent Done false
26744 7785 Acked Read 3 Done Sent Done false
26745 7786 Acked Flush 0 Done Sent Done false
26746 7787 Acked Write 1 Done Sent Done false
26747 7788 Acked Write 8 Done Sent Done false
26748 7789 Acked Write 5 Done Sent Done false
26749 7790 Acked Read 1 Done Sent Done false
26750 7791 Acked Write 3 Done Sent Done false
26751 7792 Acked Read 9 Done Sent Done false
26752 7793 Acked Write 8 Done Sent Done false
26753 7794 Acked Flush 0 Done Sent Done false
26754 7795 Acked Read 2 Done Sent Done false
26755 7796 Acked Read 6 Done Sent Done false
26756 7797 Acked Read 1 Done Sent Done false
26757 7798 Acked Read 3 Done Sent Done false
26758 7799 Acked Read 2 Done Sent Done false
26759 7800 Acked Read 4 Done Sent Done false
26760 7801 Acked Write 5 Done Sent Done false
26761 7802 Acked Write 5 Done Sent Done false
26762 7803 Acked Flush 0 Done Sent Done false
26763 7804 Acked Read 2 Done Sent Done false
26764 7805 Acked Read 2 Done Sent Done false
26765 7806 Acked Write 3 Done Sent Done false
26766 7807 Acked Write 8 Done Sent Done false
26767 7808 Acked Write 4 Done Sent Done false
26768 7809 Acked Write 9 Done Sent Done false
26769 7810 Acked Flush 0 Done Sent Done false
26770 7811 Acked Flush 0 Done Sent Done false
26771 7812 Acked Read 10 Done Sent Done false
26772 7813 Acked Read 1 Done Sent Done false
26773 7814 Acked Read 3 Done Sent Done false
26774 7815 Acked Flush 0 Done Sent Done false
26775 7816 Acked Read 6 Done Sent Done false
26776 7817 Acked Flush 0 Done Sent Done false
26777 7818 Acked Write 8 Done Sent Done false
26778 7819 Acked Flush 0 Done Sent Done false
26779 7820 Acked Flush 0 Done Sent Done false
26780 7821 Acked Read 6 Done Sent Done false
26781 7822 Acked Read 2 Done Sent Done false
26782 7823 Acked Write 7 Done Sent Done false
26783 7824 Acked Write 10 Done Sent Done false
26784 7825 Acked Read 6 Done Sent Done false
26785 7826 Acked Read 7 Done Sent Done false
26786 7827 Acked Read 1 Done Sent Done false
26787 7828 Acked Read 3 Done Sent Done false
26788 7829 Acked Flush 0 Done Sent Done false
26789 7830 Acked Read 6 Done Sent Done false
26790 7831 Acked Read 2 Done Sent Done false
26791 7832 Acked Read 5 Done Sent Done false
26792 7833 Acked Write 6 Done Sent Done false
26793 7834 Acked Read 1 Done Sent Done false
26794 7835 Acked Write 4 Done Sent Done false
26795 7836 Acked Read 3 Done Sent Done false
26796 7837 Acked Read 3 Done Sent Done false
26797 7838 Acked Write 2 Done Sent Done false
26798 7839 Acked Write 5 Done Sent Done false
26799 7840 Acked Read 1 Done Sent Done false
26800 7841 Acked Write 3 Done Sent Done false
26801 7842 Acked Write 3 Done Sent Done false
26802 7843 Acked Read 10 Done Sent Done false
26803 7844 Acked Write 5 Done Sent Done false
26804 7845 Acked Flush 0 Done Sent Done false
26805 7846 Acked Read 1 Done Sent Done false
26806 7847 Acked Write 6 Done Sent Done false
26807 7848 Acked Read 2 Done Sent Done false
26808 7849 Acked Write 4 Done Sent Done false
26809 7850 Acked Write 1 Done Sent Done false
26810 7851 Acked Read 9 Done Sent Done false
26811 7852 Acked Write 1 Done Sent Done false
26812 7853 Acked Read 2 Done Sent Done false
26813 7854 Acked Flush 0 Done Sent Done false
26814 7855 Acked Read 7 Done Sent Done false
26815 7856 Acked Write 9 Done Sent Done false
26816 7857 Acked Write 10 Done Sent Done false
26817 7858 Acked Write 7 Done Sent Done false
26818 7859 Acked Read 5 Done Sent Done false
26819 7860 Acked Read 8 Done Sent Done false
26820 7861 Acked Read 8 Done Sent Done false
26821 7862 Acked Read 2 Done Sent Done false
26822 7863 Acked Read 3 Done Sent Done false
26823 7864 Acked Read 1 Done Sent Done false
26824 7865 Acked Read 2 Done Sent Done false
26825 7866 Acked Read 1 Done Sent Done false
26826 7867 Acked Write 2 Done Sent Done false
26827 7868 Acked Write 3 Done Sent Done false
26828 7869 Acked Read 6 Done Sent Done false
26829 7870 Acked Read 8 Done Sent Done false
26830 7871 Acked Write 5 Done Sent Done false
26831 7872 Acked Flush 0 Done Sent Done false
26832 7873 Acked Flush 0 Done Sent Done false
26833 7874 Acked Write 4 Done Sent Done false
26834 7875 Acked Read 9 Done Sent Done false
26835 7876 Acked Read 8 Done Sent Done false
26836 7877 Acked Read 7 Done Sent Done false
26837 7878 Acked Read 1 Done Sent Done false
26838 7879 Acked Write 2 Done Sent Done false
26839 7880 Acked Write 10 Done Sent Done false
26840 7881 Acked Read 3 Done Sent Done false
26841 7882 Acked Read 9 Done Sent Done false
26842 7883 Acked Write 3 Done Sent Done false
26843 7884 Acked Read 10 Done Sent Done false
26844 7885 Acked Read 8 Done Sent Done false
26845 7886 Acked Read 9 Done Sent Done false
26846 7887 Acked Read 2 Done Sent Done false
26847 7888 Acked Read 9 Done Sent Done false
26848 7889 Acked Read 9 Done Sent Done false
26849 7890 Acked Read 5 Done Sent Done false
26850 7891 Acked Read 4 Done Sent Done false
26851 7892 Acked Read 6 Done Sent Done false
26852 7893 Acked Flush 0 Done Sent Done false
26853 7894 Acked Flush 0 Done Sent Done false
26854 7895 Acked Read 9 Done Sent Done false
26855 7896 Acked Read 3 Done Sent Done false
26856 7897 Acked Read 1 Done Sent Done false
26857 7898 Acked Read 9 Done Sent Done false
26858 7899 Acked Write 5 Done Sent Done false
26859 7900 Acked Flush 0 Done Sent Done false
26860 7901 Acked Write 9 Done Sent Done false
26861 7902 Acked Write 6 Done Sent Done false
26862 7903 Acked Flush 0 Done Sent Done false
26863 7904 Acked Read 9 Done Sent Done false
26864 7905 Acked Read 7 Done Sent Done false
26865 7906 Acked Read 8 Done Sent Done false
26866 7907 Acked Read 4 Done Sent Done false
26867 7908 Acked Read 7 Done Sent Done false
26868 7909 Acked Read 6 Done Sent Done false
26869 7910 Acked Read 2 Done Sent Done false
26870 7911 Acked Write 10 Done Sent Done false
26871 7912 Acked Read 9 Done Sent Done false
26872 7913 Acked Write 2 Done Sent Done false
26873 7914 Acked Read 8 Done Sent Done false
26874 7915 Acked Write 2 Done Sent Done false
26875 7916 Acked Read 10 Done Sent Done false
26876 7917 Acked Read 8 Done Sent Done false
26877 7918 Acked Write 9 Done Sent Done false
26878 7919 Acked Read 8 Done Sent Done false
26879 7920 Acked Write 1 Done Sent Done false
26880 7921 Acked Write 8 Done Sent Done false
26881 7922 Acked Write 9 Done Sent Done false
26882 7923 Acked Write 10 Done Sent Done false
26883 7924 Acked Read 10 Done Sent Done false
26884 7925 Acked Read 4 Done Sent Done false
26885 7926 Acked Read 5 Done Sent Done false
26886 7927 Acked Read 8 Done Sent Done false
26887 7928 Acked Write 8 Done Sent Done false
26888 7929 Acked Write 2 Done Sent Done false
26889 7930 Acked Read 9 Done Sent Done false
26890 7931 Acked Read 4 Done Sent Done false
26891 7932 Acked Write 7 Done Sent Done false
26892 7933 Acked Write 10 Done Sent Done false
26893 7934 Acked Read 3 Done Sent Done false
26894 7935 Acked Flush 0 Done Sent Done false
26895 7936 Acked Flush 0 Done Sent Done false
26896 7937 Acked Flush 0 Done Sent Done false
26897 7938 Acked Flush 0 Done Sent Done false
26898 7939 Acked Write 5 Done Sent Done false
26899 7940 Acked Write 7 Done Sent Done false
26900 7941 Acked Read 6 Done Sent Done false
26901 7942 Acked Write 7 Done Sent Done false
26902 STATES DS:0 DS:1 DS:2 TOTAL
26903 Sent 0 301 0 301
26904 Done 301 0 301 602
26905 Skipped 0 0 0 0
26906 Error 0 0 0 0
26907 Last Flush: Some(JobId(7938)) Some(JobId(7641)) Some(JobId(7938))
26908 Downstairs last five completed: 7641 7640 7639 7638 7637
26909 Upstairs last five completed: 7942 7941 7940 7939 7938
26910 CLIENT: Up:0 ds:301 act:2
269112025-03-17T22:40:27.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
269122025-03-17T22:40:27.863ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
26913 [8820] got start action
26914 [8820] I am going to restart
26915 Starting downstairs at port 8820
26916 Make output file at "/tmp/dsc/downstairs-8820.txt"
26917 [8820][1] reports Starting
26918 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1231)
26919 [8820][1] reports Running
269202025-03-17T22:40:28.864ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
269212025-03-17T22:40:29.865ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
26922 ----------------------------------------------------------------
26923 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26924 States: Active Offline Active
26925 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26926 7642 Acked Write 3 Done Sent Done false
26927 7643 Acked Read 8 Done Sent Done false
26928 7644 Acked Write 6 Done Sent Done false
26929 7645 Acked Read 5 Done Sent Done false
26930 7646 Acked Read 2 Done Sent Done false
26931 7647 Acked Write 4 Done Sent Done false
26932 7648 Acked Flush 0 Done Sent Done false
26933 7649 Acked Flush 0 Done Sent Done false
26934 7650 Acked Write 5 Done Sent Done false
26935 7651 Acked Read 4 Done Sent Done false
26936 7652 Acked Read 4 Done Sent Done false
26937 7653 Acked Write 6 Done Sent Done false
26938 7654 Acked Read 10 Done Sent Done false
26939 7655 Acked Write 6 Done Sent Done false
26940 7656 Acked Write 4 Done Sent Done false
26941 7657 Acked Read 3 Done Sent Done false
26942 7658 Acked Read 9 Done Sent Done false
26943 7659 Acked Flush 0 Done Sent Done false
26944 7660 Acked Read 3 Done Sent Done false
26945 7661 Acked Write 7 Done Sent Done false
26946 7662 Acked Write 6 Done Sent Done false
26947 7663 Acked Write 9 Done Sent Done false
26948 7664 Acked Read 1 Done Sent Done false
26949 7665 Acked Write 8 Done Sent Done false
26950 7666 Acked Read 10 Done Sent Done false
26951 7667 Acked Write 1 Done Sent Done false
26952 7668 Acked Read 7 Done Sent Done false
26953 7669 Acked Write 3 Done Sent Done false
26954 7670 Acked Flush 0 Done Sent Done false
26955 7671 Acked Read 4 Done Sent Done false
26956 7672 Acked Write 2 Done Sent Done false
26957 7673 Acked Write 7 Done Sent Done false
26958 7674 Acked Flush 0 Done Sent Done false
26959 7675 Acked Read 5 Done Sent Done false
26960 7676 Acked Write 10 Done Sent Done false
26961 7677 Acked Write 2 Done Sent Done false
26962 7678 Acked Read 1 Done Sent Done false
26963 7679 Acked Read 10 Done Sent Done false
26964 7680 Acked Write 9 Done Sent Done false
26965 7681 Acked Read 4 Done Sent Done false
26966 7682 Acked Flush 0 Done Sent Done false
26967 7683 Acked Read 3 Done Sent Done false
26968 7684 Acked Flush 0 Done Sent Done false
26969 7685 Acked Read 6 Done Sent Done false
26970 7686 Acked Write 7 Done Sent Done false
26971 7687 Acked Flush 0 Done Sent Done false
26972 7688 Acked Flush 0 Done Sent Done false
26973 7689 Acked Write 2 Done Sent Done false
26974 7690 Acked Write 8 Done Sent Done false
26975 7691 Acked Read 9 Done Sent Done false
26976 7692 Acked Read 3 Done Sent Done false
26977 7693 Acked Read 10 Done Sent Done false
26978 7694 Acked Read 3 Done Sent Done false
26979 7695 Acked Flush 0 Done Sent Done false
26980 7696 Acked Write 4 Done Sent Done false
26981 7697 Acked Read 5 Done Sent Done false
26982 7698 Acked Read 9 Done Sent Done false
26983 7699 Acked Write 3 Done Sent Done false
26984 7700 Acked Write 1 Done Sent Done false
26985 7701 Acked Write 1 Done Sent Done false
26986 7702 Acked Read 6 Done Sent Done false
26987 7703 Acked Read 6 Done Sent Done false
26988 7704 Acked Read 10 Done Sent Done false
26989 7705 Acked Write 10 Done Sent Done false
26990 7706 Acked Read 5 Done Sent Done false
26991 7707 Acked Write 9 Done Sent Done false
26992 7708 Acked Read 1 Done Sent Done false
26993 7709 Acked Read 10 Done Sent Done false
26994 7710 Acked Read 1 Done Sent Done false
26995 7711 Acked Write 2 Done Sent Done false
26996 7712 Acked Write 5 Done Sent Done false
26997 7713 Acked Read 10 Done Sent Done false
26998 7714 Acked Write 4 Done Sent Done false
26999 7715 Acked Write 6 Done Sent Done false
27000 7716 Acked Read 8 Done Sent Done false
27001 7717 Acked Write 3 Done Sent Done false
27002 7718 Acked Flush 0 Done Sent Done false
27003 7719 Acked Write 2 Done Sent Done false
27004 7720 Acked Write 3 Done Sent Done false
27005 7721 Acked Write 8 Done Sent Done false
27006 7722 Acked Read 3 Done Sent Done false
27007 7723 Acked Read 2 Done Sent Done false
27008 7724 Acked Read 2 Done Sent Done false
27009 7725 Acked Read 9 Done Sent Done false
27010 7726 Acked Flush 0 Done Sent Done false
27011 7727 Acked Write 2 Done Sent Done false
27012 7728 Acked Read 5 Done Sent Done false
27013 7729 Acked Read 5 Done Sent Done false
27014 7730 Acked Write 3 Done Sent Done false
27015 7731 Acked Write 8 Done Sent Done false
27016 7732 Acked Flush 0 Done Sent Done false
27017 7733 Acked Read 8 Done Sent Done false
27018 7734 Acked Read 10 Done Sent Done false
27019 7735 Acked Read 8 Done Sent Done false
27020 7736 Acked Flush 0 Done Sent Done false
27021 7737 Acked Read 8 Done Sent Done false
27022 7738 Acked Write 3 Done Sent Done false
27023 7739 Acked Write 1 Done Sent Done false
27024 7740 Acked Write 3 Done Sent Done false
27025 7741 Acked Write 1 Done Sent Done false
27026 7742 Acked Write 7 Done Sent Done false
27027 7743 Acked Write 8 Done Sent Done false
27028 7744 Acked Read 2 Done Sent Done false
27029 7745 Acked Write 10 Done Sent Done false
27030 7746 Acked Read 4 Done Sent Done false
27031 7747 Acked Read 3 Done Sent Done false
27032 7748 Acked Write 2 Done Sent Done false
27033 7749 Acked Write 8 Done Sent Done false
27034 7750 Acked Read 5 Done Sent Done false
27035 7751 Acked Read 10 Done Sent Done false
27036 7752 Acked Write 5 Done Sent Done false
27037 7753 Acked Read 3 Done Sent Done false
27038 7754 Acked Write 1 Done Sent Done false
27039 7755 Acked Read 6 Done Sent Done false
27040 7756 Acked Read 6 Done Sent Done false
27041 7757 Acked Write 7 Done Sent Done false
27042 7758 Acked Read 2 Done Sent Done false
27043 7759 Acked Read 5 Done Sent Done false
27044 7760 Acked Flush 0 Done Sent Done false
27045 7761 Acked Read 5 Done Sent Done false
27046 7762 Acked Write 10 Done Sent Done false
27047 7763 Acked Read 3 Done Sent Done false
27048 7764 Acked Read 1 Done Sent Done false
27049 7765 Acked Write 10 Done Sent Done false
27050 7766 Acked Read 4 Done Sent Done false
27051 7767 Acked Read 6 Done Sent Done false
27052 7768 Acked Write 8 Done Sent Done false
27053 7769 Acked Write 6 Done Sent Done false
27054 7770 Acked Write 7 Done Sent Done false
27055 7771 Acked Write 9 Done Sent Done false
27056 7772 Acked Read 2 Done Sent Done false
27057 7773 Acked Write 2 Done Sent Done false
27058 7774 Acked Read 4 Done Sent Done false
27059 7775 Acked Read 6 Done Sent Done false
27060 7776 Acked Write 9 Done Sent Done false
27061 7777 Acked Write 5 Done Sent Done false
27062 7778 Acked Read 10 Done Sent Done false
27063 7779 Acked Read 4 Done Sent Done false
27064 7780 Acked Write 10 Done Sent Done false
27065 7781 Acked Write 1 Done Sent Done false
27066 7782 Acked Read 9 Done Sent Done false
27067 7783 Acked Read 6 Done Sent Done false
27068 7784 Acked Read 6 Done Sent Done false
27069 7785 Acked Read 3 Done Sent Done false
27070 7786 Acked Flush 0 Done Sent Done false
27071 7787 Acked Write 1 Done Sent Done false
27072 7788 Acked Write 8 Done Sent Done false
27073 7789 Acked Write 5 Done Sent Done false
27074 7790 Acked Read 1 Done Sent Done false
27075 7791 Acked Write 3 Done Sent Done false
27076 7792 Acked Read 9 Done Sent Done false
27077 7793 Acked Write 8 Done Sent Done false
27078 7794 Acked Flush 0 Done Sent Done false
27079 7795 Acked Read 2 Done Sent Done false
27080 7796 Acked Read 6 Done Sent Done false
27081 7797 Acked Read 1 Done Sent Done false
27082 7798 Acked Read 3 Done Sent Done false
27083 7799 Acked Read 2 Done Sent Done false
27084 7800 Acked Read 4 Done Sent Done false
27085 7801 Acked Write 5 Done Sent Done false
27086 7802 Acked Write 5 Done Sent Done false
27087 7803 Acked Flush 0 Done Sent Done false
27088 7804 Acked Read 2 Done Sent Done false
27089 7805 Acked Read 2 Done Sent Done false
27090 7806 Acked Write 3 Done Sent Done false
27091 7807 Acked Write 8 Done Sent Done false
27092 7808 Acked Write 4 Done Sent Done false
27093 7809 Acked Write 9 Done Sent Done false
27094 7810 Acked Flush 0 Done Sent Done false
27095 7811 Acked Flush 0 Done Sent Done false
27096 7812 Acked Read 10 Done Sent Done false
27097 7813 Acked Read 1 Done Sent Done false
27098 7814 Acked Read 3 Done Sent Done false
27099 7815 Acked Flush 0 Done Sent Done false
27100 7816 Acked Read 6 Done Sent Done false
27101 7817 Acked Flush 0 Done Sent Done false
27102 7818 Acked Write 8 Done Sent Done false
27103 7819 Acked Flush 0 Done Sent Done false
27104 7820 Acked Flush 0 Done Sent Done false
27105 7821 Acked Read 6 Done Sent Done false
27106 7822 Acked Read 2 Done Sent Done false
27107 7823 Acked Write 7 Done Sent Done false
27108 7824 Acked Write 10 Done Sent Done false
27109 7825 Acked Read 6 Done Sent Done false
27110 7826 Acked Read 7 Done Sent Done false
27111 7827 Acked Read 1 Done Sent Done false
27112 7828 Acked Read 3 Done Sent Done false
27113 7829 Acked Flush 0 Done Sent Done false
27114 7830 Acked Read 6 Done Sent Done false
27115 7831 Acked Read 2 Done Sent Done false
27116 7832 Acked Read 5 Done Sent Done false
27117 7833 Acked Write 6 Done Sent Done false
27118 7834 Acked Read 1 Done Sent Done false
27119 7835 Acked Write 4 Done Sent Done false
27120 7836 Acked Read 3 Done Sent Done false
27121 7837 Acked Read 3 Done Sent Done false
27122 7838 Acked Write 2 Done Sent Done false
27123 7839 Acked Write 5 Done Sent Done false
27124 7840 Acked Read 1 Done Sent Done false
27125 7841 Acked Write 3 Done Sent Done false
27126 7842 Acked Write 3 Done Sent Done false
27127 7843 Acked Read 10 Done Sent Done false
27128 7844 Acked Write 5 Done Sent Done false
27129 7845 Acked Flush 0 Done Sent Done false
27130 7846 Acked Read 1 Done Sent Done false
27131 7847 Acked Write 6 Done Sent Done false
27132 7848 Acked Read 2 Done Sent Done false
27133 7849 Acked Write 4 Done Sent Done false
27134 7850 Acked Write 1 Done Sent Done false
27135 7851 Acked Read 9 Done Sent Done false
27136 7852 Acked Write 1 Done Sent Done false
27137 7853 Acked Read 2 Done Sent Done false
27138 7854 Acked Flush 0 Done Sent Done false
27139 7855 Acked Read 7 Done Sent Done false
27140 7856 Acked Write 9 Done Sent Done false
27141 7857 Acked Write 10 Done Sent Done false
27142 7858 Acked Write 7 Done Sent Done false
27143 7859 Acked Read 5 Done Sent Done false
27144 7860 Acked Read 8 Done Sent Done false
27145 7861 Acked Read 8 Done Sent Done false
27146 7862 Acked Read 2 Done Sent Done false
27147 7863 Acked Read 3 Done Sent Done false
27148 7864 Acked Read 1 Done Sent Done false
27149 7865 Acked Read 2 Done Sent Done false
27150 7866 Acked Read 1 Done Sent Done false
27151 7867 Acked Write 2 Done Sent Done false
27152 7868 Acked Write 3 Done Sent Done false
27153 7869 Acked Read 6 Done Sent Done false
27154 7870 Acked Read 8 Done Sent Done false
27155 7871 Acked Write 5 Done Sent Done false
27156 7872 Acked Flush 0 Done Sent Done false
27157 7873 Acked Flush 0 Done Sent Done false
27158 7874 Acked Write 4 Done Sent Done false
27159 7875 Acked Read 9 Done Sent Done false
27160 7876 Acked Read 8 Done Sent Done false
27161 7877 Acked Read 7 Done Sent Done false
27162 7878 Acked Read 1 Done Sent Done false
27163 7879 Acked Write 2 Done Sent Done false
27164 7880 Acked Write 10 Done Sent Done false
27165 7881 Acked Read 3 Done Sent Done false
27166 7882 Acked Read 9 Done Sent Done false
27167 7883 Acked Write 3 Done Sent Done false
27168 7884 Acked Read 10 Done Sent Done false
27169 7885 Acked Read 8 Done Sent Done false
27170 7886 Acked Read 9 Done Sent Done false
27171 7887 Acked Read 2 Done Sent Done false
27172 7888 Acked Read 9 Done Sent Done false
27173 7889 Acked Read 9 Done Sent Done false
27174 7890 Acked Read 5 Done Sent Done false
27175 7891 Acked Read 4 Done Sent Done false
27176 7892 Acked Read 6 Done Sent Done false
27177 7893 Acked Flush 0 Done Sent Done false
27178 7894 Acked Flush 0 Done Sent Done false
27179 7895 Acked Read 9 Done Sent Done false
27180 7896 Acked Read 3 Done Sent Done false
27181 7897 Acked Read 1 Done Sent Done false
27182 7898 Acked Read 9 Done Sent Done false
27183 7899 Acked Write 5 Done Sent Done false
27184 7900 Acked Flush 0 Done Sent Done false
27185 7901 Acked Write 9 Done Sent Done false
27186 7902 Acked Write 6 Done Sent Done false
27187 7903 Acked Flush 0 Done Sent Done false
27188 7904 Acked Read 9 Done Sent Done false
27189 7905 Acked Read 7 Done Sent Done false
27190 7906 Acked Read 8 Done Sent Done false
27191 7907 Acked Read 4 Done Sent Done false
27192 7908 Acked Read 7 Done Sent Done false
27193 7909 Acked Read 6 Done Sent Done false
27194 7910 Acked Read 2 Done Sent Done false
27195 7911 Acked Write 10 Done Sent Done false
27196 7912 Acked Read 9 Done Sent Done false
27197 7913 Acked Write 2 Done Sent Done false
27198 7914 Acked Read 8 Done Sent Done false
27199 7915 Acked Write 2 Done Sent Done false
27200 7916 Acked Read 10 Done Sent Done false
27201 7917 Acked Read 8 Done Sent Done false
27202 7918 Acked Write 9 Done Sent Done false
27203 7919 Acked Read 8 Done Sent Done false
27204 7920 Acked Write 1 Done Sent Done false
27205 7921 Acked Write 8 Done Sent Done false
27206 7922 Acked Write 9 Done Sent Done false
27207 7923 Acked Write 10 Done Sent Done false
27208 7924 Acked Read 10 Done Sent Done false
27209 7925 Acked Read 4 Done Sent Done false
27210 7926 Acked Read 5 Done Sent Done false
27211 7927 Acked Read 8 Done Sent Done false
27212 7928 Acked Write 8 Done Sent Done false
27213 7929 Acked Write 2 Done Sent Done false
27214 7930 Acked Read 9 Done Sent Done false
27215 7931 Acked Read 4 Done Sent Done false
27216 7932 Acked Write 7 Done Sent Done false
27217 7933 Acked Write 10 Done Sent Done false
27218 7934 Acked Read 3 Done Sent Done false
27219 7935 Acked Flush 0 Done Sent Done false
27220 7936 Acked Flush 0 Done Sent Done false
27221 7937 Acked Flush 0 Done Sent Done false
27222 7938 Acked Flush 0 Done Sent Done false
27223 7939 Acked Write 5 Done Sent Done false
27224 7940 Acked Write 7 Done Sent Done false
27225 7941 Acked Read 6 Done Sent Done false
27226 7942 Acked Write 7 Done Sent Done false
27227 STATES DS:0 DS:1 DS:2 TOTAL
27228 Sent 0 301 0 301
27229 Done 301 0 301 602
27230 Skipped 0 0 0 0
27231 Error 0 0 0 0
27232 Last Flush: Some(JobId(7938)) Some(JobId(7641)) Some(JobId(7938))
27233 Downstairs last five completed: 7641 7640 7639 7638 7637
27234 Upstairs last five completed: 7942 7941 7940 7939 7938
27235 CLIENT: Up:0 ds:301 act:2
272362025-03-17T22:40:31.171ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272372025-03-17T22:40:32.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272382025-03-17T22:40:32.869ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272392025-03-17T22:40:33.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272402025-03-17T22:40:34.424ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272412025-03-17T22:40:34.424ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37299) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272422025-03-17T22:40:34.425ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272432025-03-17T22:40:34.425ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272442025-03-17T22:40:34.425ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272452025-03-17T22:40:34.425ZINFOcrucible: send last flush ID to this DS: Some(JobId(7641)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272462025-03-17T22:40:34.425ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272472025-03-17T22:40:34.425ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7641)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272482025-03-17T22:40:34.426ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7641)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27249 ----------------------------------------------------------------
27250 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
27251 States: Active Active Active
27252 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27253 7939 Acked Write 5 Done Done Done true
27254 7940 Acked Write 7 Done Done Done true
27255 7941 Acked Read 6 Done Done Done true
27256 7942 Acked Write 7 Done Done Done true
27257 STATES DS:0 DS:1 DS:2 TOTAL
27258 Sent 0 0 0 0
27259 Done 4 4 4 12
27260 Skipped 0 0 0 0
27261 Error 0 0 0 0
27262 Last Flush: Some(JobId(7938)) Some(JobId(7938)) Some(JobId(7938))
27263 Downstairs last five completed: 7938 7937 7936 7935 7934
27264 Upstairs last five completed: 7942 7941 7940 7939 7938
27265 CLIENT: Up:0 ds:4 act:3
27266 ----------------------------------------------------------------
27267 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27268 Downstairs last five completed: 7943 7942 7941 7940 7939
27269 Upstairs last five completed: 7943 7942 7941 7940 7939
27270 CLIENT: Up:0 ds:0 act:3
27271 Replay: All jobs finished
27272 Main task has work to do, go find it
27273 got dsc Stop(1)
27274 stop 1
27275 Mar 17 22:40:38.674 INFO request completed, latency_us[8820] Got stop action so:false kr:false
27276 : 330, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a74a52ef-51ab-40e8-bc18-b19d4b551e1d, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
272772025-03-17T22:40:38.675ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272782025-03-17T22:40:38.675ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
272792025-03-17T22:40:38.675ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27280 Mar 17 22:40:38.675 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:40:38.675617199Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
27281 , method: GET, req_id: ddda500f-075f-441d-a3e1-b7d97f878902, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
27282 [8820] Exited with: ExitStatus(unix_wait_status(9))
27283 [8820][1] reports Exit
27284 Mar 17 22:40:42.677 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0782f27f-2427-4826-858d-6f04a23119c2, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
27285 001/300 Read block 4927 len 24576
27286 002/300 Write block 3946 len 20480 data: 8 7 4 3 3
27287 003/300 Write block 3931 len 8192 data: 7 8
27288 004/300 Read block 3142 len 16384
27289 005/300 Read block 3998 len 8192
27290 006/300 Write block 4652 len 16384 data: 4 5 5 5
27291 007/300 Write block 2876 len 28672 data: 5 6 5 7 5 5 5
27292 008/300 Read block 3461 len 4096
27293 009/300 Write block 726 len 36864 data: 8 8 8 8 7 8 7 8 7
27294 010/300 Flush
27295 011/300 Flush
27296 012/300 Write block 1472 len 12288 data: 3 3 5
27297 013/300 Read block 262 len 20480
27298 014/300 Read block 4790 len 36864
27299 015/300 Write block 2448 len 20480 data: 4 4 3 3 5
27300 016/300 Read block 2060 len 28672
27301 017/300 Write block 3692 len 36864 data: 4 4 4 4 4 4 3 3 2
27302 018/300 Read block 1967 len 20480
27303 019/300 Flush
27304 020/300 Read block 2489 len 20480
27305 021/300 Write block 3848 len 32768 data: 4 3 4 3 3 3 4 5
27306 022/300 Read block 776 len 36864
27307 023/300 Read block 252 len 8192
27308 024/300 Read block 1093 len 8192
27309 025/300 Read block 802 len 32768
27310 026/300 Read block 449 len 40960
27311 027/300 Write block 4066 len 40960 data: 4 5 5 4 4 4 5 4 5 5
27312 028/300 Write block 2061 len 8192 data: 3 3
27313 029/300 Read block 4329 len 36864
27314 030/300 Write block 3021 len 32768 data: 2 4 4 4 4 4 4 4
27315 031/300 Read block 1737 len 36864
27316 032/300 Read block 638 len 4096
27317 033/300 Read block 3761 len 24576
27318 034/300 Read block 113 len 32768
27319 035/300 Flush
27320 036/300 Read block 608 len 12288
27321 037/300 Write block 1885 len 8192 data: 5 5
27322 038/300 Read block 1166 len 32768
27323 039/300 Write block 3791 len 20480 data: 3 6 6 5 4
27324 040/300 Read block 1684 len 28672
27325 041/300 Read block 2880 len 20480
27326 042/300 Read block 703 len 4096
27327 043/300 Read block 2669 len 8192
27328 044/300 Read block 624 len 12288
27329 045/300 Flush
27330 046/300 Read block 1038 len 40960
27331 047/300 Read block 1253 len 36864
27332 048/300 Read block 1622 len 40960
27333 049/300 Write block 2995 len 16384 data: 5 4 5 5
27334 050/300 Write block 860 len 40960 data: 3 3 3 3 4 4 5 4 3 4
27335 051/300 Read block 4269 len 32768
27336 052/300 Write block 206 len 16384 data: 3 4 4 4
27337 053/300 Read block 3550 len 12288
27338 054/300 Flush
27339 055/300 Write block 2267 len 8192 data: 6 5
27340 056/300 Read block 3079 len 20480
27341 057/300 Write block 2617 len 28672 data: 6 6 5 6 6 4 4
27342 058/300 Write block 1277 len 4096 data: 7
27343 059/300 Read block 3747 len 16384
27344 060/300 Read block 1802 len 32768
27345 061/300 Read block 825 len 36864
27346 062/300 Flush
27347 063/300 Read block 1443 len 8192
27348 064/300 Write block 3844 len 12288 data: 6 7 5
27349 065/300 Write block 1650 len 24576 data: 5 4 4 4 4 6
27350 066/300 Write block 98 len 40960 data: 5 6 6 6 6 7 7 6 6 4
27351 067/300 Read block 4677 len 32768
27352 068/300 Read block 1604 len 12288
27353 069/300 Write block 4615 len 24576 data: 5 5 4 3 3 3
27354 070/300 Read block 1750 len 16384
27355 071/300 Write block 1054 len 40960 data: 3 3 3 3 3 3 3 3 2 2
27356 072/300 Write block 77 len 40960 data: 4 4 4 5 6 6 10 8 8 5
27357 073/300 Read block 4429 len 32768
27358 074/300 Write block 2785 len 4096 data: 5
27359 075/300 Read block 408 len 20480
27360 076/300 Write block 3360 len 36864 data: 5 6 7 8 4 4 5 6 6
27361 077/300 Read block 3044 len 28672
27362 078/300 Read block 288 len 28672
27363 079/300 Write block 4910 len 12288 data: 7 7 6
27364 080/300 Write block 2988 len 28672 data: 7 6 8 7 8 7 6
27365 081/300 Write block 266 len 12288 data: 7 8 7
27366 082/300 Flush
27367 083/300 Write block 2008 len 16384 data: 4 5 4 5
27368 084/300 Write block 3875 len 40960 data: 10 9 9 9 8 7 7 9 7 6
27369 085/300 Read block 416 len 32768
27370 086/300 Read block 1956 len 32768
27371 087/300 Write block 3266 len 20480 data: 5 5 5 5 3
27372 088/300 Read block 3057 len 12288
27373 089/300 Read block 4156 len 32768
27374 090/300 Read block 850 len 8192
27375 091/300 Write block 578 len 8192 data: 4 5
27376 092/300 Read block 3196 len 32768
27377 093/300 Write block 4753 len 20480 data: 2 2 3 3 4
27378 094/300 Write block 3683 len 8192 data: 5 5
27379 095/300 Read block 2110 len 20480
27380 096/300 Write block 3607 len 4096 data: 3
27381 097/300 Read block 3471 len 20480
27382 098/300 Read block 4466 len 24576
27383 099/300 Read block 1669 len 4096
27384 100/300 Read block 1980 len 32768
27385 101/300 Flush
27386 102/300 Read block 3372 len 28672
27387 103/300 Write block 698 len 28672 data: 5 6 6 6 7 7 8
27388 104/300 Read block 2683 len 20480
27389 105/300 Read block 3285 len 4096
27390 106/300 Read block 2666 len 36864
27391 107/300 Read block 226 len 12288
27392 108/300 Read block 1589 len 32768
27393 109/300 Read block 1229 len 12288
27394 110/300 Read block 2054 len 20480
27395 111/300 Read block 4111 len 8192
27396 112/300 Read block 4788 len 8192
27397 113/300 Flush
27398 114/300 Read block 2708 len 28672
27399 115/300 Write block 2946 len 12288 data: 8 8 6
27400 116/300 Write block 3998 len 32768 data: 4 6 6 4 4 5 5 4
27401 117/300 Write block 4698 len 40960 data: 6 5 9 9 9 8 5 6 5 3
27402 118/300 Read block 4720 len 20480
27403 119/300 Write block 4897 len 32768 data: 2 3 3 3 5 4 5 6
27404 120/300 Write block 2428 len 20480 data: 4 3 5 5 5
27405 121/300 Write block 3204 len 40960 data: 6 6 5 7 8 9 10 10 12 11
27406 122/300 Flush
27407 123/300 Read block 644 len 32768
27408 124/300 Read block 2327 len 36864
27409 125/300 Read block 2082 len 4096
27410 126/300 Flush
27411 127/300 Write block 913 len 12288 data: 2 2 2
27412 128/300 Read block 317 len 20480
27413 129/300 Write block 728 len 36864 data: 9 9 8 9 8 9 8 7 7
27414 130/300 Read block 892 len 28672
27415 131/300 Read block 1859 len 24576
27416 132/300 Write block 1858 len 36864 data: 5 4 4 5 5 5 5 5 6
27417 133/300 Read block 289 len 28672
27418 134/300 Flush
27419 135/300 Read block 4993 len 28672
27420 136/300 Write block 454 len 40960 data: 5 5 5 5 5 5 6 4 4 3
27421 137/300 Read block 3026 len 32768
27422 138/300 Write block 2164 len 28672 data: 4 4 4 3 3 3 4
27423 139/300 Write block 2548 len 4096 data: 6
27424 140/300 Read block 1208 len 4096
27425 141/300 Read block 2553 len 16384
27426 142/300 Read block 2539 len 32768
27427 143/300 Flush
27428 144/300 Write block 499 len 24576 data: 3 3 4 3 2 2
27429 145/300 Read block 3193 len 8192
27430 146/300 Read block 1666 len 8192
27431 147/300 Write block 3686 len 36864 data: 4 5 5 5 6 5 5 5 5
27432 148/300 Write block 3554 len 28672 data: 6 6 6 4 5 6 6
27433 149/300 Write block 4644 len 12288 data: 5 5 6
27434 150/300 Write block 1255 len 8192 data: 4 4
27435 151/300 Read block 2716 len 40960
27436 152/300 Flush
27437 153/300 Read block 1279 len 40960
27438 154/300 Write block 2717 len 32768 data: 4 5 5 5 5 6 3 3
27439 155/300 Write block 3204 len 32768 data: 7 7 6 8 9 10 11 11
27440 156/300 Write block 2751 len 12288 data: 7 9 9
27441 157/300 Write block 3109 len 40960 data: 5 4 3 3 3 3 3 3 3 3
27442 158/300 Flush
27443 159/300 Read block 2831 len 32768
27444 160/300 Write block 2686 len 24576 data: 4 5 5 3 2 2
27445 161/300 Read block 3927 len 32768
27446 162/300 Write block 1628 len 4096 data: 5
27447 163/300 Write block 3181 len 4096 data: 7
27448 164/300 Flush
27449 165/300 Read block 4901 len 8192
27450 166/300 Write block 1637 len 36864 data: 6 6 6 6 5 4 3 7 7
27451 167/300 Flush
27452 168/300 Read block 3918 len 16384
27453 169/300 Read block 4911 len 40960
27454 170/300 Read block 1275 len 12288
27455 171/300 Write block 136 len 12288 data: 6 5 6
27456 172/300 Read block 701 len 4096
27457 173/300 Write block 2102 len 36864 data: 6 5 5 4 4 3 3 4 4
27458 174/300 Write block 889 len 32768 data: 2 2 3 3 3 4 4 3
27459 175/300 Read block 4447 len 4096
27460 176/300 Write block 2067 len 12288 data: 4 5 6
27461 177/300 Write block 975 len 16384 data: 6 5 5 4
27462 178/300 Read block 551 len 4096
27463 179/300 Read block 2459 len 24576
27464 180/300 Write block 4049 len 20480 data: 3 3 3 3 3
27465 181/300 Write block 4833 len 24576 data: 5 5 5 4 4 5
27466 182/300 Write block 3251 len 36864 data: 5 5 4 3 3 4 4 3 3
27467 183/300 Write block 380 len 36864 data: 4 5 4 5 8 10 8 10 9
27468 184/300 Read block 3339 len 8192
27469 185/300 Flush
27470 186/300 Read block 1284 len 24576
27471 187/300 Read block 2915 len 36864
27472 188/300 Read block 3675 len 12288
27473 189/300 Read block 1014 len 36864
27474 190/300 Read block 1161 len 12288
27475 191/300 Write block 871 len 24576 data: 4 5 4 4 4 5
27476 192/300 Write block 2057 len 4096 data: 4
27477 193/300 Write block 1925 len 24576 data: 5 6 5 3 3 3
27478 194/300 Write block 3284 len 36864 data: 6 6 6 6 6 6 7 8 7
27479 195/300 Read block 2320 len 36864
27480 196/300 Write block 4527 len 28672 data: 6 7 7 7 7 6 7
27481 197/300 Read block 4075 len 20480
27482 198/300 Read block 1877 len 16384
27483 199/300 Write block 2351 len 40960 data: 6 3 4 3 3 6 5 6 5 4
27484 200/300 Write block 235 len 20480 data: 6 5 6 6 4
27485 201/300 Write block 2855 len 8192 data: 3 3
27486 202/300 Write block 3565 len 40960 data: 6 6 5 6 8 9 8 8 4 6
27487 203/300 Write block 982 len 40960 data: 7 7 6 5 3 3 4 4 4 5
27488 204/300 Read block 2721 len 12288
27489 205/300 Read block 2311 len 24576
27490 206/300 Write block 2135 len 16384 data: 7 7 6 5
27491 207/300 Read block 339 len 24576
27492 208/300 Write block 4700 len 28672 data: 10 10 10 9 6 7 6
27493 209/300 Read block 1668 len 28672
27494 210/300 Write block 2088 len 8192 data: 7 8
27495 211/300 Read block 3105 len 36864
27496 212/300 Read block 1408 len 40960
27497 213/300 Read block 1749 len 24576
27498 214/300 Read block 2203 len 12288
27499 215/300 Read block 257 len 20480
27500 216/300 Read block 2950 len 8192
27501 217/300 Read block 1385 len 8192
27502 218/300 Write block 2923 len 32768 data: 6 6 4 3 2 2 3 4
27503 219/300 Write block 3351 len 4096 data: 4
27504 220/300 Read block 157 len 32768
27505 221/300 Write block 4648 len 16384 data: 5 4 4 4
27506 222/300 Read block 311 len 12288
27507 223/300 Write block 1401 len 24576 data: 6 4 4 6 7 4
27508 224/300 Read block 2407 len 24576
27509 225/300 Write block 3919 len 32768 data: 6 4 5 5 4 4 4 5
27510 226/300 Read block 3854 len 16384
27511 227/300 Read block 2823 len 8192
27512 228/300 Read block 320 len 32768
27513 229/300 Flush
27514 230/300 Write block 806 len 40960 data: 5 5 4 3 4 4 5 6 5 6
27515 231/300 Read block 2323 len 28672
27516 232/300 Read block 4828 len 28672
27517 233/300 Read block 575 len 20480
27518 234/300 Read block 4963 len 32768
27519 235/300 Read block 4127 len 4096
27520 236/300 Write block 2468 len 36864 data: 8 7 8 8 8 8 9 8 6
27521 237/300 Read block 2399 len 16384
27522 238/300 Flush
27523 239/300 Read block 778 len 24576
27524 240/300 Read block 1684 len 24576
27525 241/300 Read block 1714 len 8192
27526 242/300 Read block 1754 len 32768
27527 243/300 Read block 4891 len 28672
27528 244/300 Flush
27529 245/300 Read block 1184 len 36864
27530 246/300 Write block 1734 len 12288 data: 6 5 6
27531 247/300 Write block 1246 len 40960 data: 5 4 3 3 3 4 4 5 5 5
27532 248/300 Read block 4083 len 20480
27533 249/300 Read block 4868 len 40960
27534 250/300 Write block 4134 len 36864 data: 6 7 6 4 6 6 6 6 8
27535 251/300 Write block 3017 len 36864 data: 6 4 3 2 3 5 5 5 5
27536 252/300 Read block 394 len 20480
27537 253/300 Read block 2605 len 20480
27538 254/300 Write block 2112 len 36864 data: 4 4 5 6 6 5 4 5 6
27539 255/300 Write block 2391 len 12288 data: 2 2 2
27540 256/300 Read block 3740 len 8192
27541 257/300 Write block 2718 len 16384 data: 6 6 6 6
27542 258/300 Read block 2066 len 16384
27543 259/300 Write block 4070 len 8192 data: 5 5
27544 260/300 Read block 2318 len 12288
27545 261/300 Read block 2817 len 16384
27546 262/300 Read block 1759 len 32768
27547 263/300 Write block 4459 len 16384 data: 5 6 5 5
27548 264/300 Write block 2255 len 4096 data: 5
27549 265/300 Write block 1959 len 32768 data: 7 7 6 8 6 4 4 4
27550 266/300 Flush
27551 267/300 Write block 3736 len 8192 data: 5 5
27552 268/300 Read block 3737 len 40960
27553 269/300 Write block 3893 len 16384 data: 6 5 5 7
27554 270/300 Read block 1487 len 16384
27555 271/300 Read block 4832 len 8192
27556 272/300 Flush
27557 273/300 Read block 2998 len 4096
27558 274/300 Write block 2518 len 16384 data: 6 6 5 5
27559 275/300 Flush
27560 276/300 Read block 1707 len 12288
27561 277/300 Read block 4063 len 16384
27562 278/300 Read block 4703 len 16384
27563 279/300 Write block 3925 len 20480 data: 5 6 7 6 6
27564 280/300 Read block 2527 len 28672
27565 281/300 Read block 1772 len 20480
27566 282/300 Write block 1262 len 24576 data: 8 6 5 6 5 6
27567 283/300 Write block 616 len 20480 data: 7 7 6 5 5
27568 284/300 Write block 4408 len 4096 data: 5
27569 285/300 Write block 1725 len 16384 data: 6 6 5 5
27570 286/300 Write block 2574 len 12288 data: 4 5 6
27571 287/300 Read block 2099 len 32768
27572 288/300 Write block 4843 len 12288 data: 6 7 6
27573 289/300 Flush
27574 290/300 Flush
27575 291/300 Write block 4156 len 36864 data: 5 6 6 6 4 3 4 5 5
27576 292/300 Read block 1019 len 40960
27577 293/300 Read block 1158 len 8192
27578 294/300 Write block 4511 len 40960 data: 3 4 6 6 5 6 5 6 7 7
27579 295/300 Read block 4916 len 16384
27580 296/300 Write block 3352 len 12288 data: 4 3 4
27581 297/300 Read block 1847 len 20480
27582 298/300 Read block 1926 len 24576
27583 299/300 Read block 1141 len 36864
27584 300/300 Write block 2652 len 4096 data: 3
27585 301/300 Write block 2851 len 12288 data: 3 3 4
27586 Main task has work to do, go find it
27587 got dsc Start(1)
27588 start 1
27589 Mar 17 22:40:42.926 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/1, method: POST, req_id: 34531472-242c-4919-8bd1-807d13e8fcbe, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
27590 Replay: started 1, returned:Ok(())
27591 ----------------------------------------------------------------
27592 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27593 States: Active Offline Active
27594 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27595 7944 Acked Read 6 Done Sent Done false
27596 7945 Acked Write 5 Done Sent Done false
27597 7946 Acked Write 2 Done Sent Done false
27598 7947 Acked Read 4 Done Sent Done false
27599 7948 Acked Read 2 Done Sent Done false
27600 7949 Acked Write 4 Done Sent Done false
27601 7950 Acked Write 7 Done Sent Done false
27602 7951 Acked Read 1 Done Sent Done false
27603 7952 Acked Write 9 Done Sent Done false
27604 7953 Acked Flush 0 Done Sent Done false
27605 7954 Acked Flush 0 Done Sent Done false
27606 7955 Acked Write 3 Done Sent Done false
27607 7956 Acked Read 5 Done Sent Done false
27608 7957 Acked Read 9 Done Sent Done false
27609 7958 Acked Write 5 Done Sent Done false
27610 7959 Acked Read 7 Done Sent Done false
27611 7960 Acked Write 9 Done Sent Done false
27612 7961 Acked Read 5 Done Sent Done false
27613 7962 Acked Flush 0 Done Sent Done false
27614 7963 Acked Read 5 Done Sent Done false
27615 7964 Acked Write 8 Done Sent Done false
27616 7965 Acked Read 9 Done Sent Done false
27617 7966 Acked Read 2 Done Sent Done false
27618 7967 Acked Read 2 Done Sent Done false
27619 7968 Acked Read 8 Done Sent Done false
27620 7969 Acked Read 10 Done Sent Done false
27621 7970 Acked Write 10 Done Sent Done false
27622 7971 Acked Write 2 Done Sent Done false
27623 7972 Acked Read 9 Done Sent Done false
27624 7973 Acked Write 8 Done Sent Done false
27625 7974 Acked Read 9 Done Sent Done false
27626 7975 Acked Read 1 Done Sent Done false
27627 7976 Acked Read 6 Done Sent Done false
27628 7977 Acked Read 8 Done Sent Done false
27629 7978 Acked Flush 0 Done Sent Done false
27630 7979 Acked Read 3 Done Sent Done false
27631 7980 Acked Write 2 Done Sent Done false
27632 7981 Acked Read 8 Done Sent Done false
27633 7982 Acked Write 5 Done Sent Done false
27634 7983 Acked Read 7 Done Sent Done false
27635 7984 Acked Read 5 Done Sent Done false
27636 7985 Acked Read 1 Done Sent Done false
27637 7986 Acked Read 2 Done Sent Done false
27638 7987 Acked Read 3 Done Sent Done false
27639 7988 Acked Flush 0 Done Sent Done false
27640 7989 Acked Read 10 Done Sent Done false
27641 7990 Acked Read 9 Done Sent Done false
27642 7991 Acked Read 10 Done Sent Done false
27643 7992 Acked Write 4 Done Sent Done false
27644 7993 Acked Write 10 Done Sent Done false
27645 7994 Acked Read 8 Done Sent Done false
27646 7995 Acked Write 4 Done Sent Done false
27647 7996 Acked Read 3 Done Sent Done false
27648 7997 Acked Flush 0 Done Sent Done false
27649 7998 Acked Write 2 Done Sent Done false
27650 7999 Acked Read 5 Done Sent Done false
27651 8000 Acked Write 7 Done Sent Done false
27652 8001 Acked Write 1 Done Sent Done false
27653 8002 Acked Read 4 Done Sent Done false
27654 8003 Acked Read 8 Done Sent Done false
27655 8004 Acked Read 9 Done Sent Done false
27656 8005 Acked Flush 0 Done Sent Done false
27657 8006 Acked Read 2 Done Sent Done false
27658 8007 Acked Write 3 Done Sent Done false
27659 8008 Acked Write 6 Done Sent Done false
27660 8009 Acked Write 10 Done Sent Done false
27661 8010 Acked Read 8 Done Sent Done false
27662 8011 Acked Read 3 Done Sent Done false
27663 8012 Acked Write 6 Done Sent Done false
27664 8013 Acked Read 4 Done Sent Done false
27665 8014 Acked Write 10 Done Sent Done false
27666 8015 Acked Write 10 Done Sent Done false
27667 8016 Acked Read 8 Done Sent Done false
27668 8017 Acked Write 1 Done Sent Done false
27669 8018 Acked Read 5 Done Sent Done false
27670 8019 Acked Write 9 Done Sent Done false
27671 8020 Acked Read 7 Done Sent Done false
27672 8021 Acked Read 7 Done Sent Done false
27673 8022 Acked Write 3 Done Sent Done false
27674 8023 Acked Write 7 Done Sent Done false
27675 8024 Acked Write 3 Done Sent Done false
27676 8025 Acked Flush 0 Done Sent Done false
27677 8026 Acked Write 4 Done Sent Done false
27678 8027 Acked Write 10 Done Sent Done false
27679 8028 Acked Read 8 Done Sent Done false
27680 8029 Acked Read 8 Done Sent Done false
27681 8030 Acked Write 5 Done Sent Done false
27682 8031 Acked Read 3 Done Sent Done false
27683 8032 Acked Read 8 Done Sent Done false
27684 8033 Acked Read 2 Done Sent Done false
27685 8034 Acked Write 2 Done Sent Done false
27686 8035 Acked Read 8 Done Sent Done false
27687 8036 Acked Write 5 Done Sent Done false
27688 8037 Acked Write 2 Done Sent Done false
27689 8038 Acked Read 5 Done Sent Done false
27690 8039 Acked Write 1 Done Sent Done false
27691 8040 Acked Read 5 Done Sent Done false
27692 8041 Acked Read 6 Done Sent Done false
27693 8042 Acked Read 1 Done Sent Done false
27694 8043 Acked Read 8 Done Sent Done false
27695 8044 Acked Flush 0 Done Sent Done false
27696 8045 Acked Read 7 Done Sent Done false
27697 8046 Acked Write 7 Done Sent Done false
27698 8047 Acked Read 5 Done Sent Done false
27699 8048 Acked Read 1 Done Sent Done false
27700 8049 Acked Read 9 Done Sent Done false
27701 8050 Acked Read 3 Done Sent Done false
27702 8051 Acked Read 8 Done Sent Done false
27703 8052 Acked Read 3 Done Sent Done false
27704 8053 Acked Read 5 Done Sent Done false
27705 8054 Acked Read 2 Done Sent Done false
27706 8055 Acked Read 2 Done Sent Done false
27707 8056 Acked Flush 0 Done Sent Done false
27708 8057 Acked Read 7 Done Sent Done false
27709 8058 Acked Write 3 Done Sent Done false
27710 8059 Acked Write 8 Done Sent Done false
27711 8060 Acked Write 10 Done Sent Done false
27712 8061 Acked Read 5 Done Sent Done false
27713 8062 Acked Write 8 Done Sent Done false
27714 8063 Acked Write 5 Done Sent Done false
27715 8064 Acked Write 10 Done Sent Done false
27716 8065 Acked Flush 0 Done Sent Done false
27717 8066 Acked Read 8 Done Sent Done false
27718 8067 Acked Read 9 Done Sent Done false
27719 8068 Acked Read 1 Done Sent Done false
27720 8069 Acked Flush 0 Done Sent Done false
27721 8070 Acked Write 3 Done Sent Done false
27722 8071 Acked Read 5 Done Sent Done false
27723 8072 Acked Write 9 Done Sent Done false
27724 8073 Acked Read 7 Done Sent Done false
27725 8074 Acked Read 6 Done Sent Done false
27726 8075 Acked Write 9 Done Sent Done false
27727 8076 Acked Read 7 Done Sent Done false
27728 8077 Acked Flush 0 Done Sent Done false
27729 8078 Acked Read 7 Done Sent Done false
27730 8079 Acked Write 10 Done Sent Done false
27731 8080 Acked Read 8 Done Sent Done false
27732 8081 Acked Write 7 Done Sent Done false
27733 8082 Acked Write 1 Done Sent Done false
27734 8083 Acked Read 1 Done Sent Done false
27735 8084 Acked Read 4 Done Sent Done false
27736 8085 Acked Read 8 Done Sent Done false
27737 8086 Acked Flush 0 Done Sent Done false
27738 8087 Acked Write 6 Done Sent Done false
27739 8088 Acked Read 2 Done Sent Done false
27740 8089 Acked Read 2 Done Sent Done false
27741 8090 Acked Write 9 Done Sent Done false
27742 8091 Acked Write 7 Done Sent Done false
27743 8092 Acked Write 3 Done Sent Done false
27744 8093 Acked Write 2 Done Sent Done false
27745 8094 Acked Read 10 Done Sent Done false
27746 8095 Acked Flush 0 Done Sent Done false
27747 8096 Acked Read 10 Done Sent Done false
27748 8097 Acked Write 8 Done Sent Done false
27749 8098 Acked Write 8 Done Sent Done false
27750 8099 Acked Write 3 Done Sent Done false
27751 8100 Acked Write 10 Done Sent Done false
27752 8101 Acked Flush 0 Done Sent Done false
27753 8102 Acked Read 8 Done Sent Done false
27754 8103 Acked Write 6 Done Sent Done false
27755 8104 Acked Read 8 Done Sent Done false
27756 8105 Acked Write 1 Done Sent Done false
27757 8106 Acked Write 1 Done Sent Done false
27758 8107 Acked Flush 0 Done Sent Done false
27759 8108 Acked Read 2 Done Sent Done false
27760 8109 Acked Write 9 Done Sent Done false
27761 8110 Acked Flush 0 Done Sent Done false
27762 8111 Acked Read 4 Done Sent Done false
27763 8112 Acked Read 10 Done Sent Done false
27764 8113 Acked Read 3 Done Sent Done false
27765 8114 Acked Write 3 Done Sent Done false
27766 8115 Acked Read 1 Done Sent Done false
27767 8116 Acked Write 9 Done Sent Done false
27768 8117 Acked Write 8 Done Sent Done false
27769 8118 Acked Read 1 Done Sent Done false
27770 8119 Acked Write 3 Done Sent Done false
27771 8120 Acked Write 4 Done Sent Done false
27772 8121 Acked Read 1 Done Sent Done false
27773 8122 Acked Read 6 Done Sent Done false
27774 8123 Acked Write 5 Done Sent Done false
27775 8124 Acked Write 6 Done Sent Done false
27776 8125 Acked Write 9 Done Sent Done false
27777 8126 Acked Write 9 Done Sent Done false
27778 8127 Acked Read 2 Done Sent Done false
27779 8128 Acked Flush 0 Done Sent Done false
27780 8129 Acked Read 6 Done Sent Done false
27781 8130 Acked Read 9 Done Sent Done false
27782 8131 Acked Read 3 Done Sent Done false
27783 8132 Acked Read 9 Done Sent Done false
27784 8133 Acked Read 3 Done Sent Done false
27785 8134 Acked Write 6 Done Sent Done false
27786 8135 Acked Write 1 Done Sent Done false
27787 8136 Acked Write 6 Done Sent Done false
27788 8137 Acked Write 9 Done Sent Done false
27789 8138 Acked Read 9 Done Sent Done false
27790 8139 Acked Write 7 Done Sent Done false
27791 8140 Acked Read 5 Done Sent Done false
27792 8141 Acked Read 4 Done Sent Done false
27793 8142 Acked Write 10 Done Sent Done false
27794 8143 Acked Write 5 Done Sent Done false
27795 8144 Acked Write 2 Done Sent Done false
27796 8145 Acked Write 10 Done Sent Done false
27797 8146 Acked Write 10 Done Sent Done false
27798 8147 Acked Read 3 Done Sent Done false
27799 8148 Acked Read 6 Done Sent Done false
27800 8149 Acked Write 4 Done Sent Done false
27801 8150 Acked Read 6 Done Sent Done false
27802 8151 Acked Write 7 Done Sent Done false
27803 8152 Acked Read 7 Done Sent Done false
27804 8153 Acked Write 2 Done Sent Done false
27805 8154 Acked Read 9 Done Sent Done false
27806 8155 Acked Read 10 Done Sent Done false
27807 8156 Acked Read 6 Done Sent Done false
27808 8157 Acked Read 3 Done Sent Done false
27809 8158 Acked Read 5 Done Sent Done false
27810 8159 Acked Read 2 Done Sent Done false
27811 8160 Acked Read 2 Done Sent Done false
27812 8161 Acked Write 8 Done Sent Done false
27813 8162 Acked Write 1 Done Sent Done false
27814 8163 Acked Read 8 Done Sent Done false
27815 8164 Acked Write 4 Done Sent Done false
27816 8165 Acked Read 3 Done Sent Done false
27817 8166 Acked Write 6 Done Sent Done false
27818 8167 Acked Read 6 Done Sent Done false
27819 8168 Acked Write 8 Done Sent Done false
27820 8169 Acked Read 4 Done Sent Done false
27821 8170 Acked Read 2 Done Sent Done false
27822 8171 Acked Read 8 Done Sent Done false
27823 8172 Acked Flush 0 Done Sent Done false
27824 8173 Acked Write 10 Done Sent Done false
27825 8174 Acked Read 7 Done Sent Done false
27826 8175 Acked Read 7 Done Sent Done false
27827 8176 Acked Read 5 Done Sent Done false
27828 8177 Acked Read 8 Done Sent Done false
27829 8178 Acked Read 1 Done Sent Done false
27830 8179 Acked Write 9 Done Sent Done false
27831 8180 Acked Read 4 Done Sent Done false
27832 8181 Acked Flush 0 Done Sent Done false
27833 8182 Acked Read 6 Done Sent Done false
27834 8183 Acked Read 6 Done Sent Done false
27835 8184 Acked Read 2 Done Sent Done false
27836 8185 Acked Read 8 Done Sent Done false
27837 8186 Acked Read 7 Done Sent Done false
27838 8187 Acked Flush 0 Done Sent Done false
27839 8188 Acked Read 9 Done Sent Done false
27840 8189 Acked Write 3 Done Sent Done false
27841 8190 Acked Write 10 Done Sent Done false
27842 8191 Acked Read 5 Done Sent Done false
27843 8192 Acked Read 10 Done Sent Done false
27844 8193 Acked Write 9 Done Sent Done false
27845 8194 Acked Write 9 Done Sent Done false
27846 8195 Acked Read 5 Done Sent Done false
27847 8196 Acked Read 5 Done Sent Done false
27848 8197 Acked Write 9 Done Sent Done false
27849 8198 Acked Write 3 Done Sent Done false
27850 8199 Acked Read 2 Done Sent Done false
27851 8200 Acked Write 4 Done Sent Done false
27852 8201 Acked Read 4 Done Sent Done false
27853 8202 Acked Write 2 Done Sent Done false
27854 8203 Acked Read 3 Done Sent Done false
27855 8204 Acked Read 4 Done Sent Done false
27856 8205 Acked Read 8 Done Sent Done false
27857 8206 Acked Write 4 Done Sent Done false
27858 8207 Acked Write 1 Done Sent Done false
27859 8208 Acked Write 8 Done Sent Done false
27860 8209 Acked Flush 0 Done Sent Done false
27861 8210 Acked Write 2 Done Sent Done false
27862 8211 Acked Read 10 Done Sent Done false
27863 8212 Acked Write 4 Done Sent Done false
27864 8213 Acked Read 4 Done Sent Done false
27865 8214 Acked Read 2 Done Sent Done false
27866 8215 Acked Flush 0 Done Sent Done false
27867 8216 Acked Read 1 Done Sent Done false
27868 8217 Acked Write 4 Done Sent Done false
27869 8218 Acked Flush 0 Done Sent Done false
27870 8219 Acked Read 3 Done Sent Done false
27871 8220 Acked Read 4 Done Sent Done false
27872 8221 Acked Read 4 Done Sent Done false
27873 8222 Acked Write 5 Done Sent Done false
27874 8223 Acked Read 7 Done Sent Done false
27875 8224 Acked Read 5 Done Sent Done false
27876 8225 Acked Write 6 Done Sent Done false
27877 8226 Acked Write 5 Done Sent Done false
27878 8227 Acked Write 1 Done Sent Done false
27879 8228 Acked Write 4 Done Sent Done false
27880 8229 Acked Write 3 Done Sent Done false
27881 8230 Acked Read 8 Done Sent Done false
27882 8231 Acked Write 3 Done Sent Done false
27883 8232 Acked Flush 0 Done Sent Done false
27884 8233 Acked Flush 0 Done Sent Done false
27885 8234 Acked Write 9 Done Sent Done false
27886 8235 Acked Read 10 Done Sent Done false
27887 8236 Acked Read 2 Done Sent Done false
27888 8237 Acked Write 10 Done Sent Done false
27889 8238 Acked Read 4 Done Sent Done false
27890 8239 Acked Write 3 Done Sent Done false
27891 8240 Acked Read 5 Done Sent Done false
27892 8241 Acked Read 6 Done Sent Done false
27893 8242 Acked Read 9 Done Sent Done false
27894 8243 Acked Write 1 Done Sent Done false
27895 8244 Acked Write 3 Done Sent Done false
27896 STATES DS:0 DS:1 DS:2 TOTAL
27897 Sent 0 301 0 301
27898 Done 301 0 301 602
27899 Skipped 0 0 0 0
27900 Error 0 0 0 0
27901 Last Flush: Some(JobId(8233)) Some(JobId(7943)) Some(JobId(8233))
27902 Downstairs last five completed: 7943 7942 7941 7940 7939
27903 Upstairs last five completed: 8244 8243 8242 8241 8240
27904 CLIENT: Up:0 ds:301 act:2
279052025-03-17T22:40:43.885ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
279062025-03-17T22:40:44.886ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
279072025-03-17T22:40:45.887ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
27908 [8820] got start action
27909 [8820] I am going to restart
27910 Starting downstairs at port 8820
27911 Make output file at "/tmp/dsc/downstairs-8820.txt"
27912 [8820][1] reports Starting
27913 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
27914 [8820][1] reports Running
27915 ----------------------------------------------------------------
27916 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27917 States: Active Offline Active
27918 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27919 7944 Acked Read 6 Done Sent Done false
27920 7945 Acked Write 5 Done Sent Done false
27921 7946 Acked Write 2 Done Sent Done false
27922 7947 Acked Read 4 Done Sent Done false
27923 7948 Acked Read 2 Done Sent Done false
27924 7949 Acked Write 4 Done Sent Done false
27925 7950 Acked Write 7 Done Sent Done false
27926 7951 Acked Read 1 Done Sent Done false
27927 7952 Acked Write 9 Done Sent Done false
27928 7953 Acked Flush 0 Done Sent Done false
27929 7954 Acked Flush 0 Done Sent Done false
27930 7955 Acked Write 3 Done Sent Done false
27931 7956 Acked Read 5 Done Sent Done false
27932 7957 Acked Read 9 Done Sent Done false
27933 7958 Acked Write 5 Done Sent Done false
27934 7959 Acked Read 7 Done Sent Done false
27935 7960 Acked Write 9 Done Sent Done false
27936 7961 Acked Read 5 Done Sent Done false
27937 7962 Acked Flush 0 Done Sent Done false
27938 7963 Acked Read 5 Done Sent Done false
27939 7964 Acked Write 8 Done Sent Done false
27940 7965 Acked Read 9 Done Sent Done false
27941 7966 Acked Read 2 Done Sent Done false
27942 7967 Acked Read 2 Done Sent Done false
27943 7968 Acked Read 8 Done Sent Done false
27944 7969 Acked Read 10 Done Sent Done false
27945 7970 Acked Write 10 Done Sent Done false
27946 7971 Acked Write 2 Done Sent Done false
27947 7972 Acked Read 9 Done Sent Done false
27948 7973 Acked Write 8 Done Sent Done false
27949 7974 Acked Read 9 Done Sent Done false
27950 7975 Acked Read 1 Done Sent Done false
27951 7976 Acked Read 6 Done Sent Done false
27952 7977 Acked Read 8 Done Sent Done false
27953 7978 Acked Flush 0 Done Sent Done false
27954 7979 Acked Read 3 Done Sent Done false
27955 7980 Acked Write 2 Done Sent Done false
27956 7981 Acked Read 8 Done Sent Done false
27957 7982 Acked Write 5 Done Sent Done false
27958 7983 Acked Read 7 Done Sent Done false
27959 7984 Acked Read 5 Done Sent Done false
27960 7985 Acked Read 1 Done Sent Done false
27961 7986 Acked Read 2 Done Sent Done false
27962 7987 Acked Read 3 Done Sent Done false
27963 7988 Acked Flush 0 Done Sent Done false
27964 7989 Acked Read 10 Done Sent Done false
27965 7990 Acked Read 9 Done Sent Done false
27966 7991 Acked Read 10 Done Sent Done false
27967 7992 Acked Write 4 Done Sent Done false
27968 7993 Acked Write 10 Done Sent Done false
27969 7994 Acked Read 8 Done Sent Done false
27970 7995 Acked Write 4 Done Sent Done false
27971 7996 Acked Read 3 Done Sent Done false
27972 7997 Acked Flush 0 Done Sent Done false
27973 7998 Acked Write 2 Done Sent Done false
27974 7999 Acked Read 5 Done Sent Done false
27975 8000 Acked Write 7 Done Sent Done false
27976 8001 Acked Write 1 Done Sent Done false
27977 8002 Acked Read 4 Done Sent Done false
27978 8003 Acked Read 8 Done Sent Done false
27979 8004 Acked Read 9 Done Sent Done false
27980 8005 Acked Flush 0 Done Sent Done false
27981 8006 Acked Read 2 Done Sent Done false
27982 8007 Acked Write 3 Done Sent Done false
27983 8008 Acked Write 6 Done Sent Done false
27984 8009 Acked Write 10 Done Sent Done false
27985 8010 Acked Read 8 Done Sent Done false
27986 8011 Acked Read 3 Done Sent Done false
27987 8012 Acked Write 6 Done Sent Done false
27988 8013 Acked Read 4 Done Sent Done false
27989 8014 Acked Write 10 Done Sent Done false
27990 8015 Acked Write 10 Done Sent Done false
27991 8016 Acked Read 8 Done Sent Done false
27992 8017 Acked Write 1 Done Sent Done false
27993 8018 Acked Read 5 Done Sent Done false
27994 8019 Acked Write 9 Done Sent Done false
27995 8020 Acked Read 7 Done Sent Done false
27996 8021 Acked Read 7 Done Sent Done false
27997 8022 Acked Write 3 Done Sent Done false
27998 8023 Acked Write 7 Done Sent Done false
27999 8024 Acked Write 3 Done Sent Done false
28000 8025 Acked Flush 0 Done Sent Done false
28001 8026 Acked Write 4 Done Sent Done false
28002 8027 Acked Write 10 Done Sent Done false
28003 8028 Acked Read 8 Done Sent Done false
28004 8029 Acked Read 8 Done Sent Done false
28005 8030 Acked Write 5 Done Sent Done false
28006 8031 Acked Read 3 Done Sent Done false
28007 8032 Acked Read 8 Done Sent Done false
28008 8033 Acked Read 2 Done Sent Done false
28009 8034 Acked Write 2 Done Sent Done false
28010 8035 Acked Read 8 Done Sent Done false
28011 8036 Acked Write 5 Done Sent Done false
28012 8037 Acked Write 2 Done Sent Done false
28013 8038 Acked Read 5 Done Sent Done false
28014 8039 Acked Write 1 Done Sent Done false
28015 8040 Acked Read 5 Done Sent Done false
28016 8041 Acked Read 6 Done Sent Done false
28017 8042 Acked Read 1 Done Sent Done false
28018 8043 Acked Read 8 Done Sent Done false
28019 8044 Acked Flush 0 Done Sent Done false
28020 8045 Acked Read 7 Done Sent Done false
28021 8046 Acked Write 7 Done Sent Done false
28022 8047 Acked Read 5 Done Sent Done false
28023 8048 Acked Read 1 Done Sent Done false
28024 8049 Acked Read 9 Done Sent Done false
28025 8050 Acked Read 3 Done Sent Done false
28026 8051 Acked Read 8 Done Sent Done false
28027 8052 Acked Read 3 Done Sent Done false
28028 8053 Acked Read 5 Done Sent Done false
28029 8054 Acked Read 2 Done Sent Done false
28030 8055 Acked Read 2 Done Sent Done false
28031 8056 Acked Flush 0 Done Sent Done false
28032 8057 Acked Read 7 Done Sent Done false
28033 8058 Acked Write 3 Done Sent Done false
28034 8059 Acked Write 8 Done Sent Done false
28035 8060 Acked Write 10 Done Sent Done false
28036 8061 Acked Read 5 Done Sent Done false
28037 8062 Acked Write 8 Done Sent Done false
28038 8063 Acked Write 5 Done Sent Done false
28039 8064 Acked Write 10 Done Sent Done false
28040 8065 Acked Flush 0 Done Sent Done false
28041 8066 Acked Read 8 Done Sent Done false
28042 8067 Acked Read 9 Done Sent Done false
28043 8068 Acked Read 1 Done Sent Done false
28044 8069 Acked Flush 0 Done Sent Done false
28045 8070 Acked Write 3 Done Sent Done false
28046 8071 Acked Read 5 Done Sent Done false
28047 8072 Acked Write 9 Done Sent Done false
28048 8073 Acked Read 7 Done Sent Done false
28049 8074 Acked Read 6 Done Sent Done false
28050 8075 Acked Write 9 Done Sent Done false
28051 8076 Acked Read 7 Done Sent Done false
28052 8077 Acked Flush 0 Done Sent Done false
28053 8078 Acked Read 7 Done Sent Done false
28054 8079 Acked Write 10 Done Sent Done false
28055 8080 Acked Read 8 Done Sent Done false
28056 8081 Acked Write 7 Done Sent Done false
28057 8082 Acked Write 1 Done Sent Done false
28058 8083 Acked Read 1 Done Sent Done false
28059 8084 Acked Read 4 Done Sent Done false
28060 8085 Acked Read 8 Done Sent Done false
28061 8086 Acked Flush 0 Done Sent Done false
28062 8087 Acked Write 6 Done Sent Done false
28063 8088 Acked Read 2 Done Sent Done false
28064 8089 Acked Read 2 Done Sent Done false
28065 8090 Acked Write 9 Done Sent Done false
28066 8091 Acked Write 7 Done Sent Done false
28067 8092 Acked Write 3 Done Sent Done false
28068 8093 Acked Write 2 Done Sent Done false
28069 8094 Acked Read 10 Done Sent Done false
28070 8095 Acked Flush 0 Done Sent Done false
28071 8096 Acked Read 10 Done Sent Done false
28072 8097 Acked Write 8 Done Sent Done false
28073 8098 Acked Write 8 Done Sent Done false
28074 8099 Acked Write 3 Done Sent Done false
28075 8100 Acked Write 10 Done Sent Done false
28076 8101 Acked Flush 0 Done Sent Done false
28077 8102 Acked Read 8 Done Sent Done false
28078 8103 Acked Write 6 Done Sent Done false
28079 8104 Acked Read 8 Done Sent Done false
28080 8105 Acked Write 1 Done Sent Done false
28081 8106 Acked Write 1 Done Sent Done false
28082 8107 Acked Flush 0 Done Sent Done false
28083 8108 Acked Read 2 Done Sent Done false
28084 8109 Acked Write 9 Done Sent Done false
28085 8110 Acked Flush 0 Done Sent Done false
28086 8111 Acked Read 4 Done Sent Done false
28087 8112 Acked Read 10 Done Sent Done false
28088 8113 Acked Read 3 Done Sent Done false
28089 8114 Acked Write 3 Done Sent Done false
28090 8115 Acked Read 1 Done Sent Done false
28091 8116 Acked Write 9 Done Sent Done false
28092 8117 Acked Write 8 Done Sent Done false
28093 8118 Acked Read 1 Done Sent Done false
28094 8119 Acked Write 3 Done Sent Done false
28095 8120 Acked Write 4 Done Sent Done false
28096 8121 Acked Read 1 Done Sent Done false
28097 8122 Acked Read 6 Done Sent Done false
28098 8123 Acked Write 5 Done Sent Done false
28099 8124 Acked Write 6 Done Sent Done false
28100 8125 Acked Write 9 Done Sent Done false
28101 8126 Acked Write 9 Done Sent Done false
28102 8127 Acked Read 2 Done Sent Done false
28103 8128 Acked Flush 0 Done Sent Done false
28104 8129 Acked Read 6 Done Sent Done false
28105 8130 Acked Read 9 Done Sent Done false
28106 8131 Acked Read 3 Done Sent Done false
28107 8132 Acked Read 9 Done Sent Done false
28108 8133 Acked Read 3 Done Sent Done false
28109 8134 Acked Write 6 Done Sent Done false
28110 8135 Acked Write 1 Done Sent Done false
28111 8136 Acked Write 6 Done Sent Done false
28112 8137 Acked Write 9 Done Sent Done false
28113 8138 Acked Read 9 Done Sent Done false
28114 8139 Acked Write 7 Done Sent Done false
28115 8140 Acked Read 5 Done Sent Done false
28116 8141 Acked Read 4 Done Sent Done false
28117 8142 Acked Write 10 Done Sent Done false
28118 8143 Acked Write 5 Done Sent Done false
28119 8144 Acked Write 2 Done Sent Done false
28120 8145 Acked Write 10 Done Sent Done false
28121 8146 Acked Write 10 Done Sent Done false
28122 8147 Acked Read 3 Done Sent Done false
28123 8148 Acked Read 6 Done Sent Done false
28124 8149 Acked Write 4 Done Sent Done false
28125 8150 Acked Read 6 Done Sent Done false
28126 8151 Acked Write 7 Done Sent Done false
28127 8152 Acked Read 7 Done Sent Done false
28128 8153 Acked Write 2 Done Sent Done false
28129 8154 Acked Read 9 Done Sent Done false
28130 8155 Acked Read 10 Done Sent Done false
28131 8156 Acked Read 6 Done Sent Done false
28132 8157 Acked Read 3 Done Sent Done false
28133 8158 Acked Read 5 Done Sent Done false
28134 8159 Acked Read 2 Done Sent Done false
28135 8160 Acked Read 2 Done Sent Done false
28136 8161 Acked Write 8 Done Sent Done false
28137 8162 Acked Write 1 Done Sent Done false
28138 8163 Acked Read 8 Done Sent Done false
28139 8164 Acked Write 4 Done Sent Done false
28140 8165 Acked Read 3 Done Sent Done false
28141 8166 Acked Write 6 Done Sent Done false
28142 8167 Acked Read 6 Done Sent Done false
28143 8168 Acked Write 8 Done Sent Done false
28144 8169 Acked Read 4 Done Sent Done false
28145 8170 Acked Read 2 Done Sent Done false
28146 8171 Acked Read 8 Done Sent Done false
28147 8172 Acked Flush 0 Done Sent Done false
28148 8173 Acked Write 10 Done Sent Done false
28149 8174 Acked Read 7 Done Sent Done false
28150 8175 Acked Read 7 Done Sent Done false
28151 8176 Acked Read 5 Done Sent Done false
28152 8177 Acked Read 8 Done Sent Done false
28153 8178 Acked Read 1 Done Sent Done false
28154 8179 Acked Write 9 Done Sent Done false
28155 8180 Acked Read 4 Done Sent Done false
28156 8181 Acked Flush 0 Done Sent Done false
28157 8182 Acked Read 6 Done Sent Done false
28158 8183 Acked Read 6 Done Sent Done false
28159 8184 Acked Read 2 Done Sent Done false
28160 8185 Acked Read 8 Done Sent Done false
28161 8186 Acked Read 7 Done Sent Done false
28162 8187 Acked Flush 0 Done Sent Done false
28163 8188 Acked Read 9 Done Sent Done false
28164 8189 Acked Write 3 Done Sent Done false
28165 8190 Acked Write 10 Done Sent Done false
28166 8191 Acked Read 5 Done Sent Done false
28167 8192 Acked Read 10 Done Sent Done false
28168 8193 Acked Write 9 Done Sent Done false
28169 8194 Acked Write 9 Done Sent Done false
28170 8195 Acked Read 5 Done Sent Done false
28171 8196 Acked Read 5 Done Sent Done false
28172 8197 Acked Write 9 Done Sent Done false
28173 8198 Acked Write 3 Done Sent Done false
28174 8199 Acked Read 2 Done Sent Done false
28175 8200 Acked Write 4 Done Sent Done false
28176 8201 Acked Read 4 Done Sent Done false
28177 8202 Acked Write 2 Done Sent Done false
28178 8203 Acked Read 3 Done Sent Done false
28179 8204 Acked Read 4 Done Sent Done false
28180 8205 Acked Read 8 Done Sent Done false
28181 8206 Acked Write 4 Done Sent Done false
28182 8207 Acked Write 1 Done Sent Done false
28183 8208 Acked Write 8 Done Sent Done false
28184 8209 Acked Flush 0 Done Sent Done false
28185 8210 Acked Write 2 Done Sent Done false
28186 8211 Acked Read 10 Done Sent Done false
28187 8212 Acked Write 4 Done Sent Done false
28188 8213 Acked Read 4 Done Sent Done false
28189 8214 Acked Read 2 Done Sent Done false
28190 8215 Acked Flush 0 Done Sent Done false
28191 8216 Acked Read 1 Done Sent Done false
28192 8217 Acked Write 4 Done Sent Done false
28193 8218 Acked Flush 0 Done Sent Done false
28194 8219 Acked Read 3 Done Sent Done false
28195 8220 Acked Read 4 Done Sent Done false
28196 8221 Acked Read 4 Done Sent Done false
28197 8222 Acked Write 5 Done Sent Done false
28198 8223 Acked Read 7 Done Sent Done false
28199 8224 Acked Read 5 Done Sent Done false
28200 8225 Acked Write 6 Done Sent Done false
28201 8226 Acked Write 5 Done Sent Done false
28202 8227 Acked Write 1 Done Sent Done false
28203 8228 Acked Write 4 Done Sent Done false
28204 8229 Acked Write 3 Done Sent Done false
28205 8230 Acked Read 8 Done Sent Done false
28206 8231 Acked Write 3 Done Sent Done false
28207 8232 Acked Flush 0 Done Sent Done false
28208 8233 Acked Flush 0 Done Sent Done false
28209 8234 Acked Write 9 Done Sent Done false
28210 8235 Acked Read 10 Done Sent Done false
28211 8236 Acked Read 2 Done Sent Done false
28212 8237 Acked Write 10 Done Sent Done false
28213 8238 Acked Read 4 Done Sent Done false
28214 8239 Acked Write 3 Done Sent Done false
28215 8240 Acked Read 5 Done Sent Done false
28216 8241 Acked Read 6 Done Sent Done false
28217 8242 Acked Read 9 Done Sent Done false
28218 8243 Acked Write 1 Done Sent Done false
28219 8244 Acked Write 3 Done Sent Done false
28220 STATES DS:0 DS:1 DS:2 TOTAL
28221 Sent 0 301 0 301
28222 Done 301 0 301 602
28223 Skipped 0 0 0 0
28224 Error 0 0 0 0
28225 Last Flush: Some(JobId(8233)) Some(JobId(7943)) Some(JobId(8233))
28226 Downstairs last five completed: 7943 7942 7941 7940 7939
28227 Upstairs last five completed: 8244 8243 8242 8241 8240
28228 CLIENT: Up:0 ds:301 act:2
282292025-03-17T22:40:47.890ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282302025-03-17T22:40:48.676ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282312025-03-17T22:40:48.677ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54848) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282322025-03-17T22:40:48.678ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282332025-03-17T22:40:48.678ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282342025-03-17T22:40:48.678ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282352025-03-17T22:40:48.678ZINFOcrucible: send last flush ID to this DS: Some(JobId(7943)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282362025-03-17T22:40:48.678ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282372025-03-17T22:40:48.678ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7943)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282382025-03-17T22:40:48.679ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7943)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28239 ----------------------------------------------------------------
28240 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28241 Downstairs last five completed: 8245 8244 8243 8242 8241
28242 Upstairs last five completed: 8245 8244 8243 8242 8241
28243 CLIENT: Up:0 ds:0 act:3
28244 Replay: All jobs finished
28245 Main task has work to do, go find it
28246 got dsc Stop(1)
28247 stop 1
28248 Mar 17 22:40:50.931 INFO [8820] Got stop action so:false kr:false
28249 request completed, latency_us: 391, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c53a9cb9-5851-4c83-b88f-229c7d381adf, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
28250 Mar 17 22:40:50.932 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/1, method: GET, req_id: bd86d966-5771-42b0-95e5-2235479928c7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
282512025-03-17T22:40:50.932ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282522025-03-17T22:40:50.932ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282532025-03-17T22:40:50.932ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
282542025-03-17T22:40:50.932ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28255 [8820] Exited with: ExitStatus(unix_wait_status(9))
28256 [8820][1] reports Exit
28257 Mar 17 22:40:54.934 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 254035f4-2429-4025-8c9b-8d90aed2c5d9, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
28258 001/300 Flush
28259 002/300 Read block 4588 len 16384
28260 003/300 Flush
28261 004/300 Flush
28262 005/300 Write block 3318 len 8192 data: 5 7
28263 006/300 Read block 2106 len 24576
28264 007/300 Write block 2288 len 4096 data: 4
28265 008/300 Flush
28266 009/300 Read block 1480 len 20480
28267 010/300 Write block 2689 len 12288 data: 4 3 3
28268 011/300 Write block 3375 len 40960 data: 4 4 3 4 4 5 5 6 6 8
28269 012/300 Write block 1231 len 40960 data: 4 4 5 5 5 4 3 4 3 3
28270 013/300 Flush
28271 014/300 Read block 4862 len 40960
28272 015/300 Read block 2160 len 8192
28273 016/300 Read block 2443 len 36864
28274 017/300 Write block 1361 len 36864 data: 7 8 8 8 6 4 4 4 3
28275 018/300 Flush
28276 019/300 Write block 2683 len 12288 data: 3 4 4
28277 020/300 Read block 2426 len 4096
28278 021/300 Read block 1466 len 28672
28279 022/300 Read block 743 len 24576
28280 023/300 Write block 566 len 12288 data: 4 5 5
28281 024/300 Write block 2230 len 24576 data: 4 4 4 5 5 5
28282 025/300 Write block 34 len 24576 data: 6 7 6 7 5 6
28283 026/300 Write block 1699 len 32768 data: 2 4 5 6 6 5 5 5
28284 027/300 Read block 4475 len 24576
28285 028/300 Write block 1609 len 12288 data: 5 5 4
28286 029/300 Read block 4725 len 40960
28287 030/300 Write block 2837 len 12288 data: 7 7 6
28288 031/300 Write block 1817 len 28672 data: 5 5 4 4 5 5 5
28289 032/300 Read block 4861 len 12288
28290 033/300 Read block 3461 len 36864
28291 034/300 Flush
28292 035/300 Read block 2721 len 12288
28293 036/300 Write block 950 len 8192 data: 5 4
28294 037/300 Read block 4026 len 40960
28295 038/300 Read block 3366 len 4096
28296 039/300 Read block 320 len 4096
28297 040/300 Write block 4110 len 8192 data: 7 7
28298 041/300 Read block 2676 len 24576
28299 042/300 Write block 4659 len 4096 data: 7
28300 043/300 Read block 2184 len 28672
28301 044/300 Write block 124 len 12288 data: 9 10 9
28302 045/300 Read block 1588 len 20480
28303 046/300 Read block 75 len 36864
28304 047/300 Flush
28305 048/300 Flush
28306 049/300 Flush
28307 050/300 Read block 1857 len 32768
28308 051/300 Write block 2697 len 32768 data: 5 6 6 6 5 4 4 4
28309 052/300 Write block 2836 len 24576 data: 7 8 8 7 5 4
28310 053/300 Write block 3185 len 36864 data: 5 6 5 7 7 5 3 4 5
28311 054/300 Write block 1638 len 12288 data: 7 7 7
28312 055/300 Read block 150 len 4096
28313 056/300 Read block 4078 len 8192
28314 057/300 Write block 3571 len 24576 data: 9 9 5 7 5 4
28315 058/300 Write block 3759 len 8192 data: 6 5
28316 059/300 Read block 1486 len 24576
28317 060/300 Write block 1776 len 8192 data: 3 2
28318 061/300 Write block 1943 len 12288 data: 4 4 5
28319 062/300 Write block 1923 len 36864 data: 5 5 6 7 6 4 4 4 3
28320 063/300 Read block 4791 len 4096
28321 064/300 Flush
28322 065/300 Read block 4037 len 28672
28323 066/300 Write block 4235 len 12288 data: 6 5 5
28324 067/300 Read block 3101 len 8192
28325 068/300 Read block 4686 len 4096
28326 069/300 Write block 3032 len 20480 data: 7 8 8 7 7
28327 070/300 Read block 3168 len 32768
28328 071/300 Write block 2683 len 40960 data: 4 5 5 5 6 6 5 4 4 3
28329 072/300 Read block 4173 len 8192
28330 073/300 Read block 2158 len 8192
28331 074/300 Write block 227 len 40960 data: 3 3 4 7 8 8 7 7 7 6
28332 075/300 Read block 3804 len 16384
28333 076/300 Flush
28334 077/300 Write block 1725 len 20480 data: 7 7 6 6 6
28335 078/300 Read block 2984 len 28672
28336 079/300 Read block 607 len 12288
28337 080/300 Read block 1023 len 8192
28338 081/300 Write block 4243 len 12288 data: 4 4 4
28339 082/300 Read block 2354 len 28672
28340 083/300 Read block 4177 len 20480
28341 084/300 Write block 2410 len 20480 data: 4 4 5 4 4
28342 085/300 Flush
28343 086/300 Read block 1472 len 28672
28344 087/300 Read block 4232 len 28672
28345 088/300 Read block 4011 len 16384
28346 089/300 Write block 1946 len 16384 data: 5 5 6 7
28347 090/300 Write block 377 len 36864 data: 2 3 3 5 6 5 6 9 11
28348 091/300 Read block 268 len 40960
28349 092/300 Read block 3154 len 36864
28350 093/300 Flush
28351 094/300 Read block 2455 len 8192
28352 095/300 Write block 2629 len 4096 data: 4
28353 096/300 Read block 628 len 4096
28354 097/300 Read block 1423 len 24576
28355 098/300 Write block 2421 len 36864 data: 2 2 4 5 5 5 5 5 4
28356 099/300 Read block 59 len 32768
28357 100/300 Read block 1827 len 20480
28358 101/300 Write block 361 len 16384 data: 2 5 5 5
28359 102/300 Read block 4048 len 36864
28360 103/300 Write block 996 len 24576 data: 5 5 5 6 6 5
28361 104/300 Read block 1172 len 32768
28362 105/300 Write block 1018 len 16384 data: 5 4 4 3
28363 106/300 Write block 4377 len 28672 data: 4 5 4 4 6 4 3
28364 107/300 Write block 1330 len 20480 data: 4 4 5 4 4
28365 108/300 Read block 4979 len 20480
28366 109/300 Write block 953 len 20480 data: 4 3 3 3 3
28367 110/300 Read block 291 len 32768
28368 111/300 Read block 2117 len 8192
28369 112/300 Flush
28370 113/300 Read block 4628 len 40960
28371 114/300 Read block 496 len 28672
28372 115/300 Write block 1059 len 12288 data: 4 4 4
28373 116/300 Read block 453 len 32768
28374 117/300 Write block 2204 len 24576 data: 8 10 9 8 6 6
28375 118/300 Read block 394 len 24576
28376 119/300 Write block 2329 len 28672 data: 6 6 6 4 3 3 4
28377 120/300 Write block 4576 len 12288 data: 7 5 6
28378 121/300 Write block 3750 len 24576 data: 2 2 2 3 4 4
28379 122/300 Write block 836 len 12288 data: 4 4 4
28380 123/300 Read block 2260 len 20480
28381 124/300 Write block 4168 len 36864 data: 6 6 5 5 6 5 6 5 6
28382 125/300 Write block 3620 len 12288 data: 3 5 5
28383 126/300 Read block 4493 len 4096
28384 127/300 Write block 4063 len 32768 data: 4 4 4 5 6 6 5 6
28385 128/300 Read block 4006 len 36864
28386 129/300 Write block 3037 len 12288 data: 6 5 5
28387 130/300 Write block 969 len 32768 data: 5 6 6 4 4 6 7 6
28388 131/300 Write block 1167 len 20480 data: 3 3 4 4 4
28389 132/300 Read block 4968 len 16384
28390 133/300 Write block 517 len 16384 data: 5 5 5 6
28391 134/300 Read block 2398 len 8192
28392 135/300 Flush
28393 136/300 Flush
28394 137/300 Write block 4401 len 20480 data: 4 4 4 4 5
28395 138/300 Write block 1758 len 16384 data: 5 6 4 4
28396 139/300 Flush
28397 140/300 Read block 3648 len 24576
28398 141/300 Read block 4241 len 28672
28399 142/300 Write block 1998 len 28672 data: 7 6 6 7 7 7 5
28400 143/300 Write block 828 len 40960 data: 4 3 4 4 4 4 4 4 5 5
28401 144/300 Write block 97 len 32768 data: 5 6 7 7 7 7 8 8
28402 145/300 Write block 1266 len 24576 data: 6 7 3 4 4 5
28403 146/300 Read block 4394 len 28672
28404 147/300 Read block 3215 len 32768
28405 148/300 Read block 2466 len 4096
28406 149/300 Flush
28407 150/300 Read block 9 len 40960
28408 151/300 Write block 4984 len 16384 data: 6 5 5 4
28409 152/300 Read block 4247 len 32768
28410 153/300 Write block 1511 len 32768 data: 5 7 7 5 5 6 6 5
28411 154/300 Read block 4993 len 28672
28412 155/300 Write block 2411 len 12288 data: 5 6 5
28413 156/300 Write block 2260 len 4096 data: 6
28414 157/300 Read block 640 len 8192
28415 158/300 Read block 908 len 40960
28416 159/300 Write block 4330 len 16384 data: 4 3 3 4
28417 160/300 Read block 3646 len 32768
28418 161/300 Write block 2492 len 24576 data: 4 6 5 5 6 4
28419 162/300 Write block 4360 len 20480 data: 3 2 2 3 3
28420 163/300 Write block 4839 len 20480 data: 3 5 5 5 7
28421 164/300 Write block 3339 len 8192 data: 7 8
28422 165/300 Write block 4162 len 12288 data: 5 6 6
28423 166/300 Read block 2462 len 36864
28424 167/300 Read block 79 len 20480
28425 168/300 Write block 2056 len 28672 data: 4 5 5 4 4 4 4
28426 169/300 Read block 4256 len 32768
28427 170/300 Write block 4698 len 24576 data: 7 6 11 11 11 10
28428 171/300 Write block 3348 len 24576 data: 3 3 4 5 5 4
28429 172/300 Write block 2761 len 4096 data: 4
28430 173/300 Write block 4230 len 12288 data: 5 6 7
28431 174/300 Write block 4549 len 40960 data: 6 7 6 6 6 7 6 6 4 4
28432 175/300 Read block 3244 len 16384
28433 176/300 Flush
28434 177/300 Read block 2941 len 36864
28435 178/300 Flush
28436 179/300 Write block 3440 len 28672 data: 5 4 4 4 4 4 3
28437 180/300 Read block 3938 len 40960
28438 181/300 Write block 465 len 32768 data: 3 3 3 6 4 2 2 3
28439 182/300 Read block 3674 len 40960
28440 183/300 Read block 2583 len 8192
28441 184/300 Write block 4519 len 4096 data: 8
28442 185/300 Write block 379 len 8192 data: 4 6
28443 186/300 Write block 1439 len 8192 data: 5 4
28444 187/300 Write block 3855 len 8192 data: 6 6
28445 188/300 Read block 3319 len 40960
28446 189/300 Read block 2486 len 40960
28447 190/300 Write block 3991 len 40960 data: 7 7 6 6 5 6 4 5 7 7
28448 191/300 Read block 1538 len 16384
28449 192/300 Write block 1020 len 8192 data: 5 4
28450 193/300 Read block 4934 len 32768
28451 194/300 Read block 4979 len 28672
28452 195/300 Write block 591 len 4096 data: 3
28453 196/300 Read block 4383 len 16384
28454 197/300 Read block 3889 len 8192
28455 198/300 Read block 3312 len 16384
28456 199/300 Read block 2171 len 28672
28457 200/300 Write block 4199 len 20480 data: 4 5 5 5 6
28458 201/300 Read block 661 len 24576
28459 202/300 Read block 2757 len 16384
28460 203/300 Read block 1953 len 32768
28461 204/300 Read block 1923 len 12288
28462 205/300 Write block 3618 len 36864 data: 3 3 4 6 6 7 7 7 8
28463 206/300 Read block 595 len 12288
28464 207/300 Flush
28465 208/300 Write block 2986 len 28672 data: 10 7 8 7 9 8 9
28466 209/300 Write block 1173 len 8192 data: 4 4
28467 210/300 Read block 4777 len 16384
28468 211/300 Write block 3114 len 16384 data: 4 4 4 4
28469 212/300 Read block 1133 len 28672
28470 213/300 Write block 4763 len 4096 data: 4
28471 214/300 Write block 3625 len 4096 data: 8
28472 215/300 Read block 2999 len 8192
28473 216/300 Read block 2764 len 32768
28474 217/300 Write block 1111 len 8192 data: 3 2
28475 218/300 Write block 1344 len 28672 data: 3 3 4 4 4 4 4
28476 219/300 Read block 1904 len 40960
28477 220/300 Read block 1565 len 36864
28478 221/300 Read block 746 len 16384
28479 222/300 Read block 4038 len 8192
28480 223/300 Read block 3257 len 36864
28481 224/300 Read block 3431 len 28672
28482 225/300 Write block 1461 len 20480 data: 6 6 6 5 5
28483 226/300 Read block 4776 len 32768
28484 227/300 Write block 119 len 16384 data: 4 6 6 8
28485 228/300 Read block 2872 len 4096
28486 229/300 Read block 4001 len 32768
28487 230/300 Read block 1840 len 40960
28488 231/300 Write block 3450 len 24576 data: 6 6 5 6 4 5
28489 232/300 Write block 1533 len 36864 data: 10 8 7 7 4 3 2 2 3
28490 233/300 Read block 195 len 40960
28491 234/300 Read block 2224 len 8192
28492 235/300 Read block 3406 len 4096
28493 236/300 Write block 4486 len 4096 data: 5
28494 237/300 Read block 1697 len 8192
28495 238/300 Write block 4642 len 32768 data: 7 5 6 6 7 6 6 5
28496 239/300 Write block 2223 len 24576 data: 4 5 4 4 4 5
28497 240/300 Flush
28498 241/300 Read block 3438 len 28672
28499 242/300 Write block 42 len 36864 data: 6 7 7 7 6 6 5 3 2
28500 243/300 Read block 4113 len 4096
28501 244/300 Write block 89 len 32768 data: 4 4 3 3 4 4 5 5
28502 245/300 Write block 3914 len 36864 data: 6 6 5 6 7 7 5 6 6
28503 246/300 Read block 2052 len 40960
28504 247/300 Write block 2604 len 36864 data: 4 4 3 3 5 6 5 4 4
28505 248/300 Write block 2393 len 40960 data: 3 2 2 2 3 3 4 6 6 7
28506 249/300 Write block 2105 len 8192 data: 5 5
28507 250/300 Read block 1579 len 8192
28508 251/300 Write block 1449 len 20480 data: 3 3 2 3 5
28509 252/300 Read block 71 len 36864
28510 253/300 Write block 2594 len 4096 data: 3
28511 254/300 Read block 3986 len 12288
28512 255/300 Write block 1752 len 32768 data: 2 2 4 4 4 5 6 7
28513 256/300 Read block 903 len 36864
28514 257/300 Write block 4544 len 12288 data: 5 5 4
28515 258/300 Write block 1343 len 8192 data: 4 4
28516 259/300 Read block 3323 len 12288
28517 260/300 Read block 439 len 28672
28518 261/300 Write block 2177 len 36864 data: 6 5 5 5 6 6 5 5 5
28519 262/300 Read block 4393 len 12288
28520 263/300 Read block 3435 len 16384
28521 264/300 Flush
28522 265/300 Write block 4440 len 32768 data: 5 5 6 7 5 5 4 4
28523 266/300 Read block 4447 len 28672
28524 267/300 Flush
28525 268/300 Read block 560 len 32768
28526 269/300 Write block 4764 len 4096 data: 4
28527 270/300 Write block 2452 len 20480 data: 6 5 5 5 6
28528 271/300 Write block 2911 len 32768 data: 6 5 5 4 4 5 6 3
28529 272/300 Read block 4245 len 20480
28530 273/300 Read block 2857 len 4096
28531 274/300 Read block 480 len 16384
28532 275/300 Write block 4595 len 4096 data: 3
28533 276/300 Read block 1195 len 36864
28534 277/300 Write block 3675 len 8192 data: 5 5
28535 278/300 Read block 2561 len 28672
28536 279/300 Read block 3660 len 20480
28537 280/300 Write block 1035 len 36864 data: 3 4 4 4 4 4 4 5 4
28538 281/300 Read block 1119 len 16384
28539 282/300 Read block 4101 len 36864
28540 283/300 Write block 441 len 28672 data: 7 6 7 8 7 7 6
28541 284/300 Write block 2371 len 32768 data: 3 4 5 5 5 3 3 3
28542 285/300 Write block 1595 len 36864 data: 10 10 10 10 8 7 7 5 4
28543 286/300 Write block 3184 len 28672 data: 5 6 7 6 8 8 6
28544 287/300 Read block 866 len 4096
28545 288/300 Read block 784 len 24576
28546 289/300 Read block 1665 len 16384
28547 290/300 Read block 904 len 40960
28548 291/300 Read block 2806 len 28672
28549 292/300 Write block 2052 len 36864 data: 6 6 7 5 5 6 6 5 5
28550 293/300 Read block 2387 len 32768
28551 294/300 Read block 3171 len 8192
28552 295/300 Read block 548 len 4096
28553 296/300 Read block 3338 len 12288
28554 297/300 Flush
28555 298/300 Write block 1504 len 24576 data: 6 4 3 6 7 7
28556 299/300 Read block 4381 len 12288
28557 300/300 Read block 611 len 8192
28558 301/300 Flush
28559 Main task has work to do, go find it
28560 got dsc Start(1)
28561 start 1
28562 Mar 17 22:40:55.175 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7a5afa2c-8b04-437b-a66e-0db4399ad0c8, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
28563 Replay: started 1, returned:Ok(())
28564 ----------------------------------------------------------------
28565 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28566 States: Active Offline Active
28567 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28568 8246 Acked Flush 0 Done Sent Done false
28569 8247 Acked Read 4 Done Sent Done false
28570 8248 Acked Flush 0 Done Sent Done false
28571 8249 Acked Flush 0 Done Sent Done false
28572 8250 Acked Write 2 Done Sent Done false
28573 8251 Acked Read 6 Done Sent Done false
28574 8252 Acked Write 1 Done Sent Done false
28575 8253 Acked Flush 0 Done Sent Done false
28576 8254 Acked Read 5 Done Sent Done false
28577 8255 Acked Write 3 Done Sent Done false
28578 8256 Acked Write 10 Done Sent Done false
28579 8257 Acked Write 10 Done Sent Done false
28580 8258 Acked Flush 0 Done Sent Done false
28581 8259 Acked Read 10 Done Sent Done false
28582 8260 Acked Read 2 Done Sent Done false
28583 8261 Acked Read 9 Done Sent Done false
28584 8262 Acked Write 9 Done Sent Done false
28585 8263 Acked Flush 0 Done Sent Done false
28586 8264 Acked Write 3 Done Sent Done false
28587 8265 Acked Read 1 Done Sent Done false
28588 8266 Acked Read 7 Done Sent Done false
28589 8267 Acked Read 6 Done Sent Done false
28590 8268 Acked Write 3 Done Sent Done false
28591 8269 Acked Write 6 Done Sent Done false
28592 8270 Acked Write 6 Done Sent Done false
28593 8271 Acked Write 8 Done Sent Done false
28594 8272 Acked Read 6 Done Sent Done false
28595 8273 Acked Write 3 Done Sent Done false
28596 8274 Acked Read 10 Done Sent Done false
28597 8275 Acked Write 3 Done Sent Done false
28598 8276 Acked Write 7 Done Sent Done false
28599 8277 Acked Read 3 Done Sent Done false
28600 8278 Acked Read 9 Done Sent Done false
28601 8279 Acked Flush 0 Done Sent Done false
28602 8280 Acked Read 3 Done Sent Done false
28603 8281 Acked Write 2 Done Sent Done false
28604 8282 Acked Read 10 Done Sent Done false
28605 8283 Acked Read 1 Done Sent Done false
28606 8284 Acked Read 1 Done Sent Done false
28607 8285 Acked Write 2 Done Sent Done false
28608 8286 Acked Read 6 Done Sent Done false
28609 8287 Acked Write 1 Done Sent Done false
28610 8288 Acked Read 7 Done Sent Done false
28611 8289 Acked Write 3 Done Sent Done false
28612 8290 Acked Read 5 Done Sent Done false
28613 8291 Acked Read 9 Done Sent Done false
28614 8292 Acked Flush 0 Done Sent Done false
28615 8293 Acked Flush 0 Done Sent Done false
28616 8294 Acked Flush 0 Done Sent Done false
28617 8295 Acked Read 8 Done Sent Done false
28618 8296 Acked Write 8 Done Sent Done false
28619 8297 Acked Write 6 Done Sent Done false
28620 8298 Acked Write 9 Done Sent Done false
28621 8299 Acked Write 3 Done Sent Done false
28622 8300 Acked Read 1 Done Sent Done false
28623 8301 Acked Read 2 Done Sent Done false
28624 8302 Acked Write 6 Done Sent Done false
28625 8303 Acked Write 2 Done Sent Done false
28626 8304 Acked Read 6 Done Sent Done false
28627 8305 Acked Write 2 Done Sent Done false
28628 8306 Acked Write 3 Done Sent Done false
28629 8307 Acked Write 9 Done Sent Done false
28630 8308 Acked Read 1 Done Sent Done false
28631 8309 Acked Flush 0 Done Sent Done false
28632 8310 Acked Read 7 Done Sent Done false
28633 8311 Acked Write 3 Done Sent Done false
28634 8312 Acked Read 2 Done Sent Done false
28635 8313 Acked Read 1 Done Sent Done false
28636 8314 Acked Write 5 Done Sent Done false
28637 8315 Acked Read 8 Done Sent Done false
28638 8316 Acked Write 10 Done Sent Done false
28639 8317 Acked Read 2 Done Sent Done false
28640 8318 Acked Read 2 Done Sent Done false
28641 8319 Acked Write 10 Done Sent Done false
28642 8320 Acked Read 4 Done Sent Done false
28643 8321 Acked Flush 0 Done Sent Done false
28644 8322 Acked Write 5 Done Sent Done false
28645 8323 Acked Read 7 Done Sent Done false
28646 8324 Acked Read 3 Done Sent Done false
28647 8325 Acked Read 2 Done Sent Done false
28648 8326 Acked Write 3 Done Sent Done false
28649 8327 Acked Read 7 Done Sent Done false
28650 8328 Acked Read 5 Done Sent Done false
28651 8329 Acked Write 5 Done Sent Done false
28652 8330 Acked Flush 0 Done Sent Done false
28653 8331 Acked Read 7 Done Sent Done false
28654 8332 Acked Read 7 Done Sent Done false
28655 8333 Acked Read 4 Done Sent Done false
28656 8334 Acked Write 4 Done Sent Done false
28657 8335 Acked Write 9 Done Sent Done false
28658 8336 Acked Read 10 Done Sent Done false
28659 8337 Acked Read 9 Done Sent Done false
28660 8338 Acked Flush 0 Done Sent Done false
28661 8339 Acked Read 2 Done Sent Done false
28662 8340 Acked Write 1 Done Sent Done false
28663 8341 Acked Read 1 Done Sent Done false
28664 8342 Acked Read 6 Done Sent Done false
28665 8343 Acked Write 9 Done Sent Done false
28666 8344 Acked Read 8 Done Sent Done false
28667 8345 Acked Read 5 Done Sent Done false
28668 8346 Acked Write 4 Done Sent Done false
28669 8347 Acked Read 9 Done Sent Done false
28670 8348 Acked Write 6 Done Sent Done false
28671 8349 Acked Read 8 Done Sent Done false
28672 8350 Acked Write 4 Done Sent Done false
28673 8351 Acked Write 7 Done Sent Done false
28674 8352 Acked Write 5 Done Sent Done false
28675 8353 Acked Read 5 Done Sent Done false
28676 8354 Acked Write 5 Done Sent Done false
28677 8355 Acked Read 8 Done Sent Done false
28678 8356 Acked Read 2 Done Sent Done false
28679 8357 Acked Flush 0 Done Sent Done false
28680 8358 Acked Read 10 Done Sent Done false
28681 8359 Acked Read 7 Done Sent Done false
28682 8360 Acked Write 3 Done Sent Done false
28683 8361 Acked Read 8 Done Sent Done false
28684 8362 Acked Write 6 Done Sent Done false
28685 8363 Acked Read 6 Done Sent Done false
28686 8364 Acked Write 7 Done Sent Done false
28687 8365 Acked Write 3 Done Sent Done false
28688 8366 Acked Write 6 Done Sent Done false
28689 8367 Acked Write 3 Done Sent Done false
28690 8368 Acked Read 5 Done Sent Done false
28691 8369 Acked Write 9 Done Sent Done false
28692 8370 Acked Write 3 Done Sent Done false
28693 8371 Acked Read 1 Done Sent Done false
28694 8372 Acked Write 8 Done Sent Done false
28695 8373 Acked Read 9 Done Sent Done false
28696 8374 Acked Write 3 Done Sent Done false
28697 8375 Acked Write 8 Done Sent Done false
28698 8376 Acked Write 5 Done Sent Done false
28699 8377 Acked Read 4 Done Sent Done false
28700 8378 Acked Write 4 Done Sent Done false
28701 8379 Acked Read 2 Done Sent Done false
28702 8380 Acked Flush 0 Done Sent Done false
28703 8381 Acked Flush 0 Done Sent Done false
28704 8382 Acked Write 5 Done Sent Done false
28705 8383 Acked Write 4 Done Sent Done false
28706 8384 Acked Flush 0 Done Sent Done false
28707 8385 Acked Read 6 Done Sent Done false
28708 8386 Acked Read 7 Done Sent Done false
28709 8387 Acked Write 7 Done Sent Done false
28710 8388 Acked Write 10 Done Sent Done false
28711 8389 Acked Write 8 Done Sent Done false
28712 8390 Acked Write 6 Done Sent Done false
28713 8391 Acked Read 7 Done Sent Done false
28714 8392 Acked Read 8 Done Sent Done false
28715 8393 Acked Read 1 Done Sent Done false
28716 8394 Acked Flush 0 Done Sent Done false
28717 8395 Acked Read 10 Done Sent Done false
28718 8396 Acked Write 4 Done Sent Done false
28719 8397 Acked Read 8 Done Sent Done false
28720 8398 Acked Write 8 Done Sent Done false
28721 8399 Acked Read 7 Done Sent Done false
28722 8400 Acked Write 3 Done Sent Done false
28723 8401 Acked Write 1 Done Sent Done false
28724 8402 Acked Read 2 Done Sent Done false
28725 8403 Acked Read 10 Done Sent Done false
28726 8404 Acked Write 4 Done Sent Done false
28727 8405 Acked Read 8 Done Sent Done false
28728 8406 Acked Write 6 Done Sent Done false
28729 8407 Acked Write 5 Done Sent Done false
28730 8408 Acked Write 5 Done Sent Done false
28731 8409 Acked Write 2 Done Sent Done false
28732 8410 Acked Write 3 Done Sent Done false
28733 8411 Acked Read 9 Done Sent Done false
28734 8412 Acked Read 5 Done Sent Done false
28735 8413 Acked Write 7 Done Sent Done false
28736 8414 Acked Read 8 Done Sent Done false
28737 8415 Acked Write 6 Done Sent Done false
28738 8416 Acked Write 6 Done Sent Done false
28739 8417 Acked Write 1 Done Sent Done false
28740 8418 Acked Write 3 Done Sent Done false
28741 8419 Acked Write 10 Done Sent Done false
28742 8420 Acked Read 4 Done Sent Done false
28743 8421 Acked Flush 0 Done Sent Done false
28744 8422 Acked Read 9 Done Sent Done false
28745 8423 Acked Flush 0 Done Sent Done false
28746 8424 Acked Write 7 Done Sent Done false
28747 8425 Acked Read 10 Done Sent Done false
28748 8426 Acked Write 8 Done Sent Done false
28749 8427 Acked Read 10 Done Sent Done false
28750 8428 Acked Read 2 Done Sent Done false
28751 8429 Acked Write 1 Done Sent Done false
28752 8430 Acked Write 2 Done Sent Done false
28753 8431 Acked Write 2 Done Sent Done false
28754 8432 Acked Write 2 Done Sent Done false
28755 8433 Acked Read 10 Done Sent Done false
28756 8434 Acked Read 10 Done Sent Done false
28757 8435 Acked Write 10 Done Sent Done false
28758 8436 Acked Read 4 Done Sent Done false
28759 8437 Acked Write 2 Done Sent Done false
28760 8438 Acked Read 8 Done Sent Done false
28761 8439 Acked Read 7 Done Sent Done false
28762 8440 Acked Write 1 Done Sent Done false
28763 8441 Acked Read 4 Done Sent Done false
28764 8442 Acked Read 2 Done Sent Done false
28765 8443 Acked Read 4 Done Sent Done false
28766 8444 Acked Read 7 Done Sent Done false
28767 8445 Acked Write 5 Done Sent Done false
28768 8446 Acked Read 6 Done Sent Done false
28769 8447 Acked Read 4 Done Sent Done false
28770 8448 Acked Read 8 Done Sent Done false
28771 8449 Acked Read 3 Done Sent Done false
28772 8450 Acked Write 9 Done Sent Done false
28773 8451 Acked Read 3 Done Sent Done false
28774 8452 Acked Flush 0 Done Sent Done false
28775 8453 Acked Write 7 Done Sent Done false
28776 8454 Acked Write 2 Done Sent Done false
28777 8455 Acked Read 4 Done Sent Done false
28778 8456 Acked Write 4 Done Sent Done false
28779 8457 Acked Read 7 Done Sent Done false
28780 8458 Acked Write 1 Done Sent Done false
28781 8459 Acked Write 1 Done Sent Done false
28782 8460 Acked Read 2 Done Sent Done false
28783 8461 Acked Read 8 Done Sent Done false
28784 8462 Acked Write 2 Done Sent Done false
28785 8463 Acked Write 7 Done Sent Done false
28786 8464 Acked Read 10 Done Sent Done false
28787 8465 Acked Read 9 Done Sent Done false
28788 8466 Acked Read 4 Done Sent Done false
28789 8467 Acked Read 2 Done Sent Done false
28790 8468 Acked Read 9 Done Sent Done false
28791 8469 Acked Read 7 Done Sent Done false
28792 8470 Acked Write 5 Done Sent Done false
28793 8471 Acked Read 8 Done Sent Done false
28794 8472 Acked Write 4 Done Sent Done false
28795 8473 Acked Read 1 Done Sent Done false
28796 8474 Acked Read 8 Done Sent Done false
28797 8475 Acked Read 10 Done Sent Done false
28798 8476 Acked Write 6 Done Sent Done false
28799 8477 Acked Write 9 Done Sent Done false
28800 8478 Acked Read 10 Done Sent Done false
28801 8479 Acked Read 2 Done Sent Done false
28802 8480 Acked Read 1 Done Sent Done false
28803 8481 Acked Write 1 Done Sent Done false
28804 8482 Acked Read 2 Done Sent Done false
28805 8483 Acked Write 8 Done Sent Done false
28806 8484 Acked Write 6 Done Sent Done false
28807 8485 Acked Flush 0 Done Sent Done false
28808 8486 Acked Read 7 Done Sent Done false
28809 8487 Acked Write 9 Done Sent Done false
28810 8488 Acked Read 1 Done Sent Done false
28811 8489 Acked Write 8 Done Sent Done false
28812 8490 Acked Write 9 Done Sent Done false
28813 8491 Acked Read 10 Done Sent Done false
28814 8492 Acked Write 9 Done Sent Done false
28815 8493 Acked Write 10 Done Sent Done false
28816 8494 Acked Write 2 Done Sent Done false
28817 8495 Acked Read 2 Done Sent Done false
28818 8496 Acked Write 5 Done Sent Done false
28819 8497 Acked Read 9 Done Sent Done false
28820 8498 Acked Write 1 Done Sent Done false
28821 8499 Acked Read 3 Done Sent Done false
28822 8500 Acked Write 8 Done Sent Done false
28823 8501 Acked Read 9 Done Sent Done false
28824 8502 Acked Write 3 Done Sent Done false
28825 8503 Acked Write 2 Done Sent Done false
28826 8504 Acked Read 3 Done Sent Done false
28827 8505 Acked Read 7 Done Sent Done false
28828 8506 Acked Write 9 Done Sent Done false
28829 8507 Acked Read 3 Done Sent Done false
28830 8508 Acked Read 4 Done Sent Done false
28831 8509 Acked Flush 0 Done Sent Done false
28832 8510 Acked Write 8 Done Sent Done false
28833 8511 Acked Read 7 Done Sent Done false
28834 8512 Acked Flush 0 Done Sent Done false
28835 8513 Acked Read 8 Done Sent Done false
28836 8514 Acked Write 1 Done Sent Done false
28837 8515 Acked Write 5 Done Sent Done false
28838 8516 Acked Write 8 Done Sent Done false
28839 8517 Acked Read 5 Done Sent Done false
28840 8518 Acked Read 1 Done Sent Done false
28841 8519 Acked Read 4 Done Sent Done false
28842 8520 Acked Write 1 Done Sent Done false
28843 8521 Acked Read 9 Done Sent Done false
28844 8522 Acked Write 2 Done Sent Done false
28845 8523 Acked Read 7 Done Sent Done false
28846 8524 Acked Read 5 Done Sent Done false
28847 8525 Acked Write 9 Done Sent Done false
28848 8526 Acked Read 4 Done Sent Done false
28849 8527 Acked Read 9 Done Sent Done false
28850 8528 Acked Write 7 Done Sent Done false
28851 8529 Acked Write 8 Done Sent Done false
28852 8530 Acked Write 9 Done Sent Done false
28853 8531 Acked Write 7 Done Sent Done false
28854 8532 Acked Read 1 Done Sent Done false
28855 8533 Acked Read 6 Done Sent Done false
28856 8534 Acked Read 4 Done Sent Done false
28857 8535 Acked Read 10 Done Sent Done false
28858 8536 Acked Read 7 Done Sent Done false
28859 8537 Acked Write 9 Done Sent Done false
28860 8538 Acked Read 8 Done Sent Done false
28861 8539 Acked Read 2 Done Sent Done false
28862 8540 Acked Read 1 Done Sent Done false
28863 8541 Acked Read 3 Done Sent Done false
28864 8542 Acked Flush 0 Done Sent Done false
28865 8543 Acked Write 6 Done Sent Done false
28866 8544 Acked Read 3 Done Sent Done false
28867 8545 Acked Read 2 Done Sent Done false
28868 8546 Acked Flush 0 Done Sent Done false
28869 STATES DS:0 DS:1 DS:2 TOTAL
28870 Sent 0 301 0 301
28871 Done 301 0 301 602
28872 Skipped 0 0 0 0
28873 Error 0 0 0 0
28874 Last Flush: Some(JobId(8546)) Some(JobId(8245)) Some(JobId(8546))
28875 Downstairs last five completed: 8245 8244 8243 8242 8241
28876 Upstairs last five completed: 8546 8545 8544 8543 8542
28877 CLIENT: Up:0 ds:301 act:2
288782025-03-17T22:40:55.904ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
288792025-03-17T22:40:57.199ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
288802025-03-17T22:40:57.907ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
288812025-03-17T22:40:58.909ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
28882 ----------------------------------------------------------------
28883 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28884 States: Active Offline Active
28885 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28886 8246 Acked Flush 0 Done Sent Done false
28887 8247 Acked Read 4 Done Sent Done false
28888 8248 Acked Flush 0 Done Sent Done false
28889 8249 Acked Flush 0 Done Sent Done false
28890 8250 Acked Write 2 Done Sent Done false
28891 8251 Acked Read 6 Done Sent Done false
28892 8252 Acked Write 1 Done Sent Done false
28893 8253 Acked Flush 0 Done Sent Done false
28894 8254 Acked Read 5 Done Sent Done false
28895 8255 Acked Write 3 Done Sent Done false
28896 8256 Acked Write 10 Done Sent Done false
28897 8257 Acked Write 10 Done Sent Done false
28898 8258 Acked Flush 0 Done Sent Done false
28899 8259 Acked Read 10 Done Sent Done false
28900 8260 Acked Read 2 Done Sent Done false
28901 8261 Acked Read 9 Done Sent Done false
28902 8262 Acked Write 9 Done Sent Done false
28903 8263 Acked Flush 0 Done Sent Done false
28904 8264 Acked Write 3 Done Sent Done false
28905 8265 Acked Read 1 Done Sent Done false
28906 8266 Acked Read 7 Done Sent Done false
28907 8267 Acked Read 6 Done Sent Done false
28908 8268 Acked Write 3 Done Sent Done false
28909 8269 Acked Write 6 Done Sent Done false
28910 8270 Acked Write 6 Done Sent Done false
28911 8271 Acked Write 8 Done Sent Done false
28912 8272 Acked Read 6 Done Sent Done false
28913 8273 Acked Write 3 Done Sent Done false
28914 8274 Acked Read 10 Done Sent Done false
28915 8275 Acked Write 3 Done Sent Done false
28916 8276 Acked Write 7 Done Sent Done false
28917 8277 Acked Read 3 Done Sent Done false
28918 8278 Acked Read 9 Done Sent Done false
28919 8279 Acked Flush 0 Done Sent Done false
28920 8280 Acked Read 3 Done Sent Done false
28921 8281 Acked Write 2 Done Sent Done false
28922 8282 Acked Read 10 Done Sent Done false
28923 8283 Acked Read 1 Done Sent Done false
28924 8284 Acked Read 1 Done Sent Done false
28925 8285 Acked Write 2 Done Sent Done false
28926 8286 Acked Read 6 Done Sent Done false
28927 8287 Acked Write 1 Done Sent Done false
28928 8288 Acked Read 7 Done Sent Done false
28929 8289 Acked Write 3 Done Sent Done false
28930 8290 Acked Read 5 Done Sent Done false
28931 8291 Acked Read 9 Done Sent Done false
28932 8292 Acked Flush 0 Done Sent Done false
28933 8293 Acked Flush 0 Done Sent Done false
28934 8294 Acked Flush 0 Done Sent Done false
28935 8295 Acked Read 8 Done Sent Done false
28936 8296 Acked Write 8 Done Sent Done false
28937 8297 Acked Write 6 Done Sent Done false
28938 8298 Acked Write 9 Done Sent Done false
28939 8299 Acked Write 3 Done Sent Done false
28940 8300 Acked Read 1 Done Sent Done false
28941 8301 Acked Read 2 Done Sent Done false
28942 8302 Acked Write 6 Done Sent Done false
28943 8303 Acked Write 2 Done Sent Done false
28944 8304 Acked Read 6 Done Sent Done false
28945 8305 Acked Write 2 Done Sent Done false
28946 8306 Acked Write 3 Done Sent Done false
28947 8307 Acked Write 9 Done Sent Done false
28948 8308 Acked Read 1 Done Sent Done false
28949 8309 Acked Flush 0 Done Sent Done false
28950 8310 Acked Read 7 Done Sent Done false
28951 8311 Acked Write 3 Done Sent Done false
28952 8312 Acked Read 2 Done Sent Done false
28953 8313 Acked Read 1 Done Sent Done false
28954 8314 Acked Write 5 Done Sent Done false
28955 8315 Acked Read 8 Done Sent Done false
28956 8316 Acked Write 10 Done Sent Done false
28957 8317 Acked Read 2 Done Sent Done false
28958 8318 Acked Read 2 Done Sent Done false
28959 8319 Acked Write 10 Done Sent Done false
28960 8320 Acked Read 4 Done Sent Done false
28961 8321 Acked Flush 0 Done Sent Done false
28962 8322 Acked Write 5 Done Sent Done false
28963 8323 Acked Read 7 Done Sent Done false
28964 8324 Acked Read 3 Done Sent Done false
28965 8325 Acked Read 2 Done Sent Done false
28966 8326 Acked Write 3 Done Sent Done false
28967 8327 Acked Read 7 Done Sent Done false
28968 8328 Acked Read 5 Done Sent Done false
28969 8329 Acked Write 5 Done Sent Done false
28970 8330 Acked Flush 0 Done Sent Done false
28971 8331 Acked Read 7 Done Sent Done false
28972 8332 Acked Read 7 Done Sent Done false
28973 8333 Acked Read 4 Done Sent Done false
28974 8334 Acked Write 4 Done Sent Done false
28975 8335 Acked Write 9 Done Sent Done false
28976 8336 Acked Read 10 Done Sent Done false
28977 8337 Acked Read 9 Done Sent Done false
28978 8338 Acked Flush 0 Done Sent Done false
28979 8339 Acked Read 2 Done Sent Done false
28980 8340 Acked Write 1 Done Sent Done false
28981 8341 Acked Read 1 Done Sent Done false
28982 8342 Acked Read 6 Done Sent Done false
28983 8343 Acked Write 9 Done Sent Done false
28984 8344 Acked Read 8 Done Sent Done false
28985 8345 Acked Read 5 Done Sent Done false
28986 8346 Acked Write 4 Done Sent Done false
28987 8347 Acked Read 9 Done Sent Done false
28988 8348 Acked Write 6 Done Sent Done false
28989 8349 Acked Read 8 Done Sent Done false
28990 8350 Acked Write 4 Done Sent Done false
28991 8351 Acked Write 7 Done Sent Done false
28992 8352 Acked Write 5 Done Sent Done false
28993 8353 Acked Read 5 Done Sent Done false
28994 8354 Acked Write 5 Done Sent Done false
28995 8355 Acked Read 8 Done Sent Done false
28996 8356 Acked Read 2 Done Sent Done false
28997 8357 Acked Flush 0 Done Sent Done false
28998 8358 Acked Read 10 Done Sent Done false
28999 8359 Acked Read 7 Done Sent Done false
29000 8360 Acked Write 3 Done Sent Done false
29001 8361 Acked Read 8 Done Sent Done false
29002 8362 Acked Write 6 Done Sent Done false
29003 8363 Acked Read 6 Done Sent Done false
29004 8364 Acked Write 7 Done Sent Done false
29005 8365 Acked Write 3 Done Sent Done false
29006 8366 Acked Write 6 Done Sent Done false
29007 8367 Acked Write 3 Done Sent Done false
29008 8368 Acked Read 5 Done Sent Done false
29009 8369 Acked Write 9 Done Sent Done false
29010 8370 Acked Write 3 Done Sent Done false
29011 8371 Acked Read 1 Done Sent Done false
29012 8372 Acked Write 8 Done Sent Done false
29013 8373 Acked Read 9 Done Sent Done false
29014 8374 Acked Write 3 Done Sent Done false
29015 8375 Acked Write 8 Done Sent Done false
29016 8376 Acked Write 5 Done Sent Done false
29017 8377 Acked Read 4 Done Sent Done false
29018 8378 Acked Write 4 Done Sent Done false
29019 8379 Acked Read 2 Done Sent Done false
29020 8380 Acked Flush 0 Done Sent Done false
29021 8381 Acked Flush 0 Done Sent Done false
29022 8382 Acked Write 5 Done Sent Done false
29023 8383 Acked Write 4 Done Sent Done false
29024 8384 Acked Flush 0 Done Sent Done false
29025 8385 Acked Read 6 Done Sent Done false
29026 8386 Acked Read 7 Done Sent Done false
29027 8387 Acked Write 7 Done Sent Done false
29028 8388 Acked Write 10 Done Sent Done false
29029 8389 Acked Write 8 Done Sent Done false
29030 8390 Acked Write 6 Done Sent Done false
29031 8391 Acked Read 7 Done Sent Done false
29032 8392 Acked Read 8 Done Sent Done false
29033 8393 Acked Read 1 Done Sent Done false
29034 8394 Acked Flush 0 Done Sent Done false
29035 8395 Acked Read 10 Done Sent Done false
29036 8396 Acked Write 4 Done Sent Done false
29037 8397 Acked Read 8 Done Sent Done false
29038 8398 Acked Write 8 Done Sent Done false
29039 8399 Acked Read 7 Done Sent Done false
29040 8400 Acked Write 3 Done Sent Done false
29041 8401 Acked Write 1 Done Sent Done false
29042 8402 Acked Read 2 Done Sent Done false
29043 8403 Acked Read 10 Done Sent Done false
29044 8404 Acked Write 4 Done Sent Done false
29045 8405 Acked Read 8 Done Sent Done false
29046 8406 Acked Write 6 Done Sent Done false
29047 8407 Acked Write 5 Done Sent Done false
29048 8408 Acked Write 5 Done Sent Done false
29049 8409 Acked Write 2 Done Sent Done false
29050 8410 Acked Write 3 Done Sent Done false
29051 8411 Acked Read 9 Done Sent Done false
29052 8412 Acked Read 5 Done Sent Done false
29053 8413 Acked Write 7 Done Sent Done false
29054 8414 Acked Read 8 Done Sent Done false
29055 8415 Acked Write 6 Done Sent Done false
29056 8416 Acked Write 6 Done Sent Done false
29057 8417 Acked Write 1 Done Sent Done false
29058 8418 Acked Write 3 Done Sent Done false
29059 8419 Acked Write 10 Done Sent Done false
29060 8420 Acked Read 4 Done Sent Done false
29061 8421 Acked Flush 0 Done Sent Done false
29062 8422 Acked Read 9 Done Sent Done false
29063 8423 Acked Flush 0 Done Sent Done false
29064 8424 Acked Write 7 Done Sent Done false
29065 8425 Acked Read 10 Done Sent Done false
29066 8426 Acked Write 8 Done Sent Done false
29067 8427 Acked Read 10 Done Sent Done false
29068 8428 Acked Read 2 Done Sent Done false
29069 8429 Acked Write 1 Done Sent Done false
29070 8430 Acked Write 2 Done Sent Done false
29071 8431 Acked Write 2 Done Sent Done false
29072 8432 Acked Write 2 Done Sent Done false
29073 8433 Acked Read 10 Done Sent Done false
29074 8434 Acked Read 10 Done Sent Done false
29075 8435 Acked Write 10 Done Sent Done false
29076 8436 Acked Read 4 Done Sent Done false
29077 8437 Acked Write 2 Done Sent Done false
29078 8438 Acked Read 8 Done Sent Done false
29079 8439 Acked Read 7 Done Sent Done false
29080 8440 Acked Write 1 Done Sent Done false
29081 8441 Acked Read 4 Done Sent Done false
29082 8442 Acked Read 2 Done Sent Done false
29083 8443 Acked Read 4 Done Sent Done false
29084 8444 Acked Read 7 Done Sent Done false
29085 8445 Acked Write 5 Done Sent Done false
29086 8446 Acked Read 6 Done Sent Done false
29087 8447 Acked Read 4 Done Sent Done false
29088 8448 Acked Read 8 Done Sent Done false
29089 8449 Acked Read 3 Done Sent Done false
29090 8450 Acked Write 9 Done Sent Done false
29091 8451 Acked Read 3 Done Sent Done false
29092 8452 Acked Flush 0 Done Sent Done false
29093 8453 Acked Write 7 Done Sent Done false
29094 8454 Acked Write 2 Done Sent Done false
29095 8455 Acked Read 4 Done Sent Done false
29096 8456 Acked Write 4 Done Sent Done false
29097 8457 Acked Read 7 Done Sent Done false
29098 8458 Acked Write 1 Done Sent Done false
29099 8459 Acked Write 1 Done Sent Done false
29100 8460 Acked Read 2 Done Sent Done false
29101 8461 Acked Read 8 Done Sent Done false
29102 8462 Acked Write 2 Done Sent Done false
29103 8463 Acked Write 7 Done Sent Done false
29104 8464 Acked Read 10 Done Sent Done false
29105 8465 Acked Read 9 Done Sent Done false
29106 8466 Acked Read 4 Done Sent Done false
29107 8467 Acked Read 2 Done Sent Done false
29108 8468 Acked Read 9 Done Sent Done false
29109 8469 Acked Read 7 Done Sent Done false
29110 8470 Acked Write 5 Done Sent Done false
29111 8471 Acked Read 8 Done Sent Done false
29112 8472 Acked Write 4 Done Sent Done false
29113 8473 Acked Read 1 Done Sent Done false
29114 8474 Acked Read 8 Done Sent Done false
29115 8475 Acked Read 10 Done Sent Done false
29116 8476 Acked Write 6 Done Sent Done false
29117 8477 Acked Write 9 Done Sent Done false
29118 8478 Acked Read 10 Done Sent Done false
29119 8479 Acked Read 2 Done Sent Done false
29120 8480 Acked Read 1 Done Sent Done false
29121 8481 Acked Write 1 Done Sent Done false
29122 8482 Acked Read 2 Done Sent Done false
29123 8483 Acked Write 8 Done Sent Done false
29124 8484 Acked Write 6 Done Sent Done false
29125 8485 Acked Flush 0 Done Sent Done false
29126 8486 Acked Read 7 Done Sent Done false
29127 8487 Acked Write 9 Done Sent Done false
29128 8488 Acked Read 1 Done Sent Done false
29129 8489 Acked Write 8 Done Sent Done false
29130 8490 Acked Write 9 Done Sent Done false
29131 8491 Acked Read 10 Done Sent Done false
29132 8492 Acked Write 9 Done Sent Done false
29133 8493 Acked Write 10 Done Sent Done false
29134 8494 Acked Write 2 Done Sent Done false
29135 8495 Acked Read 2 Done Sent Done false
29136 8496 Acked Write 5 Done Sent Done false
29137 8497 Acked Read 9 Done Sent Done false
29138 8498 Acked Write 1 Done Sent Done false
29139 8499 Acked Read 3 Done Sent Done false
29140 8500 Acked Write 8 Done Sent Done false
29141 8501 Acked Read 9 Done Sent Done false
29142 8502 Acked Write 3 Done Sent Done false
29143 8503 Acked Write 2 Done Sent Done false
29144 8504 Acked Read 3 Done Sent Done false
29145 8505 Acked Read 7 Done Sent Done false
29146 8506 Acked Write 9 Done Sent Done false
29147 8507 Acked Read 3 Done Sent Done false
29148 8508 Acked Read 4 Done Sent Done false
29149 8509 Acked Flush 0 Done Sent Done false
29150 8510 Acked Write 8 Done Sent Done false
29151 8511 Acked Read 7 Done Sent Done false
29152 8512 Acked Flush 0 Done Sent Done false
29153 8513 Acked Read 8 Done Sent Done false
29154 8514 Acked Write 1 Done Sent Done false
29155 8515 Acked Write 5 Done Sent Done false
29156 8516 Acked Write 8 Done Sent Done false
29157 8517 Acked Read 5 Done Sent Done false
29158 8518 Acked Read 1 Done Sent Done false
29159 8519 Acked Read 4 Done Sent Done false
29160 8520 Acked Write 1 Done Sent Done false
29161 8521 Acked Read 9 Done Sent Done false
29162 8522 Acked Write 2 Done Sent Done false
29163 8523 Acked Read 7 Done Sent Done false
29164 8524 Acked Read 5 Done Sent Done false
29165 8525 Acked Write 9 Done Sent Done false
29166 8526 Acked Read 4 Done Sent Done false
29167 8527 Acked Read 9 Done Sent Done false
29168 8528 Acked Write 7 Done Sent Done false
29169 8529 Acked Write 8 Done Sent Done false
29170 8530 Acked Write 9 Done Sent Done false
29171 8531 Acked Write 7 Done Sent Done false
29172 8532 Acked Read 1 Done Sent Done false
29173 8533 Acked Read 6 Done Sent Done false
29174 8534 Acked Read 4 Done Sent Done false
29175 8535 Acked Read 10 Done Sent Done false
29176 8536 Acked Read 7 Done Sent Done false
29177 8537 Acked Write 9 Done Sent Done false
29178 8538 Acked Read 8 Done Sent Done false
29179 8539 Acked Read 2 Done Sent Done false
29180 8540 Acked Read 1 Done Sent Done false
29181 8541 Acked Read 3 Done Sent Done false
29182 8542 Acked Flush 0 Done Sent Done false
29183 8543 Acked Write 6 Done Sent Done false
29184 8544 Acked Read 3 Done Sent Done false
29185 8545 Acked Read 2 Done Sent Done false
29186 8546 Acked Flush 0 Done Sent Done false
29187 STATES DS:0 DS:1 DS:2 TOTAL
29188 Sent 0 301 0 301
29189 Done 301 0 301 602
29190 Skipped 0 0 0 0
29191 Error 0 0 0 0
29192 Last Flush: Some(JobId(8546)) Some(JobId(8245)) Some(JobId(8546))
29193 Downstairs last five completed: 8245 8244 8243 8242 8241
29194 Upstairs last five completed: 8546 8545 8544 8543 8542
29195 CLIENT: Up:0 ds:301 act:2
291962025-03-17T22:40:59.911ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
291972025-03-17T22:41:00.913ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
291982025-03-17T22:41:00.933ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
291992025-03-17T22:41:00.933ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
292002025-03-17T22:41:00.933ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
292012025-03-17T22:41:00.933ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
292022025-03-17T22:41:00.934ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
29203 [8820] got start action
29204 [8820] I am going to restart
29205 Starting downstairs at port 8820
29206 Make output file at "/tmp/dsc/downstairs-8820.txt"
29207 [8820][1] reports Starting
29208 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
29209 [8820][1] reports Running
292102025-03-17T22:41:02.200ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
292112025-03-17T22:41:02.915ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
29212 ----------------------------------------------------------------
29213 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29214 States: Active Offline Active
29215 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29216 8246 Acked Flush 0 Done Sent Done false
29217 8247 Acked Read 4 Done Sent Done false
29218 8248 Acked Flush 0 Done Sent Done false
29219 8249 Acked Flush 0 Done Sent Done false
29220 8250 Acked Write 2 Done Sent Done false
29221 8251 Acked Read 6 Done Sent Done false
29222 8252 Acked Write 1 Done Sent Done false
29223 8253 Acked Flush 0 Done Sent Done false
29224 8254 Acked Read 5 Done Sent Done false
29225 8255 Acked Write 3 Done Sent Done false
29226 8256 Acked Write 10 Done Sent Done false
29227 8257 Acked Write 10 Done Sent Done false
29228 8258 Acked Flush 0 Done Sent Done false
29229 8259 Acked Read 10 Done Sent Done false
29230 8260 Acked Read 2 Done Sent Done false
29231 8261 Acked Read 9 Done Sent Done false
29232 8262 Acked Write 9 Done Sent Done false
29233 8263 Acked Flush 0 Done Sent Done false
29234 8264 Acked Write 3 Done Sent Done false
29235 8265 Acked Read 1 Done Sent Done false
29236 8266 Acked Read 7 Done Sent Done false
29237 8267 Acked Read 6 Done Sent Done false
29238 8268 Acked Write 3 Done Sent Done false
29239 8269 Acked Write 6 Done Sent Done false
29240 8270 Acked Write 6 Done Sent Done false
29241 8271 Acked Write 8 Done Sent Done false
29242 8272 Acked Read 6 Done Sent Done false
29243 8273 Acked Write 3 Done Sent Done false
29244 8274 Acked Read 10 Done Sent Done false
29245 8275 Acked Write 3 Done Sent Done false
29246 8276 Acked Write 7 Done Sent Done false
29247 8277 Acked Read 3 Done Sent Done false
29248 8278 Acked Read 9 Done Sent Done false
29249 8279 Acked Flush 0 Done Sent Done false
29250 8280 Acked Read 3 Done Sent Done false
29251 8281 Acked Write 2 Done Sent Done false
29252 8282 Acked Read 10 Done Sent Done false
29253 8283 Acked Read 1 Done Sent Done false
29254 8284 Acked Read 1 Done Sent Done false
29255 8285 Acked Write 2 Done Sent Done false
29256 8286 Acked Read 6 Done Sent Done false
29257 8287 Acked Write 1 Done Sent Done false
29258 8288 Acked Read 7 Done Sent Done false
29259 8289 Acked Write 3 Done Sent Done false
29260 8290 Acked Read 5 Done Sent Done false
29261 8291 Acked Read 9 Done Sent Done false
29262 8292 Acked Flush 0 Done Sent Done false
29263 8293 Acked Flush 0 Done Sent Done false
29264 8294 Acked Flush 0 Done Sent Done false
29265 8295 Acked Read 8 Done Sent Done false
29266 8296 Acked Write 8 Done Sent Done false
29267 8297 Acked Write 6 Done Sent Done false
29268 8298 Acked Write 9 Done Sent Done false
29269 8299 Acked Write 3 Done Sent Done false
29270 8300 Acked Read 1 Done Sent Done false
29271 8301 Acked Read 2 Done Sent Done false
29272 8302 Acked Write 6 Done Sent Done false
29273 8303 Acked Write 2 Done Sent Done false
29274 8304 Acked Read 6 Done Sent Done false
29275 8305 Acked Write 2 Done Sent Done false
29276 8306 Acked Write 3 Done Sent Done false
29277 8307 Acked Write 9 Done Sent Done false
29278 8308 Acked Read 1 Done Sent Done false
29279 8309 Acked Flush 0 Done Sent Done false
29280 8310 Acked Read 7 Done Sent Done false
29281 8311 Acked Write 3 Done Sent Done false
29282 8312 Acked Read 2 Done Sent Done false
29283 8313 Acked Read 1 Done Sent Done false
29284 8314 Acked Write 5 Done Sent Done false
29285 8315 Acked Read 8 Done Sent Done false
29286 8316 Acked Write 10 Done Sent Done false
29287 8317 Acked Read 2 Done Sent Done false
29288 8318 Acked Read 2 Done Sent Done false
29289 8319 Acked Write 10 Done Sent Done false
29290 8320 Acked Read 4 Done Sent Done false
29291 8321 Acked Flush 0 Done Sent Done false
29292 8322 Acked Write 5 Done Sent Done false
29293 8323 Acked Read 7 Done Sent Done false
29294 8324 Acked Read 3 Done Sent Done false
29295 8325 Acked Read 2 Done Sent Done false
29296 8326 Acked Write 3 Done Sent Done false
29297 8327 Acked Read 7 Done Sent Done false
29298 8328 Acked Read 5 Done Sent Done false
29299 8329 Acked Write 5 Done Sent Done false
29300 8330 Acked Flush 0 Done Sent Done false
29301 8331 Acked Read 7 Done Sent Done false
29302 8332 Acked Read 7 Done Sent Done false
29303 8333 Acked Read 4 Done Sent Done false
29304 8334 Acked Write 4 Done Sent Done false
29305 8335 Acked Write 9 Done Sent Done false
29306 8336 Acked Read 10 Done Sent Done false
29307 8337 Acked Read 9 Done Sent Done false
29308 8338 Acked Flush 0 Done Sent Done false
29309 8339 Acked Read 2 Done Sent Done false
29310 8340 Acked Write 1 Done Sent Done false
29311 8341 Acked Read 1 Done Sent Done false
29312 8342 Acked Read 6 Done Sent Done false
29313 8343 Acked Write 9 Done Sent Done false
29314 8344 Acked Read 8 Done Sent Done false
29315 8345 Acked Read 5 Done Sent Done false
29316 8346 Acked Write 4 Done Sent Done false
29317 8347 Acked Read 9 Done Sent Done false
29318 8348 Acked Write 6 Done Sent Done false
29319 8349 Acked Read 8 Done Sent Done false
29320 8350 Acked Write 4 Done Sent Done false
29321 8351 Acked Write 7 Done Sent Done false
29322 8352 Acked Write 5 Done Sent Done false
29323 8353 Acked Read 5 Done Sent Done false
29324 8354 Acked Write 5 Done Sent Done false
29325 8355 Acked Read 8 Done Sent Done false
29326 8356 Acked Read 2 Done Sent Done false
29327 8357 Acked Flush 0 Done Sent Done false
29328 8358 Acked Read 10 Done Sent Done false
29329 8359 Acked Read 7 Done Sent Done false
29330 8360 Acked Write 3 Done Sent Done false
29331 8361 Acked Read 8 Done Sent Done false
29332 8362 Acked Write 6 Done Sent Done false
29333 8363 Acked Read 6 Done Sent Done false
29334 8364 Acked Write 7 Done Sent Done false
29335 8365 Acked Write 3 Done Sent Done false
29336 8366 Acked Write 6 Done Sent Done false
29337 8367 Acked Write 3 Done Sent Done false
29338 8368 Acked Read 5 Done Sent Done false
29339 8369 Acked Write 9 Done Sent Done false
29340 8370 Acked Write 3 Done Sent Done false
29341 8371 Acked Read 1 Done Sent Done false
29342 8372 Acked Write 8 Done Sent Done false
29343 8373 Acked Read 9 Done Sent Done false
29344 8374 Acked Write 3 Done Sent Done false
29345 8375 Acked Write 8 Done Sent Done false
29346 8376 Acked Write 5 Done Sent Done false
29347 8377 Acked Read 4 Done Sent Done false
29348 8378 Acked Write 4 Done Sent Done false
29349 8379 Acked Read 2 Done Sent Done false
29350 8380 Acked Flush 0 Done Sent Done false
29351 8381 Acked Flush 0 Done Sent Done false
29352 8382 Acked Write 5 Done Sent Done false
29353 8383 Acked Write 4 Done Sent Done false
29354 8384 Acked Flush 0 Done Sent Done false
29355 8385 Acked Read 6 Done Sent Done false
29356 8386 Acked Read 7 Done Sent Done false
29357 8387 Acked Write 7 Done Sent Done false
29358 8388 Acked Write 10 Done Sent Done false
29359 8389 Acked Write 8 Done Sent Done false
29360 8390 Acked Write 6 Done Sent Done false
29361 8391 Acked Read 7 Done Sent Done false
29362 8392 Acked Read 8 Done Sent Done false
29363 8393 Acked Read 1 Done Sent Done false
29364 8394 Acked Flush 0 Done Sent Done false
29365 8395 Acked Read 10 Done Sent Done false
29366 8396 Acked Write 4 Done Sent Done false
29367 8397 Acked Read 8 Done Sent Done false
29368 8398 Acked Write 8 Done Sent Done false
29369 8399 Acked Read 7 Done Sent Done false
29370 8400 Acked Write 3 Done Sent Done false
29371 8401 Acked Write 1 Done Sent Done false
29372 8402 Acked Read 2 Done Sent Done false
29373 8403 Acked Read 10 Done Sent Done false
29374 8404 Acked Write 4 Done Sent Done false
29375 8405 Acked Read 8 Done Sent Done false
29376 8406 Acked Write 6 Done Sent Done false
29377 8407 Acked Write 5 Done Sent Done false
29378 8408 Acked Write 5 Done Sent Done false
29379 8409 Acked Write 2 Done Sent Done false
29380 8410 Acked Write 3 Done Sent Done false
29381 8411 Acked Read 9 Done Sent Done false
29382 8412 Acked Read 5 Done Sent Done false
29383 8413 Acked Write 7 Done Sent Done false
29384 8414 Acked Read 8 Done Sent Done false
29385 8415 Acked Write 6 Done Sent Done false
29386 8416 Acked Write 6 Done Sent Done false
29387 8417 Acked Write 1 Done Sent Done false
29388 8418 Acked Write 3 Done Sent Done false
29389 8419 Acked Write 10 Done Sent Done false
29390 8420 Acked Read 4 Done Sent Done false
29391 8421 Acked Flush 0 Done Sent Done false
29392 8422 Acked Read 9 Done Sent Done false
29393 8423 Acked Flush 0 Done Sent Done false
29394 8424 Acked Write 7 Done Sent Done false
29395 8425 Acked Read 10 Done Sent Done false
29396 8426 Acked Write 8 Done Sent Done false
29397 8427 Acked Read 10 Done Sent Done false
29398 8428 Acked Read 2 Done Sent Done false
29399 8429 Acked Write 1 Done Sent Done false
29400 8430 Acked Write 2 Done Sent Done false
29401 8431 Acked Write 2 Done Sent Done false
29402 8432 Acked Write 2 Done Sent Done false
29403 8433 Acked Read 10 Done Sent Done false
29404 8434 Acked Read 10 Done Sent Done false
29405 8435 Acked Write 10 Done Sent Done false
29406 8436 Acked Read 4 Done Sent Done false
29407 8437 Acked Write 2 Done Sent Done false
29408 8438 Acked Read 8 Done Sent Done false
29409 8439 Acked Read 7 Done Sent Done false
29410 8440 Acked Write 1 Done Sent Done false
29411 8441 Acked Read 4 Done Sent Done false
29412 8442 Acked Read 2 Done Sent Done false
29413 8443 Acked Read 4 Done Sent Done false
29414 8444 Acked Read 7 Done Sent Done false
29415 8445 Acked Write 5 Done Sent Done false
29416 8446 Acked Read 6 Done Sent Done false
29417 8447 Acked Read 4 Done Sent Done false
29418 8448 Acked Read 8 Done Sent Done false
29419 8449 Acked Read 3 Done Sent Done false
29420 8450 Acked Write 9 Done Sent Done false
29421 8451 Acked Read 3 Done Sent Done false
29422 8452 Acked Flush 0 Done Sent Done false
29423 8453 Acked Write 7 Done Sent Done false
29424 8454 Acked Write 2 Done Sent Done false
29425 8455 Acked Read 4 Done Sent Done false
29426 8456 Acked Write 4 Done Sent Done false
29427 8457 Acked Read 7 Done Sent Done false
29428 8458 Acked Write 1 Done Sent Done false
29429 8459 Acked Write 1 Done Sent Done false
29430 8460 Acked Read 2 Done Sent Done false
29431 8461 Acked Read 8 Done Sent Done false
29432 8462 Acked Write 2 Done Sent Done false
29433 8463 Acked Write 7 Done Sent Done false
29434 8464 Acked Read 10 Done Sent Done false
29435 8465 Acked Read 9 Done Sent Done false
29436 8466 Acked Read 4 Done Sent Done false
29437 8467 Acked Read 2 Done Sent Done false
29438 8468 Acked Read 9 Done Sent Done false
29439 8469 Acked Read 7 Done Sent Done false
29440 8470 Acked Write 5 Done Sent Done false
29441 8471 Acked Read 8 Done Sent Done false
29442 8472 Acked Write 4 Done Sent Done false
29443 8473 Acked Read 1 Done Sent Done false
29444 8474 Acked Read 8 Done Sent Done false
29445 8475 Acked Read 10 Done Sent Done false
29446 8476 Acked Write 6 Done Sent Done false
29447 8477 Acked Write 9 Done Sent Done false
29448 8478 Acked Read 10 Done Sent Done false
29449 8479 Acked Read 2 Done Sent Done false
29450 8480 Acked Read 1 Done Sent Done false
29451 8481 Acked Write 1 Done Sent Done false
29452 8482 Acked Read 2 Done Sent Done false
29453 8483 Acked Write 8 Done Sent Done false
29454 8484 Acked Write 6 Done Sent Done false
29455 8485 Acked Flush 0 Done Sent Done false
29456 8486 Acked Read 7 Done Sent Done false
29457 8487 Acked Write 9 Done Sent Done false
29458 8488 Acked Read 1 Done Sent Done false
29459 8489 Acked Write 8 Done Sent Done false
29460 8490 Acked Write 9 Done Sent Done false
29461 8491 Acked Read 10 Done Sent Done false
29462 8492 Acked Write 9 Done Sent Done false
29463 8493 Acked Write 10 Done Sent Done false
29464 8494 Acked Write 2 Done Sent Done false
29465 8495 Acked Read 2 Done Sent Done false
29466 8496 Acked Write 5 Done Sent Done false
29467 8497 Acked Read 9 Done Sent Done false
29468 8498 Acked Write 1 Done Sent Done false
29469 8499 Acked Read 3 Done Sent Done false
29470 8500 Acked Write 8 Done Sent Done false
29471 8501 Acked Read 9 Done Sent Done false
29472 8502 Acked Write 3 Done Sent Done false
29473 8503 Acked Write 2 Done Sent Done false
29474 8504 Acked Read 3 Done Sent Done false
29475 8505 Acked Read 7 Done Sent Done false
29476 8506 Acked Write 9 Done Sent Done false
29477 8507 Acked Read 3 Done Sent Done false
29478 8508 Acked Read 4 Done Sent Done false
29479 8509 Acked Flush 0 Done Sent Done false
29480 8510 Acked Write 8 Done Sent Done false
29481 8511 Acked Read 7 Done Sent Done false
29482 8512 Acked Flush 0 Done Sent Done false
29483 8513 Acked Read 8 Done Sent Done false
29484 8514 Acked Write 1 Done Sent Done false
29485 8515 Acked Write 5 Done Sent Done false
29486 8516 Acked Write 8 Done Sent Done false
29487 8517 Acked Read 5 Done Sent Done false
29488 8518 Acked Read 1 Done Sent Done false
29489 8519 Acked Read 4 Done Sent Done false
29490 8520 Acked Write 1 Done Sent Done false
29491 8521 Acked Read 9 Done Sent Done false
29492 8522 Acked Write 2 Done Sent Done false
29493 8523 Acked Read 7 Done Sent Done false
29494 8524 Acked Read 5 Done Sent Done false
29495 8525 Acked Write 9 Done Sent Done false
29496 8526 Acked Read 4 Done Sent Done false
29497 8527 Acked Read 9 Done Sent Done false
29498 8528 Acked Write 7 Done Sent Done false
29499 8529 Acked Write 8 Done Sent Done false
29500 8530 Acked Write 9 Done Sent Done false
29501 8531 Acked Write 7 Done Sent Done false
29502 8532 Acked Read 1 Done Sent Done false
29503 8533 Acked Read 6 Done Sent Done false
29504 8534 Acked Read 4 Done Sent Done false
29505 8535 Acked Read 10 Done Sent Done false
29506 8536 Acked Read 7 Done Sent Done false
29507 8537 Acked Write 9 Done Sent Done false
29508 8538 Acked Read 8 Done Sent Done false
29509 8539 Acked Read 2 Done Sent Done false
29510 8540 Acked Read 1 Done Sent Done false
29511 8541 Acked Read 3 Done Sent Done false
29512 8542 Acked Flush 0 Done Sent Done false
29513 8543 Acked Write 6 Done Sent Done false
29514 8544 Acked Read 3 Done Sent Done false
29515 8545 Acked Read 2 Done Sent Done false
29516 8546 Acked Flush 0 Done Sent Done false
29517 STATES DS:0 DS:1 DS:2 TOTAL
29518 Sent 0 301 0 301
29519 Done 301 0 301 602
29520 Skipped 0 0 0 0
29521 Error 0 0 0 0
29522 Last Flush: Some(JobId(8546)) Some(JobId(8245)) Some(JobId(8546))
29523 Downstairs last five completed: 8245 8244 8243 8242 8241
29524 Upstairs last five completed: 8546 8545 8544 8543 8542
29525 CLIENT: Up:0 ds:301 act:2
295262025-03-17T22:41:03.916ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
295272025-03-17T22:41:04.917ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
295282025-03-17T22:41:05.919ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
29529 ----------------------------------------------------------------
29530 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29531 States: Active Offline Active
29532 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29533 8246 Acked Flush 0 Done Sent Done false
29534 8247 Acked Read 4 Done Sent Done false
29535 8248 Acked Flush 0 Done Sent Done false
29536 8249 Acked Flush 0 Done Sent Done false
29537 8250 Acked Write 2 Done Sent Done false
29538 8251 Acked Read 6 Done Sent Done false
29539 8252 Acked Write 1 Done Sent Done false
29540 8253 Acked Flush 0 Done Sent Done false
29541 8254 Acked Read 5 Done Sent Done false
29542 8255 Acked Write 3 Done Sent Done false
29543 8256 Acked Write 10 Done Sent Done false
29544 8257 Acked Write 10 Done Sent Done false
29545 8258 Acked Flush 0 Done Sent Done false
29546 8259 Acked Read 10 Done Sent Done false
29547 8260 Acked Read 2 Done Sent Done false
29548 8261 Acked Read 9 Done Sent Done false
29549 8262 Acked Write 9 Done Sent Done false
29550 8263 Acked Flush 0 Done Sent Done false
29551 8264 Acked Write 3 Done Sent Done false
29552 8265 Acked Read 1 Done Sent Done false
29553 8266 Acked Read 7 Done Sent Done false
29554 8267 Acked Read 6 Done Sent Done false
29555 8268 Acked Write 3 Done Sent Done false
29556 8269 Acked Write 6 Done Sent Done false
29557 8270 Acked Write 6 Done Sent Done false
29558 8271 Acked Write 8 Done Sent Done false
29559 8272 Acked Read 6 Done Sent Done false
29560 8273 Acked Write 3 Done Sent Done false
29561 8274 Acked Read 10 Done Sent Done false
29562 8275 Acked Write 3 Done Sent Done false
29563 8276 Acked Write 7 Done Sent Done false
29564 8277 Acked Read 3 Done Sent Done false
29565 8278 Acked Read 9 Done Sent Done false
29566 8279 Acked Flush 0 Done Sent Done false
29567 8280 Acked Read 3 Done Sent Done false
29568 8281 Acked Write 2 Done Sent Done false
29569 8282 Acked Read 10 Done Sent Done false
29570 8283 Acked Read 1 Done Sent Done false
29571 8284 Acked Read 1 Done Sent Done false
29572 8285 Acked Write 2 Done Sent Done false
29573 8286 Acked Read 6 Done Sent Done false
29574 8287 Acked Write 1 Done Sent Done false
29575 8288 Acked Read 7 Done Sent Done false
29576 8289 Acked Write 3 Done Sent Done false
29577 8290 Acked Read 5 Done Sent Done false
29578 8291 Acked Read 9 Done Sent Done false
29579 8292 Acked Flush 0 Done Sent Done false
29580 8293 Acked Flush 0 Done Sent Done false
29581 8294 Acked Flush 0 Done Sent Done false
29582 8295 Acked Read 8 Done Sent Done false
29583 8296 Acked Write 8 Done Sent Done false
29584 8297 Acked Write 6 Done Sent Done false
29585 8298 Acked Write 9 Done Sent Done false
29586 8299 Acked Write 3 Done Sent Done false
29587 8300 Acked Read 1 Done Sent Done false
29588 8301 Acked Read 2 Done Sent Done false
29589 8302 Acked Write 6 Done Sent Done false
29590 8303 Acked Write 2 Done Sent Done false
29591 8304 Acked Read 6 Done Sent Done false
29592 8305 Acked Write 2 Done Sent Done false
29593 8306 Acked Write 3 Done Sent Done false
29594 8307 Acked Write 9 Done Sent Done false
29595 8308 Acked Read 1 Done Sent Done false
29596 8309 Acked Flush 0 Done Sent Done false
29597 8310 Acked Read 7 Done Sent Done false
29598 8311 Acked Write 3 Done Sent Done false
29599 8312 Acked Read 2 Done Sent Done false
29600 8313 Acked Read 1 Done Sent Done false
29601 8314 Acked Write 5 Done Sent Done false
29602 8315 Acked Read 8 Done Sent Done false
29603 8316 Acked Write 10 Done Sent Done false
29604 8317 Acked Read 2 Done Sent Done false
29605 8318 Acked Read 2 Done Sent Done false
29606 8319 Acked Write 10 Done Sent Done false
29607 8320 Acked Read 4 Done Sent Done false
29608 8321 Acked Flush 0 Done Sent Done false
29609 8322 Acked Write 5 Done Sent Done false
29610 8323 Acked Read 7 Done Sent Done false
29611 8324 Acked Read 3 Done Sent Done false
29612 8325 Acked Read 2 Done Sent Done false
29613 8326 Acked Write 3 Done Sent Done false
29614 8327 Acked Read 7 Done Sent Done false
29615 8328 Acked Read 5 Done Sent Done false
29616 8329 Acked Write 5 Done Sent Done false
29617 8330 Acked Flush 0 Done Sent Done false
29618 8331 Acked Read 7 Done Sent Done false
29619 8332 Acked Read 7 Done Sent Done false
29620 8333 Acked Read 4 Done Sent Done false
29621 8334 Acked Write 4 Done Sent Done false
29622 8335 Acked Write 9 Done Sent Done false
29623 8336 Acked Read 10 Done Sent Done false
29624 8337 Acked Read 9 Done Sent Done false
29625 8338 Acked Flush 0 Done Sent Done false
29626 8339 Acked Read 2 Done Sent Done false
29627 8340 Acked Write 1 Done Sent Done false
29628 8341 Acked Read 1 Done Sent Done false
29629 8342 Acked Read 6 Done Sent Done false
29630 8343 Acked Write 9 Done Sent Done false
29631 8344 Acked Read 8 Done Sent Done false
29632 8345 Acked Read 5 Done Sent Done false
29633 8346 Acked Write 4 Done Sent Done false
29634 8347 Acked Read 9 Done Sent Done false
29635 8348 Acked Write 6 Done Sent Done false
29636 8349 Acked Read 8 Done Sent Done false
29637 8350 Acked Write 4 Done Sent Done false
29638 8351 Acked Write 7 Done Sent Done false
29639 8352 Acked Write 5 Done Sent Done false
29640 8353 Acked Read 5 Done Sent Done false
29641 8354 Acked Write 5 Done Sent Done false
29642 8355 Acked Read 8 Done Sent Done false
29643 8356 Acked Read 2 Done Sent Done false
29644 8357 Acked Flush 0 Done Sent Done false
29645 8358 Acked Read 10 Done Sent Done false
29646 8359 Acked Read 7 Done Sent Done false
29647 8360 Acked Write 3 Done Sent Done false
29648 8361 Acked Read 8 Done Sent Done false
29649 8362 Acked Write 6 Done Sent Done false
29650 8363 Acked Read 6 Done Sent Done false
29651 8364 Acked Write 7 Done Sent Done false
29652 8365 Acked Write 3 Done Sent Done false
29653 8366 Acked Write 6 Done Sent Done false
29654 8367 Acked Write 3 Done Sent Done false
29655 8368 Acked Read 5 Done Sent Done false
29656 8369 Acked Write 9 Done Sent Done false
29657 8370 Acked Write 3 Done Sent Done false
29658 8371 Acked Read 1 Done Sent Done false
29659 8372 Acked Write 8 Done Sent Done false
29660 8373 Acked Read 9 Done Sent Done false
29661 8374 Acked Write 3 Done Sent Done false
29662 8375 Acked Write 8 Done Sent Done false
29663 8376 Acked Write 5 Done Sent Done false
29664 8377 Acked Read 4 Done Sent Done false
29665 8378 Acked Write 4 Done Sent Done false
29666 8379 Acked Read 2 Done Sent Done false
29667 8380 Acked Flush 0 Done Sent Done false
29668 8381 Acked Flush 0 Done Sent Done false
29669 8382 Acked Write 5 Done Sent Done false
29670 8383 Acked Write 4 Done Sent Done false
29671 8384 Acked Flush 0 Done Sent Done false
29672 8385 Acked Read 6 Done Sent Done false
29673 8386 Acked Read 7 Done Sent Done false
29674 8387 Acked Write 7 Done Sent Done false
29675 8388 Acked Write 10 Done Sent Done false
29676 8389 Acked Write 8 Done Sent Done false
29677 8390 Acked Write 6 Done Sent Done false
29678 8391 Acked Read 7 Done Sent Done false
29679 8392 Acked Read 8 Done Sent Done false
29680 8393 Acked Read 1 Done Sent Done false
29681 8394 Acked Flush 0 Done Sent Done false
29682 8395 Acked Read 10 Done Sent Done false
29683 8396 Acked Write 4 Done Sent Done false
29684 8397 Acked Read 8 Done Sent Done false
29685 8398 Acked Write 8 Done Sent Done false
29686 8399 Acked Read 7 Done Sent Done false
29687 8400 Acked Write 3 Done Sent Done false
29688 8401 Acked Write 1 Done Sent Done false
29689 8402 Acked Read 2 Done Sent Done false
29690 8403 Acked Read 10 Done Sent Done false
29691 8404 Acked Write 4 Done Sent Done false
29692 8405 Acked Read 8 Done Sent Done false
29693 8406 Acked Write 6 Done Sent Done false
29694 8407 Acked Write 5 Done Sent Done false
29695 8408 Acked Write 5 Done Sent Done false
29696 8409 Acked Write 2 Done Sent Done false
29697 8410 Acked Write 3 Done Sent Done false
29698 8411 Acked Read 9 Done Sent Done false
29699 8412 Acked Read 5 Done Sent Done false
29700 8413 Acked Write 7 Done Sent Done false
29701 8414 Acked Read 8 Done Sent Done false
29702 8415 Acked Write 6 Done Sent Done false
29703 8416 Acked Write 6 Done Sent Done false
29704 8417 Acked Write 1 Done Sent Done false
29705 8418 Acked Write 3 Done Sent Done false
29706 8419 Acked Write 10 Done Sent Done false
29707 8420 Acked Read 4 Done Sent Done false
29708 8421 Acked Flush 0 Done Sent Done false
29709 8422 Acked Read 9 Done Sent Done false
29710 8423 Acked Flush 0 Done Sent Done false
29711 8424 Acked Write 7 Done Sent Done false
29712 8425 Acked Read 10 Done Sent Done false
29713 8426 Acked Write 8 Done Sent Done false
29714 8427 Acked Read 10 Done Sent Done false
29715 8428 Acked Read 2 Done Sent Done false
29716 8429 Acked Write 1 Done Sent Done false
29717 8430 Acked Write 2 Done Sent Done false
29718 8431 Acked Write 2 Done Sent Done false
29719 8432 Acked Write 2 Done Sent Done false
29720 8433 Acked Read 10 Done Sent Done false
29721 8434 Acked Read 10 Done Sent Done false
29722 8435 Acked Write 10 Done Sent Done false
29723 8436 Acked Read 4 Done Sent Done false
29724 8437 Acked Write 2 Done Sent Done false
29725 8438 Acked Read 8 Done Sent Done false
29726 8439 Acked Read 7 Done Sent Done false
29727 8440 Acked Write 1 Done Sent Done false
29728 8441 Acked Read 4 Done Sent Done false
29729 8442 Acked Read 2 Done Sent Done false
29730 8443 Acked Read 4 Done Sent Done false
29731 8444 Acked Read 7 Done Sent Done false
29732 8445 Acked Write 5 Done Sent Done false
29733 8446 Acked Read 6 Done Sent Done false
29734 8447 Acked Read 4 Done Sent Done false
29735 8448 Acked Read 8 Done Sent Done false
29736 8449 Acked Read 3 Done Sent Done false
29737 8450 Acked Write 9 Done Sent Done false
29738 8451 Acked Read 3 Done Sent Done false
29739 8452 Acked Flush 0 Done Sent Done false
29740 8453 Acked Write 7 Done Sent Done false
29741 8454 Acked Write 2 Done Sent Done false
29742 8455 Acked Read 4 Done Sent Done false
29743 8456 Acked Write 4 Done Sent Done false
29744 8457 Acked Read 7 Done Sent Done false
29745 8458 Acked Write 1 Done Sent Done false
29746 8459 Acked Write 1 Done Sent Done false
29747 8460 Acked Read 2 Done Sent Done false
29748 8461 Acked Read 8 Done Sent Done false
29749 8462 Acked Write 2 Done Sent Done false
29750 8463 Acked Write 7 Done Sent Done false
29751 8464 Acked Read 10 Done Sent Done false
29752 8465 Acked Read 9 Done Sent Done false
29753 8466 Acked Read 4 Done Sent Done false
29754 8467 Acked Read 2 Done Sent Done false
29755 8468 Acked Read 9 Done Sent Done false
29756 8469 Acked Read 7 Done Sent Done false
29757 8470 Acked Write 5 Done Sent Done false
29758 8471 Acked Read 8 Done Sent Done false
29759 8472 Acked Write 4 Done Sent Done false
29760 8473 Acked Read 1 Done Sent Done false
29761 8474 Acked Read 8 Done Sent Done false
29762 8475 Acked Read 10 Done Sent Done false
29763 8476 Acked Write 6 Done Sent Done false
29764 8477 Acked Write 9 Done Sent Done false
29765 8478 Acked Read 10 Done Sent Done false
29766 8479 Acked Read 2 Done Sent Done false
29767 8480 Acked Read 1 Done Sent Done false
29768 8481 Acked Write 1 Done Sent Done false
29769 8482 Acked Read 2 Done Sent Done false
29770 8483 Acked Write 8 Done Sent Done false
29771 8484 Acked Write 6 Done Sent Done false
29772 8485 Acked Flush 0 Done Sent Done false
29773 8486 Acked Read 7 Done Sent Done false
29774 8487 Acked Write 9 Done Sent Done false
29775 8488 Acked Read 1 Done Sent Done false
29776 8489 Acked Write 8 Done Sent Done false
29777 8490 Acked Write 9 Done Sent Done false
29778 8491 Acked Read 10 Done Sent Done false
29779 8492 Acked Write 9 Done Sent Done false
29780 8493 Acked Write 10 Done Sent Done false
29781 8494 Acked Write 2 Done Sent Done false
29782 8495 Acked Read 2 Done Sent Done false
29783 8496 Acked Write 5 Done Sent Done false
29784 8497 Acked Read 9 Done Sent Done false
29785 8498 Acked Write 1 Done Sent Done false
29786 8499 Acked Read 3 Done Sent Done false
29787 8500 Acked Write 8 Done Sent Done false
29788 8501 Acked Read 9 Done Sent Done false
29789 8502 Acked Write 3 Done Sent Done false
29790 8503 Acked Write 2 Done Sent Done false
29791 8504 Acked Read 3 Done Sent Done false
29792 8505 Acked Read 7 Done Sent Done false
29793 8506 Acked Write 9 Done Sent Done false
29794 8507 Acked Read 3 Done Sent Done false
29795 8508 Acked Read 4 Done Sent Done false
29796 8509 Acked Flush 0 Done Sent Done false
29797 8510 Acked Write 8 Done Sent Done false
29798 8511 Acked Read 7 Done Sent Done false
29799 8512 Acked Flush 0 Done Sent Done false
29800 8513 Acked Read 8 Done Sent Done false
29801 8514 Acked Write 1 Done Sent Done false
29802 8515 Acked Write 5 Done Sent Done false
29803 8516 Acked Write 8 Done Sent Done false
29804 8517 Acked Read 5 Done Sent Done false
29805 8518 Acked Read 1 Done Sent Done false
29806 8519 Acked Read 4 Done Sent Done false
29807 8520 Acked Write 1 Done Sent Done false
29808 8521 Acked Read 9 Done Sent Done false
29809 8522 Acked Write 2 Done Sent Done false
29810 8523 Acked Read 7 Done Sent Done false
29811 8524 Acked Read 5 Done Sent Done false
29812 8525 Acked Write 9 Done Sent Done false
29813 8526 Acked Read 4 Done Sent Done false
29814 8527 Acked Read 9 Done Sent Done false
29815 8528 Acked Write 7 Done Sent Done false
29816 8529 Acked Write 8 Done Sent Done false
29817 8530 Acked Write 9 Done Sent Done false
29818 8531 Acked Write 7 Done Sent Done false
29819 8532 Acked Read 1 Done Sent Done false
29820 8533 Acked Read 6 Done Sent Done false
29821 8534 Acked Read 4 Done Sent Done false
29822 8535 Acked Read 10 Done Sent Done false
29823 8536 Acked Read 7 Done Sent Done false
29824 8537 Acked Write 9 Done Sent Done false
29825 8538 Acked Read 8 Done Sent Done false
29826 8539 Acked Read 2 Done Sent Done false
29827 8540 Acked Read 1 Done Sent Done false
29828 8541 Acked Read 3 Done Sent Done false
29829 8542 Acked Flush 0 Done Sent Done false
29830 8543 Acked Write 6 Done Sent Done false
29831 8544 Acked Read 3 Done Sent Done false
29832 8545 Acked Read 2 Done Sent Done false
29833 8546 Acked Flush 0 Done Sent Done false
29834 STATES DS:0 DS:1 DS:2 TOTAL
29835 Sent 0 301 0 301
29836 Done 301 0 301 602
29837 Skipped 0 0 0 0
29838 Error 0 0 0 0
29839 Last Flush: Some(JobId(8546)) Some(JobId(8245)) Some(JobId(8546))
29840 Downstairs last five completed: 8245 8244 8243 8242 8241
29841 Upstairs last five completed: 8546 8545 8544 8543 8542
29842 CLIENT: Up:0 ds:301 act:2
298432025-03-17T22:41:07.922ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298442025-03-17T22:41:08.923ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298452025-03-17T22:41:09.924ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298462025-03-17T22:41:10.926ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298472025-03-17T22:41:10.935ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298482025-03-17T22:41:10.935ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50907) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298492025-03-17T22:41:10.936ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298502025-03-17T22:41:10.937ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298512025-03-17T22:41:10.937ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298522025-03-17T22:41:10.937ZINFOcrucible: send last flush ID to this DS: Some(JobId(8245)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298532025-03-17T22:41:10.937ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298542025-03-17T22:41:10.937ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8245)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298552025-03-17T22:41:10.937ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8245)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
29856 ----------------------------------------------------------------
29857 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29858 Downstairs last five completed: 8546 8545 8544 8543 8542
29859 Upstairs last five completed: 8546 8545 8544 8543 8542
29860 CLIENT: Up:0 ds:0 act:3
29861 Replay: All jobs finished
29862 Main task has work to do, go find it
29863 got dsc Stop(0)
29864 stop 0
29865 Mar 17 22:41:11.184 INFO request completed, latency_us: 360, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9fb38710-55b8-4fcc-914b-bc5025cb0975, remote_addr: 127.0.0.1[8810] Got stop action so:false kr:false
29866 :35180, local_addr: 127.0.0.1:9998
29867 Mar 17 22:41:11.185 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/0, method: GET, req_id: bd76f066-2a9e-46d2-9619-cfd4c7a1aac0, remote_addr: 127.0.0.1:35180, local_addr: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:41:11.185596875Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
29868 127.0.0.1:9998
298692025-03-17T22:41:11.185ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298702025-03-17T22:41:11.185ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
298712025-03-17T22:41:11.185ZINFOcrucible: client task is exiting = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
29872 [8810] Exited with: ExitStatus(unix_wait_status(9))
29873 [8810][0] reports Exit
29874 Mar 17 22:41:15.187 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/0, method: GET, req_id: f28ac67e-4ba2-41a8-a645-5702be20ee9c, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
29875 001/300 Write block 4595 len 28672 data: 4 4 4 5 6 6 6
29876 002/300 Read block 3297 len 16384
29877 003/300 Read block 3796 len 20480
29878 004/300 Read block 2948 len 40960
29879 005/300 Write block 134 len 32768 data: 4 5 7 6 7 5 5 5
29880 006/300 Write block 1500 len 28672 data: 5 4 5 5 7 5 4
29881 007/300 Read block 2089 len 16384
29882 008/300 Write block 582 len 20480 data: 4 3 3 3 3
29883 009/300 Read block 2254 len 28672
29884 010/300 Read block 4319 len 24576
29885 011/300 Read block 807 len 4096
29886 012/300 Flush
29887 013/300 Write block 3281 len 16384 data: 6 5 5 7
29888 014/300 Read block 287 len 16384
29889 015/300 Read block 2134 len 40960
29890 016/300 Write block 2335 len 20480 data: 5 4 4 4 4
29891 017/300 Write block 1664 len 4096 data: 5
29892 018/300 Write block 988 len 28672 data: 5 5 5 6 6 6 5
29893 019/300 Read block 2331 len 40960
29894 020/300 Flush
29895 021/300 Write block 1611 len 4096 data: 5
29896 022/300 Write block 143 len 32768 data: 6 4 4 4 4 3 4 3
29897 023/300 Write block 4725 len 20480 data: 5 4 4 5 4
29898 024/300 Read block 1290 len 36864
29899 025/300 Read block 1345 len 8192
29900 026/300 Write block 2879 len 24576 data: 8 6 6 6 4 5
29901 027/300 Write block 2263 len 20480 data: 5 5 6 5 7
29902 028/300 Write block 4371 len 36864 data: 7 7 7 8 9 7 5 6 5
29903 029/300 Write block 1149 len 32768 data: 3 5 5 6 5 4 5 6
29904 030/300 Read block 739 len 24576
29905 031/300 Write block 1295 len 36864 data: 8 7 7 5 4 3 5 6 5
29906 032/300 Read block 1945 len 24576
29907 033/300 Write block 2196 len 20480 data: 6 5 5 5 4
29908 034/300 Read block 794 len 32768
29909 035/300 Write block 1248 len 20480 data: 4 4 4 5 5
29910 036/300 Read block 1607 len 40960
29911 037/300 Flush
29912 038/300 Read block 4794 len 16384
29913 039/300 Flush
29914 040/300 Flush
29915 041/300 Write block 4227 len 36864 data: 5 5 4 6 7 8 7 6 7
29916 042/300 Write block 1643 len 8192 data: 4 8
29917 043/300 Read block 1452 len 24576
29918 044/300 Write block 2849 len 32768 data: 3 3 4 4 5 4 4 4
29919 045/300 Read block 4123 len 4096
29920 046/300 Flush
29921 047/300 Write block 3676 len 36864 data: 6 5 7 8 8 9 8 6 6
29922 048/300 Read block 1179 len 12288
29923 049/300 Write block 646 len 20480 data: 6 4 4 3 3
29924 050/300 Flush
29925 051/300 Read block 2986 len 4096
29926 052/300 Read block 260 len 16384
29927 053/300 Write block 3423 len 32768 data: 8 8 8 7 7 6 7 6
29928 054/300 Read block 1297 len 4096
29929 055/300 Write block 1723 len 8192 data: 8 6
29930 056/300 Write block 4786 len 40960 data: 7 6 6 6 6 6 6 6 6 5
29931 057/300 Read block 4870 len 40960
29932 058/300 Write block 4071 len 4096 data: 6
29933 059/300 Read block 419 len 36864
29934 060/300 Read block 4949 len 36864
29935 061/300 Write block 1941 len 36864 data: 5 4 5 5 6 6 6 7 8
29936 062/300 Write block 3748 len 36864 data: 3 3 3 3 3 4 5 5 4
29937 063/300 Read block 489 len 36864
29938 064/300 Read block 2694 len 24576
29939 065/300 Write block 2232 len 20480 data: 5 6 6 6 3
29940 066/300 Flush
29941 067/300 Read block 2517 len 4096
29942 068/300 Read block 2036 len 28672
29943 069/300 Flush
29944 070/300 Read block 2857 len 32768
29945 071/300 Flush
29946 072/300 Read block 2476 len 28672
29947 073/300 Write block 2072 len 40960 data: 7 7 8 8 4 5 5 5 5 4
29948 074/300 Write block 4064 len 28672 data: 5 5 6 7 7 6 7
29949 075/300 Write block 242 len 36864 data: 3 2 3 3 4 4 3 4 6
29950 076/300 Write block 1598 len 12288 data: 11 9 8
29951 077/300 Read block 3599 len 40960
29952 078/300 Read block 1789 len 4096
29953 079/300 Read block 4298 len 12288
29954 080/300 Read block 2268 len 8192
29955 081/300 Read block 1447 len 28672
29956 082/300 Read block 1359 len 12288
29957 083/300 Read block 2457 len 12288
29958 084/300 Read block 1909 len 8192
29959 085/300 Write block 3430 len 36864 data: 7 4 5 8 9 7 7 5 4
29960 086/300 Read block 2844 len 4096
29961 087/300 Write block 3219 len 32768 data: 5 3 4 5 3 2 3 3
29962 088/300 Write block 3514 len 40960 data: 3 3 5 4 3 4 6 5 6 4
29963 089/300 Read block 1745 len 20480
29964 090/300 Read block 4291 len 36864
29965 091/300 Read block 1313 len 28672
29966 092/300 Read block 4164 len 20480
29967 093/300 Read block 1658 len 36864
29968 094/300 Write block 4617 len 4096 data: 5
29969 095/300 Read block 1643 len 36864
29970 096/300 Read block 635 len 28672
29971 097/300 Read block 3645 len 32768
29972 098/300 Write block 2977 len 24576 data: 9 12 13 12 10 8
29973 099/300 Read block 3396 len 36864
29974 100/300 Read block 3374 len 8192
29975 101/300 Write block 2124 len 28672 data: 3 4 3 4 4 5 5
29976 102/300 Read block 425 len 36864
29977 103/300 Write block 2444 len 32768 data: 6 4 3 4 5 5 4 4
29978 104/300 Write block 407 len 16384 data: 4 6 5 4
29979 105/300 Write block 4363 len 24576 data: 4 4 5 6 8 8
29980 106/300 Read block 1890 len 4096
29981 107/300 Read block 29 len 16384
29982 108/300 Read block 1690 len 12288
29983 109/300 Read block 3922 len 8192
29984 110/300 Read block 3267 len 12288
29985 111/300 Write block 3592 len 28672 data: 5 7 8 8 5 6 6
29986 112/300 Write block 4865 len 24576 data: 4 5 7 6 7 6
29987 113/300 Write block 4772 len 40960 data: 4 7 6 7 4 4 4 4 4 4
29988 114/300 Write block 643 len 24576 data: 6 6 7 7 5 5
29989 115/300 Write block 3068 len 12288 data: 6 7 7
29990 116/300 Write block 3011 len 16384 data: 4 4 4 3
29991 117/300 Read block 266 len 20480
29992 118/300 Read block 4486 len 40960
29993 119/300 Write block 155 len 24576 data: 4 4 5 5 6 5
29994 120/300 Read block 4107 len 12288
29995 121/300 Write block 79 len 40960 data: 5 6 7 7 11 9 9 6 4 3
29996 122/300 Read block 1414 len 4096
29997 123/300 Read block 1060 len 8192
29998 124/300 Write block 4042 len 12288 data: 7 7 6
29999 125/300 Write block 60 len 4096 data: 3
30000 126/300 Read block 400 len 16384
30001 127/300 Read block 4321 len 8192
30002 128/300 Read block 906 len 4096
30003 129/300 Read block 774 len 8192
30004 130/300 Write block 4251 len 28672 data: 3 4 4 5 5 6 5
30005 131/300 Read block 1235 len 36864
30006 132/300 Write block 3697 len 40960 data: 5 4 4 3 2 3 3 3 3 2
30007 133/300 Read block 91 len 4096
30008 134/300 Write block 210 len 8192 data: 4 5
30009 135/300 Write block 1860 len 36864 data: 5 6 6 6 6 6 7 5 4
30010 136/300 Read block 1187 len 20480
30011 137/300 Write block 2097 len 28672 data: 5 4 6 6 6 7 6
30012 138/300 Write block 1025 len 32768 data: 4 4 5 5 4 4 3 4
30013 139/300 Write block 3096 len 16384 data: 5 5 5 3
30014 140/300 Read block 980 len 4096
30015 141/300 Read block 2227 len 12288
30016 142/300 Read block 672 len 36864
30017 143/300 Read block 3449 len 8192
30018 144/300 Write block 4871 len 8192 data: 7 7
30019 145/300 Write block 858 len 24576 data: 6 4 4 4 4 4
30020 146/300 Read block 2440 len 32768
30021 147/300 Read block 2193 len 12288
30022 148/300 Write block 84 len 36864 data: 10 10 7 5 4 5 5 4 4
30023 149/300 Write block 2772 len 16384 data: 6 5 5 4
30024 150/300 Flush
30025 151/300 Write block 3894 len 32768 data: 6 6 8 5 4 4 7 8
30026 152/300 Read block 3712 len 8192
30027 153/300 Flush
30028 154/300 Read block 4146 len 28672
30029 155/300 Flush
30030 156/300 Read block 3086 len 32768
30031 157/300 Write block 2810 len 28672 data: 3 4 3 2 3 4 4
30032 158/300 Write block 2197 len 12288 data: 6 6 6
30033 159/300 Write block 4235 len 28672 data: 8 6 6 4 4 4 4
30034 160/300 Read block 2282 len 28672
30035 161/300 Write block 3534 len 4096 data: 4
30036 162/300 Flush
30037 163/300 Read block 4053 len 32768
30038 164/300 Read block 1974 len 28672
30039 165/300 Read block 336 len 40960
30040 166/300 Write block 3767 len 16384 data: 2 3 2 4
30041 167/300 Read block 1065 len 16384
30042 168/300 Write block 785 len 16384 data: 7 5 3 5
30043 169/300 Read block 2092 len 24576
30044 170/300 Read block 4835 len 20480
30045 171/300 Read block 4605 len 28672
30046 172/300 Write block 887 len 12288 data: 2 2 3
30047 173/300 Flush
30048 174/300 Write block 794 len 16384 data: 5 4 4 4
30049 175/300 Read block 2288 len 4096
30050 176/300 Write block 2153 len 24576 data: 4 4 4 5 6 7
30051 177/300 Write block 4246 len 32768 data: 4 5 4 4 4 4 5 5
30052 178/300 Read block 157 len 20480
30053 179/300 Write block 2079 len 28672 data: 6 6 5 4 5 5 5
30054 180/300 Write block 1433 len 32768 data: 6 6 7 9 7 5 6 5
30055 181/300 Read block 2153 len 40960
30056 182/300 Read block 813 len 32768
30057 183/300 Read block 1232 len 24576
30058 184/300 Read block 766 len 16384
30059 185/300 Write block 3446 len 24576 data: 4 3 3 5 7 7
30060 186/300 Write block 1734 len 16384 data: 7 6 7 5
30061 187/300 Read block 1299 len 4096
30062 188/300 Write block 3181 len 28672 data: 8 7 6 6 7 8 7
30063 189/300 Read block 4756 len 40960
30064 190/300 Write block 3505 len 24576 data: 4 5 5 6 6 4
30065 191/300 Write block 1492 len 40960 data: 3 3 3 4 5 6 7 7 6 5
30066 192/300 Read block 3637 len 20480
30067 193/300 Read block 2700 len 28672
30068 194/300 Read block 1723 len 16384
30069 195/300 Write block 1547 len 24576 data: 5 6 5 4 5 6
30070 196/300 Read block 2816 len 4096
30071 197/300 Write block 2894 len 12288 data: 3 3 3
30072 198/300 Flush
30073 199/300 Read block 2130 len 36864
30074 200/300 Read block 778 len 32768
30075 201/300 Read block 1547 len 36864
30076 202/300 Write block 3954 len 40960 data: 3 2 2 2 3 3 4 5 6 6
30077 203/300 Write block 2501 len 36864 data: 3 3 3 3 3 3 4 4 4
30078 204/300 Read block 3779 len 16384
30079 205/300 Write block 3406 len 40960 data: 6 6 6 7 7 6 8 8 7 7
30080 206/300 Write block 4839 len 32768 data: 4 6 6 6 8 8 7 7
30081 207/300 Write block 1325 len 12288 data: 7 8 8
30082 208/300 Read block 3331 len 20480
30083 209/300 Flush
30084 210/300 Read block 4770 len 20480
30085 211/300 Read block 3466 len 12288
30086 212/300 Write block 987 len 28672 data: 4 6 6 6 7 7 7
30087 213/300 Read block 1263 len 20480
30088 214/300 Read block 4265 len 20480
30089 215/300 Read block 3473 len 4096
30090 216/300 Read block 3474 len 12288
30091 217/300 Write block 1181 len 32768 data: 4 3 3 2 4 4 5 5
30092 218/300 Read block 3678 len 12288
30093 219/300 Write block 426 len 28672 data: 4 3 5 5 4 5 5
30094 220/300 Read block 4718 len 4096
30095 221/300 Read block 4716 len 12288
30096 222/300 Read block 1402 len 8192
30097 223/300 Read block 2731 len 20480
30098 224/300 Read block 1983 len 12288
30099 225/300 Write block 4940 len 36864 data: 5 4 4 4 4 4 4 4 5
30100 226/300 Write block 2568 len 16384 data: 5 5 6 7
30101 227/300 Read block 2634 len 28672
30102 228/300 Flush
30103 229/300 Write block 39 len 24576 data: 7 5 5 7 8 8
30104 230/300 Write block 4677 len 20480 data: 5 4 6 7 6
30105 231/300 Read block 1143 len 24576
30106 232/300 Flush
30107 233/300 Write block 3701 len 16384 data: 3 4 4 4
30108 234/300 Write block 2238 len 12288 data: 3 3 3
30109 235/300 Write block 237 len 32768 data: 7 7 5 4 3 4 3 4
30110 236/300 Read block 3876 len 40960
30111 237/300 Read block 4041 len 16384
30112 238/300 Flush
30113 239/300 Write block 4298 len 8192 data: 7 6
30114 240/300 Read block 3272 len 24576
30115 241/300 Read block 2812 len 40960
30116 242/300 Write block 4085 len 20480 data: 3 5 5 7 6
30117 243/300 Read block 2390 len 24576
30118 244/300 Write block 2497 len 24576 data: 5 4 3 2 4 4
30119 245/300 Read block 892 len 32768
30120 246/300 Flush
30121 247/300 Read block 1845 len 24576
30122 248/300 Read block 2246 len 4096
30123 249/300 Read block 3105 len 36864
30124 250/300 Read block 2221 len 16384
30125 251/300 Read block 2918 len 16384
30126 252/300 Read block 2258 len 16384
30127 253/300 Write block 4980 len 8192 data: 4 5
30128 254/300 Write block 105 len 32768 data: 7 7 5 6 7 6 7 6
30129 255/300 Read block 4374 len 36864
30130 256/300 Flush
30131 257/300 Read block 3033 len 8192
30132 258/300 Read block 1444 len 40960
30133 259/300 Read block 2290 len 28672
30134 260/300 Read block 641 len 24576
30135 261/300 Read block 3825 len 28672
30136 262/300 Write block 2260 len 8192 data: 7 5
30137 263/300 Write block 3319 len 8192 data: 8 10
30138 264/300 Write block 1638 len 28672 data: 8 8 8 6 5 5 9
30139 265/300 Read block 239 len 20480
30140 266/300 Read block 4034 len 20480
30141 267/300 Read block 2655 len 12288
30142 268/300 Read block 1919 len 16384
30143 269/300 Write block 3985 len 40960 data: 5 6 9 9 9 8 8 8 7 7
30144 270/300 Write block 1571 len 24576 data: 4 4 4 4 5 3
30145 271/300 Read block 3493 len 12288
30146 272/300 Flush
30147 273/300 Read block 4524 len 32768
30148 274/300 Read block 1845 len 36864
30149 275/300 Read block 946 len 4096
30150 276/300 Write block 1068 len 24576 data: 3 3 5 4 4 4
30151 277/300 Read block 2879 len 28672
30152 278/300 Read block 3284 len 20480
30153 279/300 Write block 3704 len 8192 data: 5 4
30154 280/300 Write block 4767 len 36864 data: 5 5 4 4 4 5 8 7 8
30155 281/300 Write block 4931 len 40960 data: 6 5 7 9 8 7 7 4 4 6
30156 282/300 Write block 352 len 12288 data: 3 4 3
30157 283/300 Write block 1048 len 8192 data: 4 3
30158 284/300 Read block 3528 len 28672
30159 285/300 Write block 2188 len 12288 data: 3 3 3
30160 286/300 Write block 573 len 24576 data: 4 4 4 4 4 5
30161 287/300 Read block 1896 len 20480
30162 288/300 Write block 2927 len 32768 data: 3 3 4 5 3 3 3 3
30163 289/300 Read block 3306 len 20480
30164 290/300 Read block 3802 len 4096
30165 291/300 Flush
30166 292/300 Write block 3856 len 20480 data: 7 5 4 3 4
30167 293/300 Write block 4984 len 20480 data: 7 6 6 5 4
30168 294/300 Write block 875 len 12288 data: 5 6 6
30169 295/300 Read block 1449 len 28672
30170 296/300 Read block 4083 len 20480
30171 297/300 Read block 1732 len 16384
30172 298/300 Write block 3736 len 36864 data: 6 6 5 5 3 2 3 4 2
30173 299/300 Write block 2213 len 40960 data: 4 5 5 5 4 3 3 3 3 3
30174 300/300 Read block 1856 len 32768
30175 301/300 Write block 4361 len 8192 data: 3 3
30176 Main task has work to do, go find it
30177 got dsc Start(0)
30178 start 0
30179 Mar 17 22:41:15.434 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8cff6303-0a4a-40d6-ac31-d09fced831a7, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
30180 Replay: started 0, returned:Ok(())
30181 ----------------------------------------------------------------
30182 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30183 States: Offline Active Active
30184 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30185 8547 Acked Write 7 Sent Done Done false
30186 8548 Acked Read 4 Sent Done Done false
30187 8549 Acked Read 5 Sent Done Done false
30188 8550 Acked Read 10 Sent Done Done false
30189 8551 Acked Write 8 Sent Done Done false
30190 8552 Acked Write 7 Sent Done Done false
30191 8553 Acked Read 4 Sent Done Done false
30192 8554 Acked Write 5 Sent Done Done false
30193 8555 Acked Read 7 Sent Done Done false
30194 8556 Acked Read 6 Sent Done Done false
30195 8557 Acked Read 1 Sent Done Done false
30196 8558 Acked Flush 0 Sent Done Done false
30197 8559 Acked Write 4 Sent Done Done false
30198 8560 Acked Read 4 Sent Done Done false
30199 8561 Acked Read 10 Sent Done Done false
30200 8562 Acked Write 5 Sent Done Done false
30201 8563 Acked Write 1 Sent Done Done false
30202 8564 Acked Write 7 Sent Done Done false
30203 8565 Acked Read 10 Sent Done Done false
30204 8566 Acked Flush 0 Sent Done Done false
30205 8567 Acked Write 1 Sent Done Done false
30206 8568 Acked Write 8 Sent Done Done false
30207 8569 Acked Write 5 Sent Done Done false
30208 8570 Acked Read 9 Sent Done Done false
30209 8571 Acked Read 2 Sent Done Done false
30210 8572 Acked Write 6 Sent Done Done false
30211 8573 Acked Write 5 Sent Done Done false
30212 8574 Acked Write 9 Sent Done Done false
30213 8575 Acked Write 8 Sent Done Done false
30214 8576 Acked Read 6 Sent Done Done false
30215 8577 Acked Write 9 Sent Done Done false
30216 8578 Acked Read 6 Sent Done Done false
30217 8579 Acked Write 5 Sent Done Done false
30218 8580 Acked Read 8 Sent Done Done false
30219 8581 Acked Write 5 Sent Done Done false
30220 8582 Acked Read 10 Sent Done Done false
30221 8583 Acked Flush 0 Sent Done Done false
30222 8584 Acked Read 4 Sent Done Done false
30223 8585 Acked Flush 0 Sent Done Done false
30224 8586 Acked Flush 0 Sent Done Done false
30225 8587 Acked Write 9 Sent Done Done false
30226 8588 Acked Write 2 Sent Done Done false
30227 8589 Acked Read 6 Sent Done Done false
30228 8590 Acked Write 8 Sent Done Done false
30229 8591 Acked Read 1 Sent Done Done false
30230 8592 Acked Flush 0 Sent Done Done false
30231 8593 Acked Write 9 Sent Done Done false
30232 8594 Acked Read 3 Sent Done Done false
30233 8595 Acked Write 5 Sent Done Done false
30234 8596 Acked Flush 0 Sent Done Done false
30235 8597 Acked Read 1 Sent Done Done false
30236 8598 Acked Read 4 Sent Done Done false
30237 8599 Acked Write 8 Sent Done Done false
30238 8600 Acked Read 1 Sent Done Done false
30239 8601 Acked Write 2 Sent Done Done false
30240 8602 Acked Write 10 Sent Done Done false
30241 8603 Acked Read 10 Sent Done Done false
30242 8604 Acked Write 1 Sent Done Done false
30243 8605 Acked Read 9 Sent Done Done false
30244 8606 Acked Read 9 Sent Done Done false
30245 8607 Acked Write 9 Sent Done Done false
30246 8608 Acked Write 9 Sent Done Done false
30247 8609 Acked Read 9 Sent Done Done false
30248 8610 Acked Read 6 Sent Done Done false
30249 8611 Acked Write 5 Sent Done Done false
30250 8612 Acked Flush 0 Sent Done Done false
30251 8613 Acked Read 1 Sent Done Done false
30252 8614 Acked Read 7 Sent Done Done false
30253 8615 Acked Flush 0 Sent Done Done false
30254 8616 Acked Read 8 Sent Done Done false
30255 8617 Acked Flush 0 Sent Done Done false
30256 8618 Acked Read 7 Sent Done Done false
30257 8619 Acked Write 10 Sent Done Done false
30258 8620 Acked Write 7 Sent Done Done false
30259 8621 Acked Write 9 Sent Done Done false
30260 8622 Acked Write 3 Sent Done Done false
30261 8623 Acked Read 10 Sent Done Done false
30262 8624 Acked Read 1 Sent Done Done false
30263 8625 Acked Read 3 Sent Done Done false
30264 8626 Acked Read 2 Sent Done Done false
30265 8627 Acked Read 7 Sent Done Done false
30266 8628 Acked Read 3 Sent Done Done false
30267 8629 Acked Read 3 Sent Done Done false
30268 8630 Acked Read 2 Sent Done Done false
30269 8631 Acked Write 9 Sent Done Done false
30270 8632 Acked Read 1 Sent Done Done false
30271 8633 Acked Write 8 Sent Done Done false
30272 8634 Acked Write 10 Sent Done Done false
30273 8635 Acked Read 5 Sent Done Done false
30274 8636 Acked Read 9 Sent Done Done false
30275 8637 Acked Read 7 Sent Done Done false
30276 8638 Acked Read 5 Sent Done Done false
30277 8639 Acked Read 9 Sent Done Done false
30278 8640 Acked Write 1 Sent Done Done false
30279 8641 Acked Read 9 Sent Done Done false
30280 8642 Acked Read 7 Sent Done Done false
30281 8643 Acked Read 8 Sent Done Done false
30282 8644 Acked Write 6 Sent Done Done false
30283 8645 Acked Read 9 Sent Done Done false
30284 8646 Acked Read 2 Sent Done Done false
30285 8647 Acked Write 7 Sent Done Done false
30286 8648 Acked Read 9 Sent Done Done false
30287 8649 Acked Write 8 Sent Done Done false
30288 8650 Acked Write 4 Sent Done Done false
30289 8651 Acked Write 6 Sent Done Done false
30290 8652 Acked Read 1 Sent Done Done false
30291 8653 Acked Read 4 Sent Done Done false
30292 8654 Acked Read 3 Sent Done Done false
30293 8655 Acked Read 2 Sent Done Done false
30294 8656 Acked Read 3 Sent Done Done false
30295 8657 Acked Write 7 Sent Done Done false
30296 8658 Acked Write 6 Sent Done Done false
30297 8659 Acked Write 10 Sent Done Done false
30298 8660 Acked Write 6 Sent Done Done false
30299 8661 Acked Write 3 Sent Done Done false
30300 8662 Acked Write 4 Sent Done Done false
30301 8663 Acked Read 5 Sent Done Done false
30302 8664 Acked Read 10 Sent Done Done false
30303 8665 Acked Write 6 Sent Done Done false
30304 8666 Acked Read 3 Sent Done Done false
30305 8667 Acked Write 10 Sent Done Done false
30306 8668 Acked Read 1 Sent Done Done false
30307 8669 Acked Read 2 Sent Done Done false
30308 8670 Acked Write 3 Sent Done Done false
30309 8671 Acked Write 1 Sent Done Done false
30310 8672 Acked Read 4 Sent Done Done false
30311 8673 Acked Read 2 Sent Done Done false
30312 8674 Acked Read 1 Sent Done Done false
30313 8675 Acked Read 2 Sent Done Done false
30314 8676 Acked Write 7 Sent Done Done false
30315 8677 Acked Read 9 Sent Done Done false
30316 8678 Acked Write 10 Sent Done Done false
30317 8679 Acked Read 1 Sent Done Done false
30318 8680 Acked Write 2 Sent Done Done false
30319 8681 Acked Write 9 Sent Done Done false
30320 8682 Acked Read 5 Sent Done Done false
30321 8683 Acked Write 7 Sent Done Done false
30322 8684 Acked Write 8 Sent Done Done false
30323 8685 Acked Write 4 Sent Done Done false
30324 8686 Acked Read 1 Sent Done Done false
30325 8687 Acked Read 3 Sent Done Done false
30326 8688 Acked Read 9 Sent Done Done false
30327 8689 Acked Read 2 Sent Done Done false
30328 8690 Acked Write 2 Sent Done Done false
30329 8691 Acked Write 6 Sent Done Done false
30330 8692 Acked Read 8 Sent Done Done false
30331 8693 Acked Read 3 Sent Done Done false
30332 8694 Acked Write 9 Sent Done Done false
30333 8695 Acked Write 4 Sent Done Done false
30334 8696 Acked Flush 0 Sent Done Done false
30335 8697 Acked Write 8 Sent Done Done false
30336 8698 Acked Read 2 Sent Done Done false
30337 8699 Acked Flush 0 Sent Done Done false
30338 8700 Acked Read 7 Sent Done Done false
30339 8701 Acked Flush 0 Sent Done Done false
30340 8702 Acked Read 8 Sent Done Done false
30341 8703 Acked Write 7 Sent Done Done false
30342 8704 Acked Write 3 Sent Done Done false
30343 8705 Acked Write 7 Sent Done Done false
30344 8706 Acked Read 7 Sent Done Done false
30345 8707 Acked Write 1 Sent Done Done false
30346 8708 Acked Flush 0 Sent Done Done false
30347 8709 Acked Read 8 Sent Done Done false
30348 8710 Acked Read 7 Sent Done Done false
30349 8711 Acked Read 10 Sent Done Done false
30350 8712 Acked Write 4 Sent Done Done false
30351 8713 Acked Read 4 Sent Done Done false
30352 8714 Acked Write 4 Sent Done Done false
30353 8715 Acked Read 6 Sent Done Done false
30354 8716 Acked Read 5 Sent Done Done false
30355 8717 Acked Read 7 Sent Done Done false
30356 8718 Acked Write 3 Sent Done Done false
30357 8719 Acked Flush 0 Sent Done Done false
30358 8720 Acked Write 4 Sent Done Done false
30359 8721 Acked Read 1 Sent Done Done false
30360 8722 Acked Write 6 Sent Done Done false
30361 8723 Acked Write 8 Sent Done Done false
30362 8724 Acked Read 5 Sent Done Done false
30363 8725 Acked Write 7 Sent Done Done false
30364 8726 Acked Write 8 Sent Done Done false
30365 8727 Acked Read 10 Sent Done Done false
30366 8728 Acked Read 8 Sent Done Done false
30367 8729 Acked Read 6 Sent Done Done false
30368 8730 Acked Read 4 Sent Done Done false
30369 8731 Acked Write 6 Sent Done Done false
30370 8732 Acked Write 4 Sent Done Done false
30371 8733 Acked Read 1 Sent Done Done false
30372 8734 Acked Write 7 Sent Done Done false
30373 8735 Acked Read 10 Sent Done Done false
30374 8736 Acked Write 6 Sent Done Done false
30375 8737 Acked Write 10 Sent Done Done false
30376 8738 Acked Read 5 Sent Done Done false
30377 8739 Acked Read 7 Sent Done Done false
30378 8740 Acked Read 4 Sent Done Done false
30379 8741 Acked Write 6 Sent Done Done false
30380 8742 Acked Read 1 Sent Done Done false
30381 8743 Acked Write 3 Sent Done Done false
30382 8744 Acked Flush 0 Sent Done Done false
30383 8745 Acked Read 9 Sent Done Done false
30384 8746 Acked Read 8 Sent Done Done false
30385 8747 Acked Read 9 Sent Done Done false
30386 8748 Acked Write 10 Sent Done Done false
30387 8749 Acked Write 9 Sent Done Done false
30388 8750 Acked Read 4 Sent Done Done false
30389 8751 Acked Write 10 Sent Done Done false
30390 8752 Acked Write 8 Sent Done Done false
30391 8753 Acked Write 3 Sent Done Done false
30392 8754 Acked Read 5 Sent Done Done false
30393 8755 Acked Flush 0 Sent Done Done false
30394 8756 Acked Read 5 Sent Done Done false
30395 8757 Acked Read 3 Sent Done Done false
30396 8758 Acked Write 7 Sent Done Done false
30397 8759 Acked Read 5 Sent Done Done false
30398 8760 Acked Read 5 Sent Done Done false
30399 8761 Acked Read 1 Sent Done Done false
30400 8762 Acked Read 3 Sent Done Done false
30401 8763 Acked Write 8 Sent Done Done false
30402 8764 Acked Read 3 Sent Done Done false
30403 8765 Acked Write 7 Sent Done Done false
30404 8766 Acked Read 1 Sent Done Done false
30405 8767 Acked Read 3 Sent Done Done false
30406 8768 Acked Read 2 Sent Done Done false
30407 8769 Acked Read 5 Sent Done Done false
30408 8770 Acked Read 3 Sent Done Done false
30409 8771 Acked Write 9 Sent Done Done false
30410 8772 Acked Write 4 Sent Done Done false
30411 8773 Acked Read 7 Sent Done Done false
30412 8774 Acked Flush 0 Sent Done Done false
30413 8775 Acked Write 6 Sent Done Done false
30414 8776 Acked Write 5 Sent Done Done false
30415 8777 Acked Read 6 Sent Done Done false
30416 8778 Acked Flush 0 Sent Done Done false
30417 8779 Acked Write 4 Sent Done Done false
30418 8780 Acked Write 3 Sent Done Done false
30419 8781 Acked Write 8 Sent Done Done false
30420 8782 Acked Read 10 Sent Done Done false
30421 8783 Acked Read 4 Sent Done Done false
30422 8784 Acked Flush 0 Sent Done Done false
30423 8785 Acked Write 2 Sent Done Done false
30424 8786 Acked Read 6 Sent Done Done false
30425 8787 Acked Read 10 Sent Done Done false
30426 8788 Acked Write 5 Sent Done Done false
30427 8789 Acked Read 6 Sent Done Done false
30428 8790 Acked Write 6 Sent Done Done false
30429 8791 Acked Read 8 Sent Done Done false
30430 8792 Acked Flush 0 Sent Done Done false
30431 8793 Acked Read 6 Sent Done Done false
30432 8794 Acked Read 1 Sent Done Done false
30433 8795 Acked Read 9 Sent Done Done false
30434 8796 Acked Read 4 Sent Done Done false
30435 8797 Acked Read 4 Sent Done Done false
30436 8798 Acked Read 4 Sent Done Done false
30437 8799 Acked Write 2 Sent Done Done false
30438 8800 Acked Write 8 Sent Done Done false
30439 8801 Acked Read 9 Sent Done Done false
30440 8802 Acked Flush 0 Sent Done Done false
30441 8803 Acked Read 2 Sent Done Done false
30442 8804 Acked Read 10 Sent Done Done false
30443 8805 Acked Read 7 Sent Done Done false
30444 8806 Acked Read 6 Sent Done Done false
30445 8807 Acked Read 7 Sent Done Done false
30446 8808 Acked Write 2 Sent Done Done false
30447 8809 Acked Write 2 Sent Done Done false
30448 8810 Acked Write 7 Sent Done Done false
30449 8811 Acked Read 5 Sent Done Done false
30450 8812 Acked Read 5 Sent Done Done false
30451 8813 Acked Read 3 Sent Done Done false
30452 8814 Acked Read 4 Sent Done Done false
30453 8815 Acked Write 10 Sent Done Done false
30454 8816 Acked Write 6 Sent Done Done false
30455 8817 Acked Read 3 Sent Done Done false
30456 8818 Acked Flush 0 Sent Done Done false
30457 8819 Acked Read 8 Sent Done Done false
30458 8820 Acked Read 9 Sent Done Done false
30459 8821 Acked Read 1 Sent Done Done false
30460 8822 Acked Write 6 Sent Done Done false
30461 8823 Acked Read 7 Sent Done Done false
30462 8824 Acked Read 5 Sent Done Done false
30463 8825 Acked Write 2 Sent Done Done false
30464 8826 Acked Write 9 Sent Done Done false
30465 8827 Acked Write 10 Sent Done Done false
30466 8828 Acked Write 3 Sent Done Done false
30467 8829 Acked Write 2 Sent Done Done false
30468 8830 Acked Read 7 Sent Done Done false
30469 8831 Acked Write 3 Sent Done Done false
30470 8832 Acked Write 6 Sent Done Done false
30471 8833 Acked Read 5 Sent Done Done false
30472 8834 Acked Write 8 Sent Done Done false
30473 8835 Acked Read 5 Sent Done Done false
30474 8836 Acked Read 1 Sent Done Done false
30475 8837 Acked Flush 0 Sent Done Done false
30476 8838 Acked Write 5 Sent Done Done false
30477 8839 Acked Write 5 Sent Done Done false
30478 8840 Acked Write 3 Sent Done Done false
30479 8841 Acked Read 7 Sent Done Done false
30480 8842 Acked Read 5 Sent Done Done false
30481 8843 Acked Read 4 Sent Done Done false
30482 8844 Acked Write 9 Sent Done Done false
30483 8845 Acked Write 10 Sent Done Done false
30484 8846 Acked Read 8 Sent Done Done false
30485 8847 Acked Write 2 Sent Done Done false
30486 STATES DS:0 DS:1 DS:2 TOTAL
30487 Sent 301 0 0 301
30488 Done 0 301 301 602
30489 Skipped 0 0 0 0
30490 Error 0 0 0 0
30491 Last Flush: Some(JobId(8546)) Some(JobId(8837)) Some(JobId(8837))
30492 Downstairs last five completed: 8546 8545 8544 8543 8542
30493 Upstairs last five completed: 8847 8846 8845 8844 8843
30494 CLIENT: Up:0 ds:301 act:2
304952025-03-17T22:41:15.936ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
304962025-03-17T22:41:16.935ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
304972025-03-17T22:41:17.937ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
304982025-03-17T22:41:18.939ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
30499 [8810] got start action
30500 [8810] I am going to restart
30501 Starting downstairs at port 8810
30502 Make output file at "/tmp/dsc/downstairs-8810.txt"
30503 [8810][0] reports Starting
30504 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
30505 [8810][0] reports Running
30506 ----------------------------------------------------------------
30507 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30508 States: Offline Active Active
30509 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30510 8547 Acked Write 7 Sent Done Done false
30511 8548 Acked Read 4 Sent Done Done false
30512 8549 Acked Read 5 Sent Done Done false
30513 8550 Acked Read 10 Sent Done Done false
30514 8551 Acked Write 8 Sent Done Done false
30515 8552 Acked Write 7 Sent Done Done false
30516 8553 Acked Read 4 Sent Done Done false
30517 8554 Acked Write 5 Sent Done Done false
30518 8555 Acked Read 7 Sent Done Done false
30519 8556 Acked Read 6 Sent Done Done false
30520 8557 Acked Read 1 Sent Done Done false
30521 8558 Acked Flush 0 Sent Done Done false
30522 8559 Acked Write 4 Sent Done Done false
30523 8560 Acked Read 4 Sent Done Done false
30524 8561 Acked Read 10 Sent Done Done false
30525 8562 Acked Write 5 Sent Done Done false
30526 8563 Acked Write 1 Sent Done Done false
30527 8564 Acked Write 7 Sent Done Done false
30528 8565 Acked Read 10 Sent Done Done false
30529 8566 Acked Flush 0 Sent Done Done false
30530 8567 Acked Write 1 Sent Done Done false
30531 8568 Acked Write 8 Sent Done Done false
30532 8569 Acked Write 5 Sent Done Done false
30533 8570 Acked Read 9 Sent Done Done false
30534 8571 Acked Read 2 Sent Done Done false
30535 8572 Acked Write 6 Sent Done Done false
30536 8573 Acked Write 5 Sent Done Done false
30537 8574 Acked Write 9 Sent Done Done false
30538 8575 Acked Write 8 Sent Done Done false
30539 8576 Acked Read 6 Sent Done Done false
30540 8577 Acked Write 9 Sent Done Done false
30541 8578 Acked Read 6 Sent Done Done false
30542 8579 Acked Write 5 Sent Done Done false
30543 8580 Acked Read 8 Sent Done Done false
30544 8581 Acked Write 5 Sent Done Done false
30545 8582 Acked Read 10 Sent Done Done false
30546 8583 Acked Flush 0 Sent Done Done false
30547 8584 Acked Read 4 Sent Done Done false
30548 8585 Acked Flush 0 Sent Done Done false
30549 8586 Acked Flush 0 Sent Done Done false
30550 8587 Acked Write 9 Sent Done Done false
30551 8588 Acked Write 2 Sent Done Done false
30552 8589 Acked Read 6 Sent Done Done false
30553 8590 Acked Write 8 Sent Done Done false
30554 8591 Acked Read 1 Sent Done Done false
30555 8592 Acked Flush 0 Sent Done Done false
30556 8593 Acked Write 9 Sent Done Done false
30557 8594 Acked Read 3 Sent Done Done false
30558 8595 Acked Write 5 Sent Done Done false
30559 8596 Acked Flush 0 Sent Done Done false
30560 8597 Acked Read 1 Sent Done Done false
30561 8598 Acked Read 4 Sent Done Done false
30562 8599 Acked Write 8 Sent Done Done false
30563 8600 Acked Read 1 Sent Done Done false
30564 8601 Acked Write 2 Sent Done Done false
30565 8602 Acked Write 10 Sent Done Done false
30566 8603 Acked Read 10 Sent Done Done false
30567 8604 Acked Write 1 Sent Done Done false
30568 8605 Acked Read 9 Sent Done Done false
30569 8606 Acked Read 9 Sent Done Done false
30570 8607 Acked Write 9 Sent Done Done false
30571 8608 Acked Write 9 Sent Done Done false
30572 8609 Acked Read 9 Sent Done Done false
30573 8610 Acked Read 6 Sent Done Done false
30574 8611 Acked Write 5 Sent Done Done false
30575 8612 Acked Flush 0 Sent Done Done false
30576 8613 Acked Read 1 Sent Done Done false
30577 8614 Acked Read 7 Sent Done Done false
30578 8615 Acked Flush 0 Sent Done Done false
30579 8616 Acked Read 8 Sent Done Done false
30580 8617 Acked Flush 0 Sent Done Done false
30581 8618 Acked Read 7 Sent Done Done false
30582 8619 Acked Write 10 Sent Done Done false
30583 8620 Acked Write 7 Sent Done Done false
30584 8621 Acked Write 9 Sent Done Done false
30585 8622 Acked Write 3 Sent Done Done false
30586 8623 Acked Read 10 Sent Done Done false
30587 8624 Acked Read 1 Sent Done Done false
30588 8625 Acked Read 3 Sent Done Done false
30589 8626 Acked Read 2 Sent Done Done false
30590 8627 Acked Read 7 Sent Done Done false
30591 8628 Acked Read 3 Sent Done Done false
30592 8629 Acked Read 3 Sent Done Done false
30593 8630 Acked Read 2 Sent Done Done false
30594 8631 Acked Write 9 Sent Done Done false
30595 8632 Acked Read 1 Sent Done Done false
30596 8633 Acked Write 8 Sent Done Done false
30597 8634 Acked Write 10 Sent Done Done false
30598 8635 Acked Read 5 Sent Done Done false
30599 8636 Acked Read 9 Sent Done Done false
30600 8637 Acked Read 7 Sent Done Done false
30601 8638 Acked Read 5 Sent Done Done false
30602 8639 Acked Read 9 Sent Done Done false
30603 8640 Acked Write 1 Sent Done Done false
30604 8641 Acked Read 9 Sent Done Done false
30605 8642 Acked Read 7 Sent Done Done false
30606 8643 Acked Read 8 Sent Done Done false
30607 8644 Acked Write 6 Sent Done Done false
30608 8645 Acked Read 9 Sent Done Done false
30609 8646 Acked Read 2 Sent Done Done false
30610 8647 Acked Write 7 Sent Done Done false
30611 8648 Acked Read 9 Sent Done Done false
30612 8649 Acked Write 8 Sent Done Done false
30613 8650 Acked Write 4 Sent Done Done false
30614 8651 Acked Write 6 Sent Done Done false
30615 8652 Acked Read 1 Sent Done Done false
30616 8653 Acked Read 4 Sent Done Done false
30617 8654 Acked Read 3 Sent Done Done false
30618 8655 Acked Read 2 Sent Done Done false
30619 8656 Acked Read 3 Sent Done Done false
30620 8657 Acked Write 7 Sent Done Done false
30621 8658 Acked Write 6 Sent Done Done false
30622 8659 Acked Write 10 Sent Done Done false
30623 8660 Acked Write 6 Sent Done Done false
30624 8661 Acked Write 3 Sent Done Done false
30625 8662 Acked Write 4 Sent Done Done false
30626 8663 Acked Read 5 Sent Done Done false
30627 8664 Acked Read 10 Sent Done Done false
30628 8665 Acked Write 6 Sent Done Done false
30629 8666 Acked Read 3 Sent Done Done false
30630 8667 Acked Write 10 Sent Done Done false
30631 8668 Acked Read 1 Sent Done Done false
30632 8669 Acked Read 2 Sent Done Done false
30633 8670 Acked Write 3 Sent Done Done false
30634 8671 Acked Write 1 Sent Done Done false
30635 8672 Acked Read 4 Sent Done Done false
30636 8673 Acked Read 2 Sent Done Done false
30637 8674 Acked Read 1 Sent Done Done false
30638 8675 Acked Read 2 Sent Done Done false
30639 8676 Acked Write 7 Sent Done Done false
30640 8677 Acked Read 9 Sent Done Done false
30641 8678 Acked Write 10 Sent Done Done false
30642 8679 Acked Read 1 Sent Done Done false
30643 8680 Acked Write 2 Sent Done Done false
30644 8681 Acked Write 9 Sent Done Done false
30645 8682 Acked Read 5 Sent Done Done false
30646 8683 Acked Write 7 Sent Done Done false
30647 8684 Acked Write 8 Sent Done Done false
30648 8685 Acked Write 4 Sent Done Done false
30649 8686 Acked Read 1 Sent Done Done false
30650 8687 Acked Read 3 Sent Done Done false
30651 8688 Acked Read 9 Sent Done Done false
30652 8689 Acked Read 2 Sent Done Done false
30653 8690 Acked Write 2 Sent Done Done false
30654 8691 Acked Write 6 Sent Done Done false
30655 8692 Acked Read 8 Sent Done Done false
30656 8693 Acked Read 3 Sent Done Done false
30657 8694 Acked Write 9 Sent Done Done false
30658 8695 Acked Write 4 Sent Done Done false
30659 8696 Acked Flush 0 Sent Done Done false
30660 8697 Acked Write 8 Sent Done Done false
30661 8698 Acked Read 2 Sent Done Done false
30662 8699 Acked Flush 0 Sent Done Done false
30663 8700 Acked Read 7 Sent Done Done false
30664 8701 Acked Flush 0 Sent Done Done false
30665 8702 Acked Read 8 Sent Done Done false
30666 8703 Acked Write 7 Sent Done Done false
30667 8704 Acked Write 3 Sent Done Done false
30668 8705 Acked Write 7 Sent Done Done false
30669 8706 Acked Read 7 Sent Done Done false
30670 8707 Acked Write 1 Sent Done Done false
30671 8708 Acked Flush 0 Sent Done Done false
30672 8709 Acked Read 8 Sent Done Done false
30673 8710 Acked Read 7 Sent Done Done false
30674 8711 Acked Read 10 Sent Done Done false
30675 8712 Acked Write 4 Sent Done Done false
30676 8713 Acked Read 4 Sent Done Done false
30677 8714 Acked Write 4 Sent Done Done false
30678 8715 Acked Read 6 Sent Done Done false
30679 8716 Acked Read 5 Sent Done Done false
30680 8717 Acked Read 7 Sent Done Done false
30681 8718 Acked Write 3 Sent Done Done false
30682 8719 Acked Flush 0 Sent Done Done false
30683 8720 Acked Write 4 Sent Done Done false
30684 8721 Acked Read 1 Sent Done Done false
30685 8722 Acked Write 6 Sent Done Done false
30686 8723 Acked Write 8 Sent Done Done false
30687 8724 Acked Read 5 Sent Done Done false
30688 8725 Acked Write 7 Sent Done Done false
30689 8726 Acked Write 8 Sent Done Done false
30690 8727 Acked Read 10 Sent Done Done false
30691 8728 Acked Read 8 Sent Done Done false
30692 8729 Acked Read 6 Sent Done Done false
30693 8730 Acked Read 4 Sent Done Done false
30694 8731 Acked Write 6 Sent Done Done false
30695 8732 Acked Write 4 Sent Done Done false
30696 8733 Acked Read 1 Sent Done Done false
30697 8734 Acked Write 7 Sent Done Done false
30698 8735 Acked Read 10 Sent Done Done false
30699 8736 Acked Write 6 Sent Done Done false
30700 8737 Acked Write 10 Sent Done Done false
30701 8738 Acked Read 5 Sent Done Done false
30702 8739 Acked Read 7 Sent Done Done false
30703 8740 Acked Read 4 Sent Done Done false
30704 8741 Acked Write 6 Sent Done Done false
30705 8742 Acked Read 1 Sent Done Done false
30706 8743 Acked Write 3 Sent Done Done false
30707 8744 Acked Flush 0 Sent Done Done false
30708 8745 Acked Read 9 Sent Done Done false
30709 8746 Acked Read 8 Sent Done Done false
30710 8747 Acked Read 9 Sent Done Done false
30711 8748 Acked Write 10 Sent Done Done false
30712 8749 Acked Write 9 Sent Done Done false
30713 8750 Acked Read 4 Sent Done Done false
30714 8751 Acked Write 10 Sent Done Done false
30715 8752 Acked Write 8 Sent Done Done false
30716 8753 Acked Write 3 Sent Done Done false
30717 8754 Acked Read 5 Sent Done Done false
30718 8755 Acked Flush 0 Sent Done Done false
30719 8756 Acked Read 5 Sent Done Done false
30720 8757 Acked Read 3 Sent Done Done false
30721 8758 Acked Write 7 Sent Done Done false
30722 8759 Acked Read 5 Sent Done Done false
30723 8760 Acked Read 5 Sent Done Done false
30724 8761 Acked Read 1 Sent Done Done false
30725 8762 Acked Read 3 Sent Done Done false
30726 8763 Acked Write 8 Sent Done Done false
30727 8764 Acked Read 3 Sent Done Done false
30728 8765 Acked Write 7 Sent Done Done false
30729 8766 Acked Read 1 Sent Done Done false
30730 8767 Acked Read 3 Sent Done Done false
30731 8768 Acked Read 2 Sent Done Done false
30732 8769 Acked Read 5 Sent Done Done false
30733 8770 Acked Read 3 Sent Done Done false
30734 8771 Acked Write 9 Sent Done Done false
30735 8772 Acked Write 4 Sent Done Done false
30736 8773 Acked Read 7 Sent Done Done false
30737 8774 Acked Flush 0 Sent Done Done false
30738 8775 Acked Write 6 Sent Done Done false
30739 8776 Acked Write 5 Sent Done Done false
30740 8777 Acked Read 6 Sent Done Done false
30741 8778 Acked Flush 0 Sent Done Done false
30742 8779 Acked Write 4 Sent Done Done false
30743 8780 Acked Write 3 Sent Done Done false
30744 8781 Acked Write 8 Sent Done Done false
30745 8782 Acked Read 10 Sent Done Done false
30746 8783 Acked Read 4 Sent Done Done false
30747 8784 Acked Flush 0 Sent Done Done false
30748 8785 Acked Write 2 Sent Done Done false
30749 8786 Acked Read 6 Sent Done Done false
30750 8787 Acked Read 10 Sent Done Done false
30751 8788 Acked Write 5 Sent Done Done false
30752 8789 Acked Read 6 Sent Done Done false
30753 8790 Acked Write 6 Sent Done Done false
30754 8791 Acked Read 8 Sent Done Done false
30755 8792 Acked Flush 0 Sent Done Done false
30756 8793 Acked Read 6 Sent Done Done false
30757 8794 Acked Read 1 Sent Done Done false
30758 8795 Acked Read 9 Sent Done Done false
30759 8796 Acked Read 4 Sent Done Done false
30760 8797 Acked Read 4 Sent Done Done false
30761 8798 Acked Read 4 Sent Done Done false
30762 8799 Acked Write 2 Sent Done Done false
30763 8800 Acked Write 8 Sent Done Done false
30764 8801 Acked Read 9 Sent Done Done false
30765 8802 Acked Flush 0 Sent Done Done false
30766 8803 Acked Read 2 Sent Done Done false
30767 8804 Acked Read 10 Sent Done Done false
30768 8805 Acked Read 7 Sent Done Done false
30769 8806 Acked Read 6 Sent Done Done false
30770 8807 Acked Read 7 Sent Done Done false
30771 8808 Acked Write 2 Sent Done Done false
30772 8809 Acked Write 2 Sent Done Done false
30773 8810 Acked Write 7 Sent Done Done false
30774 8811 Acked Read 5 Sent Done Done false
30775 8812 Acked Read 5 Sent Done Done false
30776 8813 Acked Read 3 Sent Done Done false
30777 8814 Acked Read 4 Sent Done Done false
30778 8815 Acked Write 10 Sent Done Done false
30779 8816 Acked Write 6 Sent Done Done false
30780 8817 Acked Read 3 Sent Done Done false
30781 8818 Acked Flush 0 Sent Done Done false
30782 8819 Acked Read 8 Sent Done Done false
30783 8820 Acked Read 9 Sent Done Done false
30784 8821 Acked Read 1 Sent Done Done false
30785 8822 Acked Write 6 Sent Done Done false
30786 8823 Acked Read 7 Sent Done Done false
30787 8824 Acked Read 5 Sent Done Done false
30788 8825 Acked Write 2 Sent Done Done false
30789 8826 Acked Write 9 Sent Done Done false
30790 8827 Acked Write 10 Sent Done Done false
30791 8828 Acked Write 3 Sent Done Done false
30792 8829 Acked Write 2 Sent Done Done false
30793 8830 Acked Read 7 Sent Done Done false
30794 8831 Acked Write 3 Sent Done Done false
30795 8832 Acked Write 6 Sent Done Done false
30796 8833 Acked Read 5 Sent Done Done false
30797 8834 Acked Write 8 Sent Done Done false
30798 8835 Acked Read 5 Sent Done Done false
30799 8836 Acked Read 1 Sent Done Done false
30800 8837 Acked Flush 0 Sent Done Done false
30801 8838 Acked Write 5 Sent Done Done false
30802 8839 Acked Write 5 Sent Done Done false
30803 8840 Acked Write 3 Sent Done Done false
30804 8841 Acked Read 7 Sent Done Done false
30805 8842 Acked Read 5 Sent Done Done false
30806 8843 Acked Read 4 Sent Done Done false
30807 8844 Acked Write 9 Sent Done Done false
30808 8845 Acked Write 10 Sent Done Done false
30809 8846 Acked Read 8 Sent Done Done false
30810 8847 Acked Write 2 Sent Done Done false
30811 STATES DS:0 DS:1 DS:2 TOTAL
30812 Sent 301 0 0 301
30813 Done 0 301 301 602
30814 Skipped 0 0 0 0
30815 Error 0 0 0 0
30816 Last Flush: Some(JobId(8546)) Some(JobId(8837)) Some(JobId(8837))
30817 Downstairs last five completed: 8546 8545 8544 8543 8542
30818 Upstairs last five completed: 8847 8846 8845 8844 8843
30819 CLIENT: Up:0 ds:301 act:2
308202025-03-17T22:41:19.940ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308212025-03-17T22:41:21.186ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308222025-03-17T22:41:21.186ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52765) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308232025-03-17T22:41:21.187ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308242025-03-17T22:41:21.188ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308252025-03-17T22:41:21.188ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308262025-03-17T22:41:21.188ZINFOcrucible: send last flush ID to this DS: Some(JobId(8546)) = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308272025-03-17T22:41:21.188ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308282025-03-17T22:41:21.188ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8546)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308292025-03-17T22:41:21.188ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8546)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
30830 ----------------------------------------------------------------
30831 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30832 Downstairs last five completed: 8848 8847 8846 8845 8844
30833 Upstairs last five completed: 8848 8847 8846 8845 8844
30834 CLIENT: Up:0 ds:0 act:3
30835 Replay: All jobs finished
30836 Main task has work to do, go find it
30837 got dsc Stop(2)
30838 stop 2
30839 Mar 17 22:41:23.439 INFO request completed, latency_us: 335, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
30840 , method: POST, req_id: cda3cf66-5bf2-4eba-9a14-ebd0089763c1, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
30841 Mar 17 22:41:23.440 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1de9e2d9-fc90-478f-8ffc-37d0e70c73ed, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
308422025-03-17T22:41:23.440ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308432025-03-17T22:41:23.440ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308442025-03-17T22:41:23.440ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
308452025-03-17T22:41:23.440ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
30846 [8830] Exited with: ExitStatus(unix_wait_status(9))
30847 [8830][2] reports Exit
30848 Mar 17 22:41:27.442 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/2, method: GET, req_id: fedf5503-7c8d-439a-9b16-b8bbe7e04c79, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
30849 001/300 Flush
30850 002/300 Read block 4052 len 36864
30851 003/300 Read block 4585 len 20480
30852 004/300 Read block 2600 len 8192
30853 005/300 Flush
30854 006/300 Write block 3183 len 12288 data: 7 7 8
30855 007/300 Read block 736 len 12288
30856 008/300 Write block 1360 len 8192 data: 6 8
30857 009/300 Read block 3401 len 8192
30858 010/300 Read block 284 len 40960
30859 011/300 Write block 68 len 8192 data: 6 5
30860 012/300 Write block 3024 len 8192 data: 6 6
30861 013/300 Read block 4838 len 40960
30862 014/300 Read block 2378 len 20480
30863 015/300 Write block 2013 len 28672 data: 5 5 5 3 4 4 4
30864 016/300 Write block 4118 len 28672 data: 7 8 7 5 3 3 2
30865 017/300 Read block 4295 len 40960
30866 018/300 Write block 60 len 36864 data: 4 4 4 3 4 5 6 7 7
30867 019/300 Write block 1672 len 32768 data: 5 5 6 4 4 5 3 4
30868 020/300 Write block 201 len 4096 data: 4
30869 021/300 Flush
30870 022/300 Read block 1684 len 8192
30871 023/300 Write block 2339 len 24576 data: 5 4 4 5 5 5
30872 024/300 Write block 440 len 36864 data: 8 8 7 8 9 8 8 7 5
30873 025/300 Write block 4121 len 16384 data: 6 4 4 3
30874 026/300 Read block 2969 len 20480
30875 027/300 Write block 3525 len 28672 data: 3 5 5 5 5 4 3
30876 028/300 Read block 3247 len 32768
30877 029/300 Read block 1788 len 8192
30878 030/300 Write block 1141 len 16384 data: 7 7 5 4
30879 031/300 Read block 489 len 32768
30880 032/300 Read block 2109 len 16384
30881 033/300 Read block 4519 len 28672
30882 034/300 Read block 318 len 40960
30883 035/300 Read block 601 len 16384
30884 036/300 Read block 2601 len 24576
30885 037/300 Read block 3260 len 12288
30886 038/300 Read block 4626 len 40960
30887 039/300 Read block 535 len 8192
30888 040/300 Read block 3976 len 36864
30889 041/300 Write block 4122 len 4096 data: 5
30890 042/300 Flush
30891 043/300 Read block 1434 len 16384
30892 044/300 Read block 3419 len 28672
30893 045/300 Write block 1807 len 24576 data: 5 5 5 4 5 5
30894 046/300 Read block 3606 len 40960
30895 047/300 Flush
30896 048/300 Read block 3325 len 40960
30897 049/300 Read block 2288 len 4096
30898 050/300 Read block 4192 len 32768
30899 051/300 Write block 273 len 28672 data: 7 8 9 8 7 7 6
30900 052/300 Read block 2566 len 36864
30901 053/300 Write block 487 len 16384 data: 7 7 6 5
30902 054/300 Read block 1791 len 20480
30903 055/300 Write block 4353 len 20480 data: 9 10 9 9 9
30904 056/300 Flush
30905 057/300 Read block 1878 len 8192
30906 058/300 Write block 741 len 4096 data: 4
30907 059/300 Read block 2841 len 24576
30908 060/300 Flush
30909 061/300 Read block 3422 len 24576
30910 062/300 Read block 3274 len 20480
30911 063/300 Read block 3001 len 28672
30912 064/300 Flush
30913 065/300 Flush
30914 066/300 Flush
30915 067/300 Read block 274 len 4096
30916 068/300 Read block 4750 len 36864
30917 069/300 Write block 177 len 8192 data: 6 8
30918 070/300 Flush
30919 071/300 Flush
30920 072/300 Read block 3428 len 16384
30921 073/300 Write block 4557 len 32768 data: 5 5 4 6 7 8 8 8
30922 074/300 Read block 2817 len 8192
30923 075/300 Read block 3818 len 40960
30924 076/300 Flush
30925 077/300 Write block 3760 len 40960 data: 6 3 3 2 2 2 2 3 4 3
30926 078/300 Write block 4790 len 20480 data: 7 7 7 7 7
30927 079/300 Write block 2422 len 36864 data: 3 5 6 6 6 6 6 5 6
30928 080/300 Read block 35 len 8192
30929 081/300 Read block 3325 len 16384
30930 082/300 Flush
30931 083/300 Read block 877 len 20480
30932 084/300 Flush
30933 085/300 Read block 4732 len 8192
30934 086/300 Write block 4803 len 36864 data: 8 8 7 7 7 6 7 6 6
30935 087/300 Read block 185 len 28672
30936 088/300 Write block 341 len 32768 data: 7 8 7 6 4 4 4 3
30937 089/300 Read block 3898 len 40960
30938 090/300 Read block 1005 len 4096
30939 091/300 Read block 3288 len 24576
30940 092/300 Read block 1662 len 20480
30941 093/300 Write block 4402 len 28672 data: 5 5 5 6 5 5 6
30942 094/300 Write block 1333 len 4096 data: 5
30943 095/300 Read block 4707 len 32768
30944 096/300 Read block 2079 len 32768
30945 097/300 Write block 569 len 36864 data: 6 5 4 4 5 5 5 5 5
30946 098/300 Flush
30947 099/300 Write block 4122 len 36864 data: 6 5 4 2 2 2 4 4 5
30948 100/300 Read block 3525 len 4096
30949 101/300 Write block 701 len 16384 data: 7 8 8 9
30950 102/300 Read block 4121 len 28672
30951 103/300 Write block 523 len 32768 data: 8 8 7 6 6 7 6 4
30952 104/300 Write block 3676 len 16384 data: 7 6 8 9
30953 105/300 Read block 3289 len 40960
30954 106/300 Read block 2024 len 12288
30955 107/300 Read block 3124 len 16384
30956 108/300 Write block 695 len 8192 data: 2 3
30957 109/300 Flush
30958 110/300 Read block 4969 len 4096
30959 111/300 Flush
30960 112/300 Read block 2066 len 4096
30961 113/300 Read block 3716 len 12288
30962 114/300 Read block 1476 len 32768
30963 115/300 Read block 837 len 36864
30964 116/300 Flush
30965 117/300 Write block 3919 len 20480 data: 8 6 7 7 5
30966 118/300 Write block 2245 len 28672 data: 2 2 2 2 2 2 3
30967 119/300 Flush
30968 120/300 Write block 1746 len 20480 data: 7 5 5 4 3
30969 121/300 Write block 2655 len 24576 data: 5 7 7 6 6 6
30970 122/300 Read block 903 len 32768
30971 123/300 Flush
30972 124/300 Read block 2442 len 12288
30973 125/300 Write block 184 len 40960 data: 4 4 3 3 3 4 5 5 6 7
30974 126/300 Read block 2601 len 36864
30975 127/300 Read block 624 len 32768
30976 128/300 Read block 2516 len 32768
30977 129/300 Flush
30978 130/300 Write block 3963 len 32768 data: 7 6 6 6 5 5 6 6
30979 131/300 Read block 4567 len 8192
30980 132/300 Flush
30981 133/300 Read block 3439 len 36864
30982 134/300 Read block 2838 len 12288
30983 135/300 Write block 4014 len 8192 data: 9 7
30984 136/300 Read block 4922 len 20480
30985 137/300 Flush
30986 138/300 Read block 2451 len 16384
30987 139/300 Read block 2774 len 4096
30988 140/300 Read block 3728 len 24576
30989 141/300 Write block 4995 len 4096 data: 3
30990 142/300 Write block 1047 len 4096 data: 3
30991 143/300 Write block 1543 len 32768 data: 4 4 5 5 6 7 6 5
30992 144/300 Read block 2527 len 32768
30993 145/300 Read block 4103 len 8192
30994 146/300 Read block 1262 len 8192
30995 147/300 Write block 2808 len 40960 data: 4 4 4 5 4 3 4 5 5 3
30996 148/300 Flush
30997 149/300 Read block 1160 len 32768
30998 150/300 Flush
30999 151/300 Write block 3956 len 16384 data: 3 3 4 4
31000 152/300 Read block 1391 len 40960
31001 153/300 Read block 3331 len 24576
31002 154/300 Read block 2756 len 32768
31003 155/300 Flush
31004 156/300 Read block 4970 len 28672
31005 157/300 Write block 1076 len 36864 data: 6 4 4 5 5 4 5 5 5
31006 158/300 Read block 339 len 40960
31007 159/300 Read block 1143 len 28672
31008 160/300 Write block 1526 len 12288 data: 6 5 4
31009 161/300 Read block 1703 len 12288
31010 162/300 Write block 2555 len 40960 data: 8 9 8 7 4 5 5 6 7 5
31011 163/300 Read block 1316 len 40960
31012 164/300 Write block 2209 len 4096 data: 7
31013 165/300 Read block 1249 len 40960
31014 166/300 Read block 120 len 12288
31015 167/300 Write block 3872 len 20480 data: 6 7 10 11 10
31016 168/300 Read block 314 len 36864
31017 169/300 Write block 4008 len 4096 data: 6
31018 170/300 Write block 2858 len 28672 data: 2 3 3 3 3 4 5
31019 171/300 Read block 1184 len 16384
31020 172/300 Write block 3411 len 8192 data: 7 9
31021 173/300 Read block 2169 len 16384
31022 174/300 Write block 2681 len 36864 data: 4 3 5 6 6 6 7 7 6
31023 175/300 Write block 3979 len 24576 data: 3 3 3 3 3 3
31024 176/300 Read block 3136 len 28672
31025 177/300 Write block 1696 len 32768 data: 3 2 2 3 5 6 7 7
31026 178/300 Write block 3790 len 16384 data: 4 4 7 7
31027 179/300 Read block 685 len 28672
31028 180/300 Read block 3552 len 4096
31029 181/300 Write block 1207 len 40960 data: 4 4 3 4 4 5 4 4 4 4
31030 182/300 Write block 4950 len 36864 data: 4 4 5 6 5 4 4 4 4
31031 183/300 Read block 1728 len 32768
31032 184/300 Write block 793 len 4096 data: 7
31033 185/300 Write block 1450 len 32768 data: 4 3 4 6 8 8 8 5
31034 186/300 Read block 4253 len 36864
31035 187/300 Read block 2033 len 12288
31036 188/300 Read block 2946 len 28672
31037 189/300 Read block 559 len 40960
31038 190/300 Read block 3574 len 20480
31039 191/300 Read block 3023 len 20480
31040 192/300 Read block 3800 len 28672
31041 193/300 Flush
31042 194/300 Read block 3430 len 16384
31043 195/300 Read block 4788 len 36864
31044 196/300 Write block 4930 len 24576 data: 6 7 6 8 10 9
31045 197/300 Read block 338 len 40960
31046 198/300 Write block 1566 len 16384 data: 3 4 4 4
31047 199/300 Read block 4502 len 32768
31048 200/300 Read block 3159 len 4096
31049 201/300 Read block 1743 len 4096
31050 202/300 Flush
31051 203/300 Read block 975 len 20480
31052 204/300 Read block 2580 len 16384
31053 205/300 Write block 2828 len 16384 data: 4 2 3 4
31054 206/300 Write block 3639 len 12288 data: 2 3 3
31055 207/300 Read block 482 len 20480
31056 208/300 Read block 4060 len 36864
31057 209/300 Write block 3232 len 4096 data: 5
31058 210/300 Read block 4328 len 4096
31059 211/300 Read block 4576 len 4096
31060 212/300 Read block 1930 len 16384
31061 213/300 Flush
31062 214/300 Write block 3234 len 36864 data: 7 7 6 7 7 7 7 6 5
31063 215/300 Flush
31064 216/300 Read block 2303 len 32768
31065 217/300 Write block 3702 len 12288 data: 5 5 6
31066 218/300 Read block 1716 len 4096
31067 219/300 Read block 1553 len 28672
31068 220/300 Write block 1960 len 12288 data: 8 7 9
31069 221/300 Write block 1250 len 36864 data: 5 6 6 6 6 6 5 4 6
31070 222/300 Write block 4233 len 32768 data: 8 7 9 7 7 5 5 5
31071 223/300 Write block 4735 len 36864 data: 4 4 5 4 2 3 3 5 6
31072 224/300 Read block 1434 len 16384
31073 225/300 Read block 4076 len 36864
31074 226/300 Read block 3230 len 32768
31075 227/300 Read block 2618 len 24576
31076 228/300 Write block 3366 len 16384 data: 6 7 7 6
31077 229/300 Read block 4557 len 40960
31078 230/300 Write block 7 len 20480 data: 4 5 5 6 5
31079 231/300 Read block 1028 len 20480
31080 232/300 Write block 4004 len 20480 data: 6 5 7 5 7
31081 233/300 Flush
31082 234/300 Read block 1619 len 24576
31083 235/300 Write block 654 len 28672 data: 6 6 6 7 4 4 5
31084 236/300 Flush
31085 237/300 Write block 2298 len 28672 data: 5 5 6 4 5 3 2
31086 238/300 Write block 2610 len 32768 data: 6 5 5 5 6 6 5 7
31087 239/300 Read block 34 len 20480
31088 240/300 Write block 1127 len 28672 data: 2 2 2 2 2 2 3
31089 241/300 Read block 3987 len 20480
31090 242/300 Read block 4726 len 12288
31091 243/300 Read block 1371 len 12288
31092 244/300 Read block 1920 len 16384
31093 245/300 Write block 4380 len 20480 data: 5 7 5 4 3
31094 246/300 Read block 176 len 28672
31095 247/300 Write block 2247 len 24576 data: 3 3 3 3 4 3
31096 248/300 Write block 4844 len 32768 data: 9 8 8 7 5 5 4 3
31097 249/300 Read block 2088 len 12288
31098 250/300 Write block 3086 len 20480 data: 4 3 5 5 4
31099 251/300 Write block 558 len 24576 data: 5 6 6 5 7 7
31100 252/300 Write block 4227 len 20480 data: 6 6 5 7 8
31101 253/300 Read block 3256 len 40960
31102 254/300 Write block 1854 len 24576 data: 5 5 4 4 6 5
31103 255/300 Write block 1547 len 16384 data: 7 8 7 6
31104 256/300 Read block 2185 len 20480
31105 257/300 Read block 1462 len 36864
31106 258/300 Read block 1209 len 40960
31107 259/300 Read block 1167 len 16384
31108 260/300 Flush
31109 261/300 Write block 2280 len 8192 data: 3 4
31110 262/300 Write block 3810 len 12288 data: 7 7 6
31111 263/300 Read block 3596 len 16384
31112 264/300 Read block 2133 len 24576
31113 265/300 Read block 2009 len 24576
31114 266/300 Read block 3955 len 40960
31115 267/300 Write block 1801 len 36864 data: 4 4 6 6 4 5 6 6 6
31116 268/300 Read block 2002 len 12288
31117 269/300 Write block 3227 len 36864 data: 5 4 4 3 4 6 7 8 8
31118 270/300 Read block 1016 len 32768
31119 271/300 Write block 923 len 24576 data: 3 2 4 5 5 7
31120 272/300 Read block 3979 len 24576
31121 273/300 Read block 3790 len 24576
31122 274/300 Write block 4610 len 16384 data: 4 5 6 6
31123 275/300 Read block 2652 len 36864
31124 276/300 Flush
31125 277/300 Read block 1632 len 40960
31126 278/300 Read block 709 len 32768
31127 279/300 Read block 3904 len 4096
31128 280/300 Write block 148 len 28672 data: 4 5 4 3 3 4 3
31129 281/300 Flush
31130 282/300 Read block 2196 len 40960
31131 283/300 Write block 2881 len 40960 data: 7 7 5 6 6 5 5 5 5 5
31132 284/300 Read block 2728 len 8192
31133 285/300 Read block 4527 len 20480
31134 286/300 Write block 2691 len 32768 data: 5 4 3 4 5 6 6 7
31135 287/300 Write block 3048 len 36864 data: 3 3 3 4 4 5 5 6 6
31136 288/300 Read block 4258 len 24576
31137 289/300 Read block 2908 len 32768
31138 290/300 Read block 4809 len 28672
31139 291/300 Write block 1449 len 28672 data: 4 5 4 5 7 9 9
31140 292/300 Read block 33 len 20480
31141 293/300 Read block 4274 len 12288
31142 294/300 Read block 4315 len 32768
31143 295/300 Read block 3364 len 20480
31144 296/300 Write block 1687 len 32768 data: 6 6 6 6 6 5 4 4
31145 297/300 Read block 3513 len 4096
31146 298/300 Read block 2310 len 32768
31147 299/300 Read block 2374 len 24576
31148 300/300 Read block 1434 len 12288
31149 301/300 Read block 3037 len 8192
31150 Main task has work to do, go find it
31151 got dsc Start(2)
31152 start 2
31153 Mar 17 22:41:27.702 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3717dc8a-c86e-4a36-8262-44f64caf3785, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
31154 Replay: started 2, returned:Ok(())
31155 ----------------------------------------------------------------
31156 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31157 States: Active Active Offline
31158 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31159 8849 Acked Flush 0 Done Done Sent false
31160 8850 Acked Read 9 Done Done Sent false
31161 8851 Acked Read 5 Done Done Sent false
31162 8852 Acked Read 2 Done Done Sent false
31163 8853 Acked Flush 0 Done Done Sent false
31164 8854 Acked Write 3 Done Done Sent false
31165 8855 Acked Read 3 Done Done Sent false
31166 8856 Acked Write 2 Done Done Sent false
31167 8857 Acked Read 2 Done Done Sent false
31168 8858 Acked Read 10 Done Done Sent false
31169 8859 Acked Write 2 Done Done Sent false
31170 8860 Acked Write 2 Done Done Sent false
31171 8861 Acked Read 10 Done Done Sent false
31172 8862 Acked Read 5 Done Done Sent false
31173 8863 Acked Write 7 Done Done Sent false
31174 8864 Acked Write 7 Done Done Sent false
31175 8865 Acked Read 10 Done Done Sent false
31176 8866 Acked Write 9 Done Done Sent false
31177 8867 Acked Write 8 Done Done Sent false
31178 8868 Acked Write 1 Done Done Sent false
31179 8869 Acked Flush 0 Done Done Sent false
31180 8870 Acked Read 2 Done Done Sent false
31181 8871 Acked Write 6 Done Done Sent false
31182 8872 Acked Write 9 Done Done Sent false
31183 8873 Acked Write 4 Done Done Sent false
31184 8874 Acked Read 5 Done Done Sent false
31185 8875 Acked Write 7 Done Done Sent false
31186 8876 Acked Read 8 Done Done Sent false
31187 8877 Acked Read 2 Done Done Sent false
31188 8878 Acked Write 4 Done Done Sent false
31189 8879 Acked Read 8 Done Done Sent false
31190 8880 Acked Read 4 Done Done Sent false
31191 8881 Acked Read 7 Done Done Sent false
31192 8882 Acked Read 10 Done Done Sent false
31193 8883 Acked Read 4 Done Done Sent false
31194 8884 Acked Read 6 Done Done Sent false
31195 8885 Acked Read 3 Done Done Sent false
31196 8886 Acked Read 10 Done Done Sent false
31197 8887 Acked Read 2 Done Done Sent false
31198 8888 Acked Read 9 Done Done Sent false
31199 8889 Acked Write 1 Done Done Sent false
31200 8890 Acked Flush 0 Done Done Sent false
31201 8891 Acked Read 4 Done Done Sent false
31202 8892 Acked Read 7 Done Done Sent false
31203 8893 Acked Write 6 Done Done Sent false
31204 8894 Acked Read 10 Done Done Sent false
31205 8895 Acked Flush 0 Done Done Sent false
31206 8896 Acked Read 10 Done Done Sent false
31207 8897 Acked Read 1 Done Done Sent false
31208 8898 Acked Read 8 Done Done Sent false
31209 8899 Acked Write 7 Done Done Sent false
31210 8900 Acked Read 9 Done Done Sent false
31211 8901 Acked Write 4 Done Done Sent false
31212 8902 Acked Read 5 Done Done Sent false
31213 8903 Acked Write 5 Done Done Sent false
31214 8904 Acked Flush 0 Done Done Sent false
31215 8905 Acked Read 2 Done Done Sent false
31216 8906 Acked Write 1 Done Done Sent false
31217 8907 Acked Read 6 Done Done Sent false
31218 8908 Acked Flush 0 Done Done Sent false
31219 8909 Acked Read 6 Done Done Sent false
31220 8910 Acked Read 5 Done Done Sent false
31221 8911 Acked Read 7 Done Done Sent false
31222 8912 Acked Flush 0 Done Done Sent false
31223 8913 Acked Flush 0 Done Done Sent false
31224 8914 Acked Flush 0 Done Done Sent false
31225 8915 Acked Read 1 Done Done Sent false
31226 8916 Acked Read 9 Done Done Sent false
31227 8917 Acked Write 2 Done Done Sent false
31228 8918 Acked Flush 0 Done Done Sent false
31229 8919 Acked Flush 0 Done Done Sent false
31230 8920 Acked Read 4 Done Done Sent false
31231 8921 Acked Write 8 Done Done Sent false
31232 8922 Acked Read 2 Done Done Sent false
31233 8923 Acked Read 10 Done Done Sent false
31234 8924 Acked Flush 0 Done Done Sent false
31235 8925 Acked Write 10 Done Done Sent false
31236 8926 Acked Write 5 Done Done Sent false
31237 8927 Acked Write 9 Done Done Sent false
31238 8928 Acked Read 2 Done Done Sent false
31239 8929 Acked Read 4 Done Done Sent false
31240 8930 Acked Flush 0 Done Done Sent false
31241 8931 Acked Read 5 Done Done Sent false
31242 8932 Acked Flush 0 Done Done Sent false
31243 8933 Acked Read 2 Done Done Sent false
31244 8934 Acked Write 9 Done Done Sent false
31245 8935 Acked Read 7 Done Done Sent false
31246 8936 Acked Write 8 Done Done Sent false
31247 8937 Acked Read 10 Done Done Sent false
31248 8938 Acked Read 1 Done Done Sent false
31249 8939 Acked Read 6 Done Done Sent false
31250 8940 Acked Read 5 Done Done Sent false
31251 8941 Acked Write 7 Done Done Sent false
31252 8942 Acked Write 1 Done Done Sent false
31253 8943 Acked Read 8 Done Done Sent false
31254 8944 Acked Read 8 Done Done Sent false
31255 8945 Acked Write 9 Done Done Sent false
31256 8946 Acked Flush 0 Done Done Sent false
31257 8947 Acked Write 9 Done Done Sent false
31258 8948 Acked Read 1 Done Done Sent false
31259 8949 Acked Write 4 Done Done Sent false
31260 8950 Acked Read 7 Done Done Sent false
31261 8951 Acked Write 8 Done Done Sent false
31262 8952 Acked Write 4 Done Done Sent false
31263 8953 Acked Read 10 Done Done Sent false
31264 8954 Acked Read 3 Done Done Sent false
31265 8955 Acked Read 4 Done Done Sent false
31266 8956 Acked Write 2 Done Done Sent false
31267 8957 Acked Flush 0 Done Done Sent false
31268 8958 Acked Read 1 Done Done Sent false
31269 8959 Acked Flush 0 Done Done Sent false
31270 8960 Acked Read 1 Done Done Sent false
31271 8961 Acked Read 3 Done Done Sent false
31272 8962 Acked Read 8 Done Done Sent false
31273 8963 Acked Read 9 Done Done Sent false
31274 8964 Acked Flush 0 Done Done Sent false
31275 8965 Acked Write 5 Done Done Sent false
31276 8966 Acked Write 7 Done Done Sent false
31277 8967 Acked Flush 0 Done Done Sent false
31278 8968 Acked Write 5 Done Done Sent false
31279 8969 Acked Write 6 Done Done Sent false
31280 8970 Acked Read 8 Done Done Sent false
31281 8971 Acked Flush 0 Done Done Sent false
31282 8972 Acked Read 3 Done Done Sent false
31283 8973 Acked Write 10 Done Done Sent false
31284 8974 Acked Read 9 Done Done Sent false
31285 8975 Acked Read 8 Done Done Sent false
31286 8976 Acked Read 8 Done Done Sent false
31287 8977 Acked Flush 0 Done Done Sent false
31288 8978 Acked Write 8 Done Done Sent false
31289 8979 Acked Read 2 Done Done Sent false
31290 8980 Acked Flush 0 Done Done Sent false
31291 8981 Acked Read 9 Done Done Sent false
31292 8982 Acked Read 3 Done Done Sent false
31293 8983 Acked Write 2 Done Done Sent false
31294 8984 Acked Read 5 Done Done Sent false
31295 8985 Acked Flush 0 Done Done Sent false
31296 8986 Acked Read 4 Done Done Sent false
31297 8987 Acked Read 1 Done Done Sent false
31298 8988 Acked Read 6 Done Done Sent false
31299 8989 Acked Write 1 Done Done Sent false
31300 8990 Acked Write 1 Done Done Sent false
31301 8991 Acked Write 8 Done Done Sent false
31302 8992 Acked Read 8 Done Done Sent false
31303 8993 Acked Read 2 Done Done Sent false
31304 8994 Acked Read 2 Done Done Sent false
31305 8995 Acked Write 10 Done Done Sent false
31306 8996 Acked Flush 0 Done Done Sent false
31307 8997 Acked Read 8 Done Done Sent false
31308 8998 Acked Flush 0 Done Done Sent false
31309 8999 Acked Write 4 Done Done Sent false
31310 9000 Acked Read 10 Done Done Sent false
31311 9001 Acked Read 6 Done Done Sent false
31312 9002 Acked Read 8 Done Done Sent false
31313 9003 Acked Flush 0 Done Done Sent false
31314 9004 Acked Read 7 Done Done Sent false
31315 9005 Acked Write 9 Done Done Sent false
31316 9006 Acked Read 10 Done Done Sent false
31317 9007 Acked Read 7 Done Done Sent false
31318 9008 Acked Write 3 Done Done Sent false
31319 9009 Acked Read 3 Done Done Sent false
31320 9010 Acked Write 10 Done Done Sent false
31321 9011 Acked Read 10 Done Done Sent false
31322 9012 Acked Write 1 Done Done Sent false
31323 9013 Acked Read 10 Done Done Sent false
31324 9014 Acked Read 3 Done Done Sent false
31325 9015 Acked Write 5 Done Done Sent false
31326 9016 Acked Read 9 Done Done Sent false
31327 9017 Acked Write 1 Done Done Sent false
31328 9018 Acked Write 7 Done Done Sent false
31329 9019 Acked Read 4 Done Done Sent false
31330 9020 Acked Write 2 Done Done Sent false
31331 9021 Acked Read 4 Done Done Sent false
31332 9022 Acked Write 9 Done Done Sent false
31333 9023 Acked Write 6 Done Done Sent false
31334 9024 Acked Read 7 Done Done Sent false
31335 9025 Acked Write 8 Done Done Sent false
31336 9026 Acked Write 4 Done Done Sent false
31337 9027 Acked Read 7 Done Done Sent false
31338 9028 Acked Read 1 Done Done Sent false
31339 9029 Acked Write 10 Done Done Sent false
31340 9030 Acked Write 9 Done Done Sent false
31341 9031 Acked Read 8 Done Done Sent false
31342 9032 Acked Write 1 Done Done Sent false
31343 9033 Acked Write 8 Done Done Sent false
31344 9034 Acked Read 9 Done Done Sent false
31345 9035 Acked Read 3 Done Done Sent false
31346 9036 Acked Read 7 Done Done Sent false
31347 9037 Acked Read 10 Done Done Sent false
31348 9038 Acked Read 5 Done Done Sent false
31349 9039 Acked Read 5 Done Done Sent false
31350 9040 Acked Read 7 Done Done Sent false
31351 9041 Acked Flush 0 Done Done Sent false
31352 9042 Acked Read 4 Done Done Sent false
31353 9043 Acked Read 9 Done Done Sent false
31354 9044 Acked Write 6 Done Done Sent false
31355 9045 Acked Read 10 Done Done Sent false
31356 9046 Acked Write 4 Done Done Sent false
31357 9047 Acked Read 8 Done Done Sent false
31358 9048 Acked Read 1 Done Done Sent false
31359 9049 Acked Read 1 Done Done Sent false
31360 9050 Acked Flush 0 Done Done Sent false
31361 9051 Acked Read 5 Done Done Sent false
31362 9052 Acked Read 4 Done Done Sent false
31363 9053 Acked Write 4 Done Done Sent false
31364 9054 Acked Write 3 Done Done Sent false
31365 9055 Acked Read 5 Done Done Sent false
31366 9056 Acked Read 9 Done Done Sent false
31367 9057 Acked Write 1 Done Done Sent false
31368 9058 Acked Read 1 Done Done Sent false
31369 9059 Acked Read 1 Done Done Sent false
31370 9060 Acked Read 4 Done Done Sent false
31371 9061 Acked Flush 0 Done Done Sent false
31372 9062 Acked Write 9 Done Done Sent false
31373 9063 Acked Flush 0 Done Done Sent false
31374 9064 Acked Read 8 Done Done Sent false
31375 9065 Acked Write 3 Done Done Sent false
31376 9066 Acked Read 1 Done Done Sent false
31377 9067 Acked Read 7 Done Done Sent false
31378 9068 Acked Write 3 Done Done Sent false
31379 9069 Acked Write 9 Done Done Sent false
31380 9070 Acked Write 8 Done Done Sent false
31381 9071 Acked Write 9 Done Done Sent false
31382 9072 Acked Read 4 Done Done Sent false
31383 9073 Acked Read 9 Done Done Sent false
31384 9074 Acked Read 8 Done Done Sent false
31385 9075 Acked Read 6 Done Done Sent false
31386 9076 Acked Write 4 Done Done Sent false
31387 9077 Acked Read 10 Done Done Sent false
31388 9078 Acked Write 5 Done Done Sent false
31389 9079 Acked Read 5 Done Done Sent false
31390 9080 Acked Write 5 Done Done Sent false
31391 9081 Acked Flush 0 Done Done Sent false
31392 9082 Acked Read 6 Done Done Sent false
31393 9083 Acked Write 7 Done Done Sent false
31394 9084 Acked Flush 0 Done Done Sent false
31395 9085 Acked Write 7 Done Done Sent false
31396 9086 Acked Write 8 Done Done Sent false
31397 9087 Acked Read 5 Done Done Sent false
31398 9088 Acked Write 7 Done Done Sent false
31399 9089 Acked Read 5 Done Done Sent false
31400 9090 Acked Read 3 Done Done Sent false
31401 9091 Acked Read 3 Done Done Sent false
31402 9092 Acked Read 4 Done Done Sent false
31403 9093 Acked Write 5 Done Done Sent false
31404 9094 Acked Read 7 Done Done Sent false
31405 9095 Acked Write 6 Done Done Sent false
31406 9096 Acked Write 8 Done Done Sent false
31407 9097 Acked Read 3 Done Done Sent false
31408 9098 Acked Write 5 Done Done Sent false
31409 9099 Acked Write 6 Done Done Sent false
31410 9100 Acked Write 5 Done Done Sent false
31411 9101 Acked Read 10 Done Done Sent false
31412 9102 Acked Write 6 Done Done Sent false
31413 9103 Acked Write 4 Done Done Sent false
31414 9104 Acked Read 5 Done Done Sent false
31415 9105 Acked Read 9 Done Done Sent false
31416 9106 Acked Read 10 Done Done Sent false
31417 9107 Acked Read 4 Done Done Sent false
31418 9108 Acked Flush 0 Done Done Sent false
31419 9109 Acked Write 2 Done Done Sent false
31420 9110 Acked Write 3 Done Done Sent false
31421 9111 Acked Read 4 Done Done Sent false
31422 9112 Acked Read 6 Done Done Sent false
31423 9113 Acked Read 6 Done Done Sent false
31424 9114 Acked Read 10 Done Done Sent false
31425 9115 Acked Write 9 Done Done Sent false
31426 9116 Acked Read 3 Done Done Sent false
31427 9117 Acked Write 9 Done Done Sent false
31428 9118 Acked Read 8 Done Done Sent false
31429 9119 Acked Write 6 Done Done Sent false
31430 9120 Acked Read 6 Done Done Sent false
31431 9121 Acked Read 6 Done Done Sent false
31432 9122 Acked Write 4 Done Done Sent false
31433 9123 Acked Read 9 Done Done Sent false
31434 9124 Acked Flush 0 Done Done Sent false
31435 9125 Acked Read 10 Done Done Sent false
31436 9126 Acked Read 8 Done Done Sent false
31437 9127 Acked Read 1 Done Done Sent false
31438 9128 Acked Write 7 Done Done Sent false
31439 9129 Acked Flush 0 Done Done Sent false
31440 9130 Acked Read 10 Done Done Sent false
31441 9131 Acked Write 10 Done Done Sent false
31442 9132 Acked Read 2 Done Done Sent false
31443 9133 Acked Read 5 Done Done Sent false
31444 9134 Acked Write 8 Done Done Sent false
31445 9135 Acked Write 9 Done Done Sent false
31446 9136 Acked Read 6 Done Done Sent false
31447 9137 Acked Read 8 Done Done Sent false
31448 9138 Acked Read 7 Done Done Sent false
31449 9139 Acked Write 7 Done Done Sent false
31450 9140 Acked Read 5 Done Done Sent false
31451 9141 Acked Read 3 Done Done Sent false
31452 9142 Acked Read 8 Done Done Sent false
31453 9143 Acked Read 5 Done Done Sent false
31454 9144 Acked Write 8 Done Done Sent false
31455 9145 Acked Read 1 Done Done Sent false
31456 9146 Acked Read 8 Done Done Sent false
31457 9147 Acked Read 6 Done Done Sent false
31458 9148 Acked Read 3 Done Done Sent false
31459 9149 Acked Read 2 Done Done Sent false
31460 STATES DS:0 DS:1 DS:2 TOTAL
31461 Sent 0 0 301 301
31462 Done 301 301 0 602
31463 Skipped 0 0 0 0
31464 Error 0 0 0 0
31465 Last Flush: Some(JobId(9129)) Some(JobId(9129)) Some(JobId(8848))
31466 Downstairs last five completed: 8848 8847 8846 8845 8844
31467 Upstairs last five completed: 9149 9148 9147 9146 9145
31468 CLIENT: Up:0 ds:301 act:2
314692025-03-17T22:41:28.205ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
314702025-03-17T22:41:28.953ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
314712025-03-17T22:41:29.955ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
31472 ----------------------------------------------------------------
31473 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31474 States: Active Active Offline
31475 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31476 8849 Acked Flush 0 Done Done Sent false
31477 8850 Acked Read 9 Done Done Sent false
31478 8851 Acked Read 5 Done Done Sent false
31479 8852 Acked Read 2 Done Done Sent false
31480 8853 Acked Flush 0 Done Done Sent false
31481 8854 Acked Write 3 Done Done Sent false
31482 8855 Acked Read 3 Done Done Sent false
31483 8856 Acked Write 2 Done Done Sent false
31484 8857 Acked Read 2 Done Done Sent false
31485 8858 Acked Read 10 Done Done Sent false
31486 8859 Acked Write 2 Done Done Sent false
31487 8860 Acked Write 2 Done Done Sent false
31488 8861 Acked Read 10 Done Done Sent false
31489 8862 Acked Read 5 Done Done Sent false
31490 8863 Acked Write 7 Done Done Sent false
31491 8864 Acked Write 7 Done Done Sent false
31492 8865 Acked Read 10 Done Done Sent false
31493 8866 Acked Write 9 Done Done Sent false
31494 8867 Acked Write 8 Done Done Sent false
31495 8868 Acked Write 1 Done Done Sent false
31496 8869 Acked Flush 0 Done Done Sent false
31497 8870 Acked Read 2 Done Done Sent false
31498 8871 Acked Write 6 Done Done Sent false
31499 8872 Acked Write 9 Done Done Sent false
31500 8873 Acked Write 4 Done Done Sent false
31501 8874 Acked Read 5 Done Done Sent false
31502 8875 Acked Write 7 Done Done Sent false
31503 8876 Acked Read 8 Done Done Sent false
31504 8877 Acked Read 2 Done Done Sent false
31505 8878 Acked Write 4 Done Done Sent false
31506 8879 Acked Read 8 Done Done Sent false
31507 8880 Acked Read 4 Done Done Sent false
31508 8881 Acked Read 7 Done Done Sent false
31509 8882 Acked Read 10 Done Done Sent false
31510 8883 Acked Read 4 Done Done Sent false
31511 8884 Acked Read 6 Done Done Sent false
31512 8885 Acked Read 3 Done Done Sent false
31513 8886 Acked Read 10 Done Done Sent false
31514 8887 Acked Read 2 Done Done Sent false
31515 8888 Acked Read 9 Done Done Sent false
31516 8889 Acked Write 1 Done Done Sent false
31517 8890 Acked Flush 0 Done Done Sent false
31518 8891 Acked Read 4 Done Done Sent false
31519 8892 Acked Read 7 Done Done Sent false
31520 8893 Acked Write 6 Done Done Sent false
31521 8894 Acked Read 10 Done Done Sent false
31522 8895 Acked Flush 0 Done Done Sent false
31523 8896 Acked Read 10 Done Done Sent false
31524 8897 Acked Read 1 Done Done Sent false
31525 8898 Acked Read 8 Done Done Sent false
31526 8899 Acked Write 7 Done Done Sent false
31527 8900 Acked Read 9 Done Done Sent false
31528 8901 Acked Write 4 Done Done Sent false
31529 8902 Acked Read 5 Done Done Sent false
31530 8903 Acked Write 5 Done Done Sent false
31531 8904 Acked Flush 0 Done Done Sent false
31532 8905 Acked Read 2 Done Done Sent false
31533 8906 Acked Write 1 Done Done Sent false
31534 8907 Acked Read 6 Done Done Sent false
31535 8908 Acked Flush 0 Done Done Sent false
31536 8909 Acked Read 6 Done Done Sent false
31537 8910 Acked Read 5 Done Done Sent false
31538 8911 Acked Read 7 Done Done Sent false
31539 8912 Acked Flush 0 Done Done Sent false
31540 8913 Acked Flush 0 Done Done Sent false
31541 8914 Acked Flush 0 Done Done Sent false
31542 8915 Acked Read 1 Done Done Sent false
31543 8916 Acked Read 9 Done Done Sent false
31544 8917 Acked Write 2 Done Done Sent false
31545 8918 Acked Flush 0 Done Done Sent false
31546 8919 Acked Flush 0 Done Done Sent false
31547 8920 Acked Read 4 Done Done Sent false
31548 8921 Acked Write 8 Done Done Sent false
31549 8922 Acked Read 2 Done Done Sent false
31550 8923 Acked Read 10 Done Done Sent false
31551 8924 Acked Flush 0 Done Done Sent false
31552 8925 Acked Write 10 Done Done Sent false
31553 8926 Acked Write 5 Done Done Sent false
31554 8927 Acked Write 9 Done Done Sent false
31555 8928 Acked Read 2 Done Done Sent false
31556 8929 Acked Read 4 Done Done Sent false
31557 8930 Acked Flush 0 Done Done Sent false
31558 8931 Acked Read 5 Done Done Sent false
31559 8932 Acked Flush 0 Done Done Sent false
31560 8933 Acked Read 2 Done Done Sent false
31561 8934 Acked Write 9 Done Done Sent false
31562 8935 Acked Read 7 Done Done Sent false
31563 8936 Acked Write 8 Done Done Sent false
31564 8937 Acked Read 10 Done Done Sent false
31565 8938 Acked Read 1 Done Done Sent false
31566 8939 Acked Read 6 Done Done Sent false
31567 8940 Acked Read 5 Done Done Sent false
31568 8941 Acked Write 7 Done Done Sent false
31569 8942 Acked Write 1 Done Done Sent false
31570 8943 Acked Read 8 Done Done Sent false
31571 8944 Acked Read 8 Done Done Sent false
31572 8945 Acked Write 9 Done Done Sent false
31573 8946 Acked Flush 0 Done Done Sent false
31574 8947 Acked Write 9 Done Done Sent false
31575 8948 Acked Read 1 Done Done Sent false
31576 8949 Acked Write 4 Done Done Sent false
31577 8950 Acked Read 7 Done Done Sent false
31578 8951 Acked Write 8 Done Done Sent false
31579 8952 Acked Write 4 Done Done Sent false
31580 8953 Acked Read 10 Done Done Sent false
31581 8954 Acked Read 3 Done Done Sent false
31582 8955 Acked Read 4 Done Done Sent false
31583 8956 Acked Write 2 Done Done Sent false
31584 8957 Acked Flush 0 Done Done Sent false
31585 8958 Acked Read 1 Done Done Sent false
31586 8959 Acked Flush 0 Done Done Sent false
31587 8960 Acked Read 1 Done Done Sent false
31588 8961 Acked Read 3 Done Done Sent false
31589 8962 Acked Read 8 Done Done Sent false
31590 8963 Acked Read 9 Done Done Sent false
31591 8964 Acked Flush 0 Done Done Sent false
31592 8965 Acked Write 5 Done Done Sent false
31593 8966 Acked Write 7 Done Done Sent false
31594 8967 Acked Flush 0 Done Done Sent false
31595 8968 Acked Write 5 Done Done Sent false
31596 8969 Acked Write 6 Done Done Sent false
31597 8970 Acked Read 8 Done Done Sent false
31598 8971 Acked Flush 0 Done Done Sent false
31599 8972 Acked Read 3 Done Done Sent false
31600 8973 Acked Write 10 Done Done Sent false
31601 8974 Acked Read 9 Done Done Sent false
31602 8975 Acked Read 8 Done Done Sent false
31603 8976 Acked Read 8 Done Done Sent false
31604 8977 Acked Flush 0 Done Done Sent false
31605 8978 Acked Write 8 Done Done Sent false
31606 8979 Acked Read 2 Done Done Sent false
31607 8980 Acked Flush 0 Done Done Sent false
31608 8981 Acked Read 9 Done Done Sent false
31609 8982 Acked Read 3 Done Done Sent false
31610 8983 Acked Write 2 Done Done Sent false
31611 8984 Acked Read 5 Done Done Sent false
31612 8985 Acked Flush 0 Done Done Sent false
31613 8986 Acked Read 4 Done Done Sent false
31614 8987 Acked Read 1 Done Done Sent false
31615 8988 Acked Read 6 Done Done Sent false
31616 8989 Acked Write 1 Done Done Sent false
31617 8990 Acked Write 1 Done Done Sent false
31618 8991 Acked Write 8 Done Done Sent false
31619 8992 Acked Read 8 Done Done Sent false
31620 8993 Acked Read 2 Done Done Sent false
31621 8994 Acked Read 2 Done Done Sent false
31622 8995 Acked Write 10 Done Done Sent false
31623 8996 Acked Flush 0 Done Done Sent false
31624 8997 Acked Read 8 Done Done Sent false
31625 8998 Acked Flush 0 Done Done Sent false
31626 8999 Acked Write 4 Done Done Sent false
31627 9000 Acked Read 10 Done Done Sent false
31628 9001 Acked Read 6 Done Done Sent false
31629 9002 Acked Read 8 Done Done Sent false
31630 9003 Acked Flush 0 Done Done Sent false
31631 9004 Acked Read 7 Done Done Sent false
31632 9005 Acked Write 9 Done Done Sent false
31633 9006 Acked Read 10 Done Done Sent false
31634 9007 Acked Read 7 Done Done Sent false
31635 9008 Acked Write 3 Done Done Sent false
31636 9009 Acked Read 3 Done Done Sent false
31637 9010 Acked Write 10 Done Done Sent false
31638 9011 Acked Read 10 Done Done Sent false
31639 9012 Acked Write 1 Done Done Sent false
31640 9013 Acked Read 10 Done Done Sent false
31641 9014 Acked Read 3 Done Done Sent false
31642 9015 Acked Write 5 Done Done Sent false
31643 9016 Acked Read 9 Done Done Sent false
31644 9017 Acked Write 1 Done Done Sent false
31645 9018 Acked Write 7 Done Done Sent false
31646 9019 Acked Read 4 Done Done Sent false
31647 9020 Acked Write 2 Done Done Sent false
31648 9021 Acked Read 4 Done Done Sent false
31649 9022 Acked Write 9 Done Done Sent false
31650 9023 Acked Write 6 Done Done Sent false
31651 9024 Acked Read 7 Done Done Sent false
31652 9025 Acked Write 8 Done Done Sent false
31653 9026 Acked Write 4 Done Done Sent false
31654 9027 Acked Read 7 Done Done Sent false
31655 9028 Acked Read 1 Done Done Sent false
31656 9029 Acked Write 10 Done Done Sent false
31657 9030 Acked Write 9 Done Done Sent false
31658 9031 Acked Read 8 Done Done Sent false
31659 9032 Acked Write 1 Done Done Sent false
31660 9033 Acked Write 8 Done Done Sent false
31661 9034 Acked Read 9 Done Done Sent false
31662 9035 Acked Read 3 Done Done Sent false
31663 9036 Acked Read 7 Done Done Sent false
31664 9037 Acked Read 10 Done Done Sent false
31665 9038 Acked Read 5 Done Done Sent false
31666 9039 Acked Read 5 Done Done Sent false
31667 9040 Acked Read 7 Done Done Sent false
31668 9041 Acked Flush 0 Done Done Sent false
31669 9042 Acked Read 4 Done Done Sent false
31670 9043 Acked Read 9 Done Done Sent false
31671 9044 Acked Write 6 Done Done Sent false
31672 9045 Acked Read 10 Done Done Sent false
31673 9046 Acked Write 4 Done Done Sent false
31674 9047 Acked Read 8 Done Done Sent false
31675 9048 Acked Read 1 Done Done Sent false
31676 9049 Acked Read 1 Done Done Sent false
31677 9050 Acked Flush 0 Done Done Sent false
31678 9051 Acked Read 5 Done Done Sent false
31679 9052 Acked Read 4 Done Done Sent false
31680 9053 Acked Write 4 Done Done Sent false
31681 9054 Acked Write 3 Done Done Sent false
31682 9055 Acked Read 5 Done Done Sent false
31683 9056 Acked Read 9 Done Done Sent false
31684 9057 Acked Write 1 Done Done Sent false
31685 9058 Acked Read 1 Done Done Sent false
31686 9059 Acked Read 1 Done Done Sent false
31687 9060 Acked Read 4 Done Done Sent false
31688 9061 Acked Flush 0 Done Done Sent false
31689 9062 Acked Write 9 Done Done Sent false
31690 9063 Acked Flush 0 Done Done Sent false
31691 9064 Acked Read 8 Done Done Sent false
31692 9065 Acked Write 3 Done Done Sent false
31693 9066 Acked Read 1 Done Done Sent false
31694 9067 Acked Read 7 Done Done Sent false
31695 9068 Acked Write 3 Done Done Sent false
31696 9069 Acked Write 9 Done Done Sent false
31697 9070 Acked Write 8 Done Done Sent false
31698 9071 Acked Write 9 Done Done Sent false
31699 9072 Acked Read 4 Done Done Sent false
31700 9073 Acked Read 9 Done Done Sent false
31701 9074 Acked Read 8 Done Done Sent false
31702 9075 Acked Read 6 Done Done Sent false
31703 9076 Acked Write 4 Done Done Sent false
31704 9077 Acked Read 10 Done Done Sent false
31705 9078 Acked Write 5 Done Done Sent false
31706 9079 Acked Read 5 Done Done Sent false
31707 9080 Acked Write 5 Done Done Sent false
31708 9081 Acked Flush 0 Done Done Sent false
31709 9082 Acked Read 6 Done Done Sent false
31710 9083 Acked Write 7 Done Done Sent false
31711 9084 Acked Flush 0 Done Done Sent false
31712 9085 Acked Write 7 Done Done Sent false
31713 9086 Acked Write 8 Done Done Sent false
31714 9087 Acked Read 5 Done Done Sent false
31715 9088 Acked Write 7 Done Done Sent false
31716 9089 Acked Read 5 Done Done Sent false
31717 9090 Acked Read 3 Done Done Sent false
31718 9091 Acked Read 3 Done Done Sent false
31719 9092 Acked Read 4 Done Done Sent false
31720 9093 Acked Write 5 Done Done Sent false
31721 9094 Acked Read 7 Done Done Sent false
31722 9095 Acked Write 6 Done Done Sent false
31723 9096 Acked Write 8 Done Done Sent false
31724 9097 Acked Read 3 Done Done Sent false
31725 9098 Acked Write 5 Done Done Sent false
31726 9099 Acked Write 6 Done Done Sent false
31727 9100 Acked Write 5 Done Done Sent false
31728 9101 Acked Read 10 Done Done Sent false
31729 9102 Acked Write 6 Done Done Sent false
31730 9103 Acked Write 4 Done Done Sent false
31731 9104 Acked Read 5 Done Done Sent false
31732 9105 Acked Read 9 Done Done Sent false
31733 9106 Acked Read 10 Done Done Sent false
31734 9107 Acked Read 4 Done Done Sent false
31735 9108 Acked Flush 0 Done Done Sent false
31736 9109 Acked Write 2 Done Done Sent false
31737 9110 Acked Write 3 Done Done Sent false
31738 9111 Acked Read 4 Done Done Sent false
31739 9112 Acked Read 6 Done Done Sent false
31740 9113 Acked Read 6 Done Done Sent false
31741 9114 Acked Read 10 Done Done Sent false
31742 9115 Acked Write 9 Done Done Sent false
31743 9116 Acked Read 3 Done Done Sent false
31744 9117 Acked Write 9 Done Done Sent false
31745 9118 Acked Read 8 Done Done Sent false
31746 9119 Acked Write 6 Done Done Sent false
31747 9120 Acked Read 6 Done Done Sent false
31748 9121 Acked Read 6 Done Done Sent false
31749 9122 Acked Write 4 Done Done Sent false
31750 9123 Acked Read 9 Done Done Sent false
31751 9124 Acked Flush 0 Done Done Sent false
31752 9125 Acked Read 10 Done Done Sent false
31753 9126 Acked Read 8 Done Done Sent false
31754 9127 Acked Read 1 Done Done Sent false
31755 9128 Acked Write 7 Done Done Sent false
31756 9129 Acked Flush 0 Done Done Sent false
31757 9130 Acked Read 10 Done Done Sent false
31758 9131 Acked Write 10 Done Done Sent false
31759 9132 Acked Read 2 Done Done Sent false
31760 9133 Acked Read 5 Done Done Sent false
31761 9134 Acked Write 8 Done Done Sent false
31762 9135 Acked Write 9 Done Done Sent false
31763 9136 Acked Read 6 Done Done Sent false
31764 9137 Acked Read 8 Done Done Sent false
31765 9138 Acked Read 7 Done Done Sent false
31766 9139 Acked Write 7 Done Done Sent false
31767 9140 Acked Read 5 Done Done Sent false
31768 9141 Acked Read 3 Done Done Sent false
31769 9142 Acked Read 8 Done Done Sent false
31770 9143 Acked Read 5 Done Done Sent false
31771 9144 Acked Write 8 Done Done Sent false
31772 9145 Acked Read 1 Done Done Sent false
31773 9146 Acked Read 8 Done Done Sent false
31774 9147 Acked Read 6 Done Done Sent false
31775 9148 Acked Read 3 Done Done Sent false
31776 9149 Acked Read 2 Done Done Sent false
31777 STATES DS:0 DS:1 DS:2 TOTAL
31778 Sent 0 0 301 301
31779 Done 301 301 0 602
31780 Skipped 0 0 0 0
31781 Error 0 0 0 0
31782 Last Flush: Some(JobId(9129)) Some(JobId(9129)) Some(JobId(8848))
31783 Downstairs last five completed: 8848 8847 8846 8845 8844
31784 Upstairs last five completed: 9149 9148 9147 9146 9145
31785 CLIENT: Up:0 ds:301 act:2
317862025-03-17T22:41:32.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317872025-03-17T22:41:32.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317882025-03-17T22:41:33.441ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317892025-03-17T22:41:33.441ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317902025-03-17T22:41:33.441ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317912025-03-17T22:41:33.441ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317922025-03-17T22:41:33.442ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317932025-03-17T22:41:33.962ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
317942025-03-17T22:41:34.964ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
31795 [8830] got start action
31796 [8830] I am going to restart
31797 Starting downstairs at port 8830
31798 Make output file at "/tmp/dsc/downstairs-8830.txt"
31799 [8830][2] reports Starting
31800 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1235)
31801 [8830][2] reports Running
31802 ----------------------------------------------------------------
31803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31804 States: Active Active Offline
31805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31806 8849 Acked Flush 0 Done Done Sent false
31807 8850 Acked Read 9 Done Done Sent false
31808 8851 Acked Read 5 Done Done Sent false
31809 8852 Acked Read 2 Done Done Sent false
31810 8853 Acked Flush 0 Done Done Sent false
31811 8854 Acked Write 3 Done Done Sent false
31812 8855 Acked Read 3 Done Done Sent false
31813 8856 Acked Write 2 Done Done Sent false
31814 8857 Acked Read 2 Done Done Sent false
31815 8858 Acked Read 10 Done Done Sent false
31816 8859 Acked Write 2 Done Done Sent false
31817 8860 Acked Write 2 Done Done Sent false
31818 8861 Acked Read 10 Done Done Sent false
31819 8862 Acked Read 5 Done Done Sent false
31820 8863 Acked Write 7 Done Done Sent false
31821 8864 Acked Write 7 Done Done Sent false
31822 8865 Acked Read 10 Done Done Sent false
31823 8866 Acked Write 9 Done Done Sent false
31824 8867 Acked Write 8 Done Done Sent false
31825 8868 Acked Write 1 Done Done Sent false
31826 8869 Acked Flush 0 Done Done Sent false
31827 8870 Acked Read 2 Done Done Sent false
31828 8871 Acked Write 6 Done Done Sent false
31829 8872 Acked Write 9 Done Done Sent false
31830 8873 Acked Write 4 Done Done Sent false
31831 8874 Acked Read 5 Done Done Sent false
31832 8875 Acked Write 7 Done Done Sent false
31833 8876 Acked Read 8 Done Done Sent false
31834 8877 Acked Read 2 Done Done Sent false
31835 8878 Acked Write 4 Done Done Sent false
31836 8879 Acked Read 8 Done Done Sent false
31837 8880 Acked Read 4 Done Done Sent false
31838 8881 Acked Read 7 Done Done Sent false
31839 8882 Acked Read 10 Done Done Sent false
31840 8883 Acked Read 4 Done Done Sent false
31841 8884 Acked Read 6 Done Done Sent false
31842 8885 Acked Read 3 Done Done Sent false
31843 8886 Acked Read 10 Done Done Sent false
31844 8887 Acked Read 2 Done Done Sent false
31845 8888 Acked Read 9 Done Done Sent false
31846 8889 Acked Write 1 Done Done Sent false
31847 8890 Acked Flush 0 Done Done Sent false
31848 8891 Acked Read 4 Done Done Sent false
31849 8892 Acked Read 7 Done Done Sent false
31850 8893 Acked Write 6 Done Done Sent false
31851 8894 Acked Read 10 Done Done Sent false
31852 8895 Acked Flush 0 Done Done Sent false
31853 8896 Acked Read 10 Done Done Sent false
31854 8897 Acked Read 1 Done Done Sent false
31855 8898 Acked Read 8 Done Done Sent false
31856 8899 Acked Write 7 Done Done Sent false
31857 8900 Acked Read 9 Done Done Sent false
31858 8901 Acked Write 4 Done Done Sent false
31859 8902 Acked Read 5 Done Done Sent false
31860 8903 Acked Write 5 Done Done Sent false
31861 8904 Acked Flush 0 Done Done Sent false
31862 8905 Acked Read 2 Done Done Sent false
31863 8906 Acked Write 1 Done Done Sent false
31864 8907 Acked Read 6 Done Done Sent false
31865 8908 Acked Flush 0 Done Done Sent false
31866 8909 Acked Read 6 Done Done Sent false
31867 8910 Acked Read 5 Done Done Sent false
31868 8911 Acked Read 7 Done Done Sent false
31869 8912 Acked Flush 0 Done Done Sent false
31870 8913 Acked Flush 0 Done Done Sent false
31871 8914 Acked Flush 0 Done Done Sent false
31872 8915 Acked Read 1 Done Done Sent false
31873 8916 Acked Read 9 Done Done Sent false
31874 8917 Acked Write 2 Done Done Sent false
31875 8918 Acked Flush 0 Done Done Sent false
31876 8919 Acked Flush 0 Done Done Sent false
31877 8920 Acked Read 4 Done Done Sent false
31878 8921 Acked Write 8 Done Done Sent false
31879 8922 Acked Read 2 Done Done Sent false
31880 8923 Acked Read 10 Done Done Sent false
31881 8924 Acked Flush 0 Done Done Sent false
31882 8925 Acked Write 10 Done Done Sent false
31883 8926 Acked Write 5 Done Done Sent false
31884 8927 Acked Write 9 Done Done Sent false
31885 8928 Acked Read 2 Done Done Sent false
31886 8929 Acked Read 4 Done Done Sent false
31887 8930 Acked Flush 0 Done Done Sent false
31888 8931 Acked Read 5 Done Done Sent false
31889 8932 Acked Flush 0 Done Done Sent false
31890 8933 Acked Read 2 Done Done Sent false
31891 8934 Acked Write 9 Done Done Sent false
31892 8935 Acked Read 7 Done Done Sent false
31893 8936 Acked Write 8 Done Done Sent false
31894 8937 Acked Read 10 Done Done Sent false
31895 8938 Acked Read 1 Done Done Sent false
31896 8939 Acked Read 6 Done Done Sent false
31897 8940 Acked Read 5 Done Done Sent false
31898 8941 Acked Write 7 Done Done Sent false
31899 8942 Acked Write 1 Done Done Sent false
31900 8943 Acked Read 8 Done Done Sent false
31901 8944 Acked Read 8 Done Done Sent false
31902 8945 Acked Write 9 Done Done Sent false
31903 8946 Acked Flush 0 Done Done Sent false
31904 8947 Acked Write 9 Done Done Sent false
31905 8948 Acked Read 1 Done Done Sent false
31906 8949 Acked Write 4 Done Done Sent false
31907 8950 Acked Read 7 Done Done Sent false
31908 8951 Acked Write 8 Done Done Sent false
31909 8952 Acked Write 4 Done Done Sent false
31910 8953 Acked Read 10 Done Done Sent false
31911 8954 Acked Read 3 Done Done Sent false
31912 8955 Acked Read 4 Done Done Sent false
31913 8956 Acked Write 2 Done Done Sent false
31914 8957 Acked Flush 0 Done Done Sent false
31915 8958 Acked Read 1 Done Done Sent false
31916 8959 Acked Flush 0 Done Done Sent false
31917 8960 Acked Read 1 Done Done Sent false
31918 8961 Acked Read 3 Done Done Sent false
31919 8962 Acked Read 8 Done Done Sent false
31920 8963 Acked Read 9 Done Done Sent false
31921 8964 Acked Flush 0 Done Done Sent false
31922 8965 Acked Write 5 Done Done Sent false
31923 8966 Acked Write 7 Done Done Sent false
31924 8967 Acked Flush 0 Done Done Sent false
31925 8968 Acked Write 5 Done Done Sent false
31926 8969 Acked Write 6 Done Done Sent false
31927 8970 Acked Read 8 Done Done Sent false
31928 8971 Acked Flush 0 Done Done Sent false
31929 8972 Acked Read 3 Done Done Sent false
31930 8973 Acked Write 10 Done Done Sent false
31931 8974 Acked Read 9 Done Done Sent false
31932 8975 Acked Read 8 Done Done Sent false
31933 8976 Acked Read 8 Done Done Sent false
31934 8977 Acked Flush 0 Done Done Sent false
31935 8978 Acked Write 8 Done Done Sent false
31936 8979 Acked Read 2 Done Done Sent false
31937 8980 Acked Flush 0 Done Done Sent false
31938 8981 Acked Read 9 Done Done Sent false
31939 8982 Acked Read 3 Done Done Sent false
31940 8983 Acked Write 2 Done Done Sent false
31941 8984 Acked Read 5 Done Done Sent false
31942 8985 Acked Flush 0 Done Done Sent false
31943 8986 Acked Read 4 Done Done Sent false
31944 8987 Acked Read 1 Done Done Sent false
31945 8988 Acked Read 6 Done Done Sent false
31946 8989 Acked Write 1 Done Done Sent false
31947 8990 Acked Write 1 Done Done Sent false
31948 8991 Acked Write 8 Done Done Sent false
31949 8992 Acked Read 8 Done Done Sent false
31950 8993 Acked Read 2 Done Done Sent false
31951 8994 Acked Read 2 Done Done Sent false
31952 8995 Acked Write 10 Done Done Sent false
31953 8996 Acked Flush 0 Done Done Sent false
31954 8997 Acked Read 8 Done Done Sent false
31955 8998 Acked Flush 0 Done Done Sent false
31956 8999 Acked Write 4 Done Done Sent false
31957 9000 Acked Read 10 Done Done Sent false
31958 9001 Acked Read 6 Done Done Sent false
31959 9002 Acked Read 8 Done Done Sent false
31960 9003 Acked Flush 0 Done Done Sent false
31961 9004 Acked Read 7 Done Done Sent false
31962 9005 Acked Write 9 Done Done Sent false
31963 9006 Acked Read 10 Done Done Sent false
31964 9007 Acked Read 7 Done Done Sent false
31965 9008 Acked Write 3 Done Done Sent false
31966 9009 Acked Read 3 Done Done Sent false
31967 9010 Acked Write 10 Done Done Sent false
31968 9011 Acked Read 10 Done Done Sent false
31969 9012 Acked Write 1 Done Done Sent false
31970 9013 Acked Read 10 Done Done Sent false
31971 9014 Acked Read 3 Done Done Sent false
31972 9015 Acked Write 5 Done Done Sent false
31973 9016 Acked Read 9 Done Done Sent false
31974 9017 Acked Write 1 Done Done Sent false
31975 9018 Acked Write 7 Done Done Sent false
31976 9019 Acked Read 4 Done Done Sent false
31977 9020 Acked Write 2 Done Done Sent false
31978 9021 Acked Read 4 Done Done Sent false
31979 9022 Acked Write 9 Done Done Sent false
31980 9023 Acked Write 6 Done Done Sent false
31981 9024 Acked Read 7 Done Done Sent false
31982 9025 Acked Write 8 Done Done Sent false
31983 9026 Acked Write 4 Done Done Sent false
31984 9027 Acked Read 7 Done Done Sent false
31985 9028 Acked Read 1 Done Done Sent false
31986 9029 Acked Write 10 Done Done Sent false
31987 9030 Acked Write 9 Done Done Sent false
31988 9031 Acked Read 8 Done Done Sent false
31989 9032 Acked Write 1 Done Done Sent false
31990 9033 Acked Write 8 Done Done Sent false
31991 9034 Acked Read 9 Done Done Sent false
31992 9035 Acked Read 3 Done Done Sent false
31993 9036 Acked Read 7 Done Done Sent false
31994 9037 Acked Read 10 Done Done Sent false
31995 9038 Acked Read 5 Done Done Sent false
31996 9039 Acked Read 5 Done Done Sent false
31997 9040 Acked Read 7 Done Done Sent false
31998 9041 Acked Flush 0 Done Done Sent false
31999 9042 Acked Read 4 Done Done Sent false
32000 9043 Acked Read 9 Done Done Sent false
32001 9044 Acked Write 6 Done Done Sent false
32002 9045 Acked Read 10 Done Done Sent false
32003 9046 Acked Write 4 Done Done Sent false
32004 9047 Acked Read 8 Done Done Sent false
32005 9048 Acked Read 1 Done Done Sent false
32006 9049 Acked Read 1 Done Done Sent false
32007 9050 Acked Flush 0 Done Done Sent false
32008 9051 Acked Read 5 Done Done Sent false
32009 9052 Acked Read 4 Done Done Sent false
32010 9053 Acked Write 4 Done Done Sent false
32011 9054 Acked Write 3 Done Done Sent false
32012 9055 Acked Read 5 Done Done Sent false
32013 9056 Acked Read 9 Done Done Sent false
32014 9057 Acked Write 1 Done Done Sent false
32015 9058 Acked Read 1 Done Done Sent false
32016 9059 Acked Read 1 Done Done Sent false
32017 9060 Acked Read 4 Done Done Sent false
32018 9061 Acked Flush 0 Done Done Sent false
32019 9062 Acked Write 9 Done Done Sent false
32020 9063 Acked Flush 0 Done Done Sent false
32021 9064 Acked Read 8 Done Done Sent false
32022 9065 Acked Write 3 Done Done Sent false
32023 9066 Acked Read 1 Done Done Sent false
32024 9067 Acked Read 7 Done Done Sent false
32025 9068 Acked Write 3 Done Done Sent false
32026 9069 Acked Write 9 Done Done Sent false
32027 9070 Acked Write 8 Done Done Sent false
32028 9071 Acked Write 9 Done Done Sent false
32029 9072 Acked Read 4 Done Done Sent false
32030 9073 Acked Read 9 Done Done Sent false
32031 9074 Acked Read 8 Done Done Sent false
32032 9075 Acked Read 6 Done Done Sent false
32033 9076 Acked Write 4 Done Done Sent false
32034 9077 Acked Read 10 Done Done Sent false
32035 9078 Acked Write 5 Done Done Sent false
32036 9079 Acked Read 5 Done Done Sent false
32037 9080 Acked Write 5 Done Done Sent false
32038 9081 Acked Flush 0 Done Done Sent false
32039 9082 Acked Read 6 Done Done Sent false
32040 9083 Acked Write 7 Done Done Sent false
32041 9084 Acked Flush 0 Done Done Sent false
32042 9085 Acked Write 7 Done Done Sent false
32043 9086 Acked Write 8 Done Done Sent false
32044 9087 Acked Read 5 Done Done Sent false
32045 9088 Acked Write 7 Done Done Sent false
32046 9089 Acked Read 5 Done Done Sent false
32047 9090 Acked Read 3 Done Done Sent false
32048 9091 Acked Read 3 Done Done Sent false
32049 9092 Acked Read 4 Done Done Sent false
32050 9093 Acked Write 5 Done Done Sent false
32051 9094 Acked Read 7 Done Done Sent false
32052 9095 Acked Write 6 Done Done Sent false
32053 9096 Acked Write 8 Done Done Sent false
32054 9097 Acked Read 3 Done Done Sent false
32055 9098 Acked Write 5 Done Done Sent false
32056 9099 Acked Write 6 Done Done Sent false
32057 9100 Acked Write 5 Done Done Sent false
32058 9101 Acked Read 10 Done Done Sent false
32059 9102 Acked Write 6 Done Done Sent false
32060 9103 Acked Write 4 Done Done Sent false
32061 9104 Acked Read 5 Done Done Sent false
32062 9105 Acked Read 9 Done Done Sent false
32063 9106 Acked Read 10 Done Done Sent false
32064 9107 Acked Read 4 Done Done Sent false
32065 9108 Acked Flush 0 Done Done Sent false
32066 9109 Acked Write 2 Done Done Sent false
32067 9110 Acked Write 3 Done Done Sent false
32068 9111 Acked Read 4 Done Done Sent false
32069 9112 Acked Read 6 Done Done Sent false
32070 9113 Acked Read 6 Done Done Sent false
32071 9114 Acked Read 10 Done Done Sent false
32072 9115 Acked Write 9 Done Done Sent false
32073 9116 Acked Read 3 Done Done Sent false
32074 9117 Acked Write 9 Done Done Sent false
32075 9118 Acked Read 8 Done Done Sent false
32076 9119 Acked Write 6 Done Done Sent false
32077 9120 Acked Read 6 Done Done Sent false
32078 9121 Acked Read 6 Done Done Sent false
32079 9122 Acked Write 4 Done Done Sent false
32080 9123 Acked Read 9 Done Done Sent false
32081 9124 Acked Flush 0 Done Done Sent false
32082 9125 Acked Read 10 Done Done Sent false
32083 9126 Acked Read 8 Done Done Sent false
32084 9127 Acked Read 1 Done Done Sent false
32085 9128 Acked Write 7 Done Done Sent false
32086 9129 Acked Flush 0 Done Done Sent false
32087 9130 Acked Read 10 Done Done Sent false
32088 9131 Acked Write 10 Done Done Sent false
32089 9132 Acked Read 2 Done Done Sent false
32090 9133 Acked Read 5 Done Done Sent false
32091 9134 Acked Write 8 Done Done Sent false
32092 9135 Acked Write 9 Done Done Sent false
32093 9136 Acked Read 6 Done Done Sent false
32094 9137 Acked Read 8 Done Done Sent false
32095 9138 Acked Read 7 Done Done Sent false
32096 9139 Acked Write 7 Done Done Sent false
32097 9140 Acked Read 5 Done Done Sent false
32098 9141 Acked Read 3 Done Done Sent false
32099 9142 Acked Read 8 Done Done Sent false
32100 9143 Acked Read 5 Done Done Sent false
32101 9144 Acked Write 8 Done Done Sent false
32102 9145 Acked Read 1 Done Done Sent false
32103 9146 Acked Read 8 Done Done Sent false
32104 9147 Acked Read 6 Done Done Sent false
32105 9148 Acked Read 3 Done Done Sent false
32106 9149 Acked Read 2 Done Done Sent false
32107 STATES DS:0 DS:1 DS:2 TOTAL
32108 Sent 0 0 301 301
32109 Done 301 301 0 602
32110 Skipped 0 0 0 0
32111 Error 0 0 0 0
32112 Last Flush: Some(JobId(9129)) Some(JobId(9129)) Some(JobId(8848))
32113 Downstairs last five completed: 8848 8847 8846 8845 8844
32114 Upstairs last five completed: 9149 9148 9147 9146 9145
32115 CLIENT: Up:0 ds:301 act:2
321162025-03-17T22:41:36.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
321172025-03-17T22:41:37.969ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
321182025-03-17T22:41:38.970ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
32119 ----------------------------------------------------------------
32120 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32121 States: Active Active Offline
32122 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32123 8849 Acked Flush 0 Done Done Sent false
32124 8850 Acked Read 9 Done Done Sent false
32125 8851 Acked Read 5 Done Done Sent false
32126 8852 Acked Read 2 Done Done Sent false
32127 8853 Acked Flush 0 Done Done Sent false
32128 8854 Acked Write 3 Done Done Sent false
32129 8855 Acked Read 3 Done Done Sent false
32130 8856 Acked Write 2 Done Done Sent false
32131 8857 Acked Read 2 Done Done Sent false
32132 8858 Acked Read 10 Done Done Sent false
32133 8859 Acked Write 2 Done Done Sent false
32134 8860 Acked Write 2 Done Done Sent false
32135 8861 Acked Read 10 Done Done Sent false
32136 8862 Acked Read 5 Done Done Sent false
32137 8863 Acked Write 7 Done Done Sent false
32138 8864 Acked Write 7 Done Done Sent false
32139 8865 Acked Read 10 Done Done Sent false
32140 8866 Acked Write 9 Done Done Sent false
32141 8867 Acked Write 8 Done Done Sent false
32142 8868 Acked Write 1 Done Done Sent false
32143 8869 Acked Flush 0 Done Done Sent false
32144 8870 Acked Read 2 Done Done Sent false
32145 8871 Acked Write 6 Done Done Sent false
32146 8872 Acked Write 9 Done Done Sent false
32147 8873 Acked Write 4 Done Done Sent false
32148 8874 Acked Read 5 Done Done Sent false
32149 8875 Acked Write 7 Done Done Sent false
32150 8876 Acked Read 8 Done Done Sent false
32151 8877 Acked Read 2 Done Done Sent false
32152 8878 Acked Write 4 Done Done Sent false
32153 8879 Acked Read 8 Done Done Sent false
32154 8880 Acked Read 4 Done Done Sent false
32155 8881 Acked Read 7 Done Done Sent false
32156 8882 Acked Read 10 Done Done Sent false
32157 8883 Acked Read 4 Done Done Sent false
32158 8884 Acked Read 6 Done Done Sent false
32159 8885 Acked Read 3 Done Done Sent false
32160 8886 Acked Read 10 Done Done Sent false
32161 8887 Acked Read 2 Done Done Sent false
32162 8888 Acked Read 9 Done Done Sent false
32163 8889 Acked Write 1 Done Done Sent false
32164 8890 Acked Flush 0 Done Done Sent false
32165 8891 Acked Read 4 Done Done Sent false
32166 8892 Acked Read 7 Done Done Sent false
32167 8893 Acked Write 6 Done Done Sent false
32168 8894 Acked Read 10 Done Done Sent false
32169 8895 Acked Flush 0 Done Done Sent false
32170 8896 Acked Read 10 Done Done Sent false
32171 8897 Acked Read 1 Done Done Sent false
32172 8898 Acked Read 8 Done Done Sent false
32173 8899 Acked Write 7 Done Done Sent false
32174 8900 Acked Read 9 Done Done Sent false
32175 8901 Acked Write 4 Done Done Sent false
32176 8902 Acked Read 5 Done Done Sent false
32177 8903 Acked Write 5 Done Done Sent false
32178 8904 Acked Flush 0 Done Done Sent false
32179 8905 Acked Read 2 Done Done Sent false
32180 8906 Acked Write 1 Done Done Sent false
32181 8907 Acked Read 6 Done Done Sent false
32182 8908 Acked Flush 0 Done Done Sent false
32183 8909 Acked Read 6 Done Done Sent false
32184 8910 Acked Read 5 Done Done Sent false
32185 8911 Acked Read 7 Done Done Sent false
32186 8912 Acked Flush 0 Done Done Sent false
32187 8913 Acked Flush 0 Done Done Sent false
32188 8914 Acked Flush 0 Done Done Sent false
32189 8915 Acked Read 1 Done Done Sent false
32190 8916 Acked Read 9 Done Done Sent false
32191 8917 Acked Write 2 Done Done Sent false
32192 8918 Acked Flush 0 Done Done Sent false
32193 8919 Acked Flush 0 Done Done Sent false
32194 8920 Acked Read 4 Done Done Sent false
32195 8921 Acked Write 8 Done Done Sent false
32196 8922 Acked Read 2 Done Done Sent false
32197 8923 Acked Read 10 Done Done Sent false
32198 8924 Acked Flush 0 Done Done Sent false
32199 8925 Acked Write 10 Done Done Sent false
32200 8926 Acked Write 5 Done Done Sent false
32201 8927 Acked Write 9 Done Done Sent false
32202 8928 Acked Read 2 Done Done Sent false
32203 8929 Acked Read 4 Done Done Sent false
32204 8930 Acked Flush 0 Done Done Sent false
32205 8931 Acked Read 5 Done Done Sent false
32206 8932 Acked Flush 0 Done Done Sent false
32207 8933 Acked Read 2 Done Done Sent false
32208 8934 Acked Write 9 Done Done Sent false
32209 8935 Acked Read 7 Done Done Sent false
32210 8936 Acked Write 8 Done Done Sent false
32211 8937 Acked Read 10 Done Done Sent false
32212 8938 Acked Read 1 Done Done Sent false
32213 8939 Acked Read 6 Done Done Sent false
32214 8940 Acked Read 5 Done Done Sent false
32215 8941 Acked Write 7 Done Done Sent false
32216 8942 Acked Write 1 Done Done Sent false
32217 8943 Acked Read 8 Done Done Sent false
32218 8944 Acked Read 8 Done Done Sent false
32219 8945 Acked Write 9 Done Done Sent false
32220 8946 Acked Flush 0 Done Done Sent false
32221 8947 Acked Write 9 Done Done Sent false
32222 8948 Acked Read 1 Done Done Sent false
32223 8949 Acked Write 4 Done Done Sent false
32224 8950 Acked Read 7 Done Done Sent false
32225 8951 Acked Write 8 Done Done Sent false
32226 8952 Acked Write 4 Done Done Sent false
32227 8953 Acked Read 10 Done Done Sent false
32228 8954 Acked Read 3 Done Done Sent false
32229 8955 Acked Read 4 Done Done Sent false
32230 8956 Acked Write 2 Done Done Sent false
32231 8957 Acked Flush 0 Done Done Sent false
32232 8958 Acked Read 1 Done Done Sent false
32233 8959 Acked Flush 0 Done Done Sent false
32234 8960 Acked Read 1 Done Done Sent false
32235 8961 Acked Read 3 Done Done Sent false
32236 8962 Acked Read 8 Done Done Sent false
32237 8963 Acked Read 9 Done Done Sent false
32238 8964 Acked Flush 0 Done Done Sent false
32239 8965 Acked Write 5 Done Done Sent false
32240 8966 Acked Write 7 Done Done Sent false
32241 8967 Acked Flush 0 Done Done Sent false
32242 8968 Acked Write 5 Done Done Sent false
32243 8969 Acked Write 6 Done Done Sent false
32244 8970 Acked Read 8 Done Done Sent false
32245 8971 Acked Flush 0 Done Done Sent false
32246 8972 Acked Read 3 Done Done Sent false
32247 8973 Acked Write 10 Done Done Sent false
32248 8974 Acked Read 9 Done Done Sent false
32249 8975 Acked Read 8 Done Done Sent false
32250 8976 Acked Read 8 Done Done Sent false
32251 8977 Acked Flush 0 Done Done Sent false
32252 8978 Acked Write 8 Done Done Sent false
32253 8979 Acked Read 2 Done Done Sent false
32254 8980 Acked Flush 0 Done Done Sent false
32255 8981 Acked Read 9 Done Done Sent false
32256 8982 Acked Read 3 Done Done Sent false
32257 8983 Acked Write 2 Done Done Sent false
32258 8984 Acked Read 5 Done Done Sent false
32259 8985 Acked Flush 0 Done Done Sent false
32260 8986 Acked Read 4 Done Done Sent false
32261 8987 Acked Read 1 Done Done Sent false
32262 8988 Acked Read 6 Done Done Sent false
32263 8989 Acked Write 1 Done Done Sent false
32264 8990 Acked Write 1 Done Done Sent false
32265 8991 Acked Write 8 Done Done Sent false
32266 8992 Acked Read 8 Done Done Sent false
32267 8993 Acked Read 2 Done Done Sent false
32268 8994 Acked Read 2 Done Done Sent false
32269 8995 Acked Write 10 Done Done Sent false
32270 8996 Acked Flush 0 Done Done Sent false
32271 8997 Acked Read 8 Done Done Sent false
32272 8998 Acked Flush 0 Done Done Sent false
32273 8999 Acked Write 4 Done Done Sent false
32274 9000 Acked Read 10 Done Done Sent false
32275 9001 Acked Read 6 Done Done Sent false
32276 9002 Acked Read 8 Done Done Sent false
32277 9003 Acked Flush 0 Done Done Sent false
32278 9004 Acked Read 7 Done Done Sent false
32279 9005 Acked Write 9 Done Done Sent false
32280 9006 Acked Read 10 Done Done Sent false
32281 9007 Acked Read 7 Done Done Sent false
32282 9008 Acked Write 3 Done Done Sent false
32283 9009 Acked Read 3 Done Done Sent false
32284 9010 Acked Write 10 Done Done Sent false
32285 9011 Acked Read 10 Done Done Sent false
32286 9012 Acked Write 1 Done Done Sent false
32287 9013 Acked Read 10 Done Done Sent false
32288 9014 Acked Read 3 Done Done Sent false
32289 9015 Acked Write 5 Done Done Sent false
32290 9016 Acked Read 9 Done Done Sent false
32291 9017 Acked Write 1 Done Done Sent false
32292 9018 Acked Write 7 Done Done Sent false
32293 9019 Acked Read 4 Done Done Sent false
32294 9020 Acked Write 2 Done Done Sent false
32295 9021 Acked Read 4 Done Done Sent false
32296 9022 Acked Write 9 Done Done Sent false
32297 9023 Acked Write 6 Done Done Sent false
32298 9024 Acked Read 7 Done Done Sent false
32299 9025 Acked Write 8 Done Done Sent false
32300 9026 Acked Write 4 Done Done Sent false
32301 9027 Acked Read 7 Done Done Sent false
32302 9028 Acked Read 1 Done Done Sent false
32303 9029 Acked Write 10 Done Done Sent false
32304 9030 Acked Write 9 Done Done Sent false
32305 9031 Acked Read 8 Done Done Sent false
32306 9032 Acked Write 1 Done Done Sent false
32307 9033 Acked Write 8 Done Done Sent false
32308 9034 Acked Read 9 Done Done Sent false
32309 9035 Acked Read 3 Done Done Sent false
32310 9036 Acked Read 7 Done Done Sent false
32311 9037 Acked Read 10 Done Done Sent false
32312 9038 Acked Read 5 Done Done Sent false
32313 9039 Acked Read 5 Done Done Sent false
32314 9040 Acked Read 7 Done Done Sent false
32315 9041 Acked Flush 0 Done Done Sent false
32316 9042 Acked Read 4 Done Done Sent false
32317 9043 Acked Read 9 Done Done Sent false
32318 9044 Acked Write 6 Done Done Sent false
32319 9045 Acked Read 10 Done Done Sent false
32320 9046 Acked Write 4 Done Done Sent false
32321 9047 Acked Read 8 Done Done Sent false
32322 9048 Acked Read 1 Done Done Sent false
32323 9049 Acked Read 1 Done Done Sent false
32324 9050 Acked Flush 0 Done Done Sent false
32325 9051 Acked Read 5 Done Done Sent false
32326 9052 Acked Read 4 Done Done Sent false
32327 9053 Acked Write 4 Done Done Sent false
32328 9054 Acked Write 3 Done Done Sent false
32329 9055 Acked Read 5 Done Done Sent false
32330 9056 Acked Read 9 Done Done Sent false
32331 9057 Acked Write 1 Done Done Sent false
32332 9058 Acked Read 1 Done Done Sent false
32333 9059 Acked Read 1 Done Done Sent false
32334 9060 Acked Read 4 Done Done Sent false
32335 9061 Acked Flush 0 Done Done Sent false
32336 9062 Acked Write 9 Done Done Sent false
32337 9063 Acked Flush 0 Done Done Sent false
32338 9064 Acked Read 8 Done Done Sent false
32339 9065 Acked Write 3 Done Done Sent false
32340 9066 Acked Read 1 Done Done Sent false
32341 9067 Acked Read 7 Done Done Sent false
32342 9068 Acked Write 3 Done Done Sent false
32343 9069 Acked Write 9 Done Done Sent false
32344 9070 Acked Write 8 Done Done Sent false
32345 9071 Acked Write 9 Done Done Sent false
32346 9072 Acked Read 4 Done Done Sent false
32347 9073 Acked Read 9 Done Done Sent false
32348 9074 Acked Read 8 Done Done Sent false
32349 9075 Acked Read 6 Done Done Sent false
32350 9076 Acked Write 4 Done Done Sent false
32351 9077 Acked Read 10 Done Done Sent false
32352 9078 Acked Write 5 Done Done Sent false
32353 9079 Acked Read 5 Done Done Sent false
32354 9080 Acked Write 5 Done Done Sent false
32355 9081 Acked Flush 0 Done Done Sent false
32356 9082 Acked Read 6 Done Done Sent false
32357 9083 Acked Write 7 Done Done Sent false
32358 9084 Acked Flush 0 Done Done Sent false
32359 9085 Acked Write 7 Done Done Sent false
32360 9086 Acked Write 8 Done Done Sent false
32361 9087 Acked Read 5 Done Done Sent false
32362 9088 Acked Write 7 Done Done Sent false
32363 9089 Acked Read 5 Done Done Sent false
32364 9090 Acked Read 3 Done Done Sent false
32365 9091 Acked Read 3 Done Done Sent false
32366 9092 Acked Read 4 Done Done Sent false
32367 9093 Acked Write 5 Done Done Sent false
32368 9094 Acked Read 7 Done Done Sent false
32369 9095 Acked Write 6 Done Done Sent false
32370 9096 Acked Write 8 Done Done Sent false
32371 9097 Acked Read 3 Done Done Sent false
32372 9098 Acked Write 5 Done Done Sent false
32373 9099 Acked Write 6 Done Done Sent false
32374 9100 Acked Write 5 Done Done Sent false
32375 9101 Acked Read 10 Done Done Sent false
32376 9102 Acked Write 6 Done Done Sent false
32377 9103 Acked Write 4 Done Done Sent false
32378 9104 Acked Read 5 Done Done Sent false
32379 9105 Acked Read 9 Done Done Sent false
32380 9106 Acked Read 10 Done Done Sent false
32381 9107 Acked Read 4 Done Done Sent false
32382 9108 Acked Flush 0 Done Done Sent false
32383 9109 Acked Write 2 Done Done Sent false
32384 9110 Acked Write 3 Done Done Sent false
32385 9111 Acked Read 4 Done Done Sent false
32386 9112 Acked Read 6 Done Done Sent false
32387 9113 Acked Read 6 Done Done Sent false
32388 9114 Acked Read 10 Done Done Sent false
32389 9115 Acked Write 9 Done Done Sent false
32390 9116 Acked Read 3 Done Done Sent false
32391 9117 Acked Write 9 Done Done Sent false
32392 9118 Acked Read 8 Done Done Sent false
32393 9119 Acked Write 6 Done Done Sent false
32394 9120 Acked Read 6 Done Done Sent false
32395 9121 Acked Read 6 Done Done Sent false
32396 9122 Acked Write 4 Done Done Sent false
32397 9123 Acked Read 9 Done Done Sent false
32398 9124 Acked Flush 0 Done Done Sent false
32399 9125 Acked Read 10 Done Done Sent false
32400 9126 Acked Read 8 Done Done Sent false
32401 9127 Acked Read 1 Done Done Sent false
32402 9128 Acked Write 7 Done Done Sent false
32403 9129 Acked Flush 0 Done Done Sent false
32404 9130 Acked Read 10 Done Done Sent false
32405 9131 Acked Write 10 Done Done Sent false
32406 9132 Acked Read 2 Done Done Sent false
32407 9133 Acked Read 5 Done Done Sent false
32408 9134 Acked Write 8 Done Done Sent false
32409 9135 Acked Write 9 Done Done Sent false
32410 9136 Acked Read 6 Done Done Sent false
32411 9137 Acked Read 8 Done Done Sent false
32412 9138 Acked Read 7 Done Done Sent false
32413 9139 Acked Write 7 Done Done Sent false
32414 9140 Acked Read 5 Done Done Sent false
32415 9141 Acked Read 3 Done Done Sent false
32416 9142 Acked Read 8 Done Done Sent false
32417 9143 Acked Read 5 Done Done Sent false
32418 9144 Acked Write 8 Done Done Sent false
32419 9145 Acked Read 1 Done Done Sent false
32420 9146 Acked Read 8 Done Done Sent false
32421 9147 Acked Read 6 Done Done Sent false
32422 9148 Acked Read 3 Done Done Sent false
32423 9149 Acked Read 2 Done Done Sent false
32424 STATES DS:0 DS:1 DS:2 TOTAL
32425 Sent 0 0 301 301
32426 Done 301 301 0 602
32427 Skipped 0 0 0 0
32428 Error 0 0 0 0
32429 Last Flush: Some(JobId(9129)) Some(JobId(9129)) Some(JobId(8848))
32430 Downstairs last five completed: 8848 8847 8846 8845 8844
32431 Upstairs last five completed: 9149 9148 9147 9146 9145
32432 CLIENT: Up:0 ds:301 act:2
324332025-03-17T22:41:40.211ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324342025-03-17T22:41:41.974ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324352025-03-17T22:41:42.976ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324362025-03-17T22:41:43.443ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324372025-03-17T22:41:43.443ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37665) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324382025-03-17T22:41:43.444ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324392025-03-17T22:41:43.444ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324402025-03-17T22:41:43.444ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324412025-03-17T22:41:43.445ZINFOcrucible: send last flush ID to this DS: Some(JobId(8848)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324422025-03-17T22:41:43.445ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324432025-03-17T22:41:43.445ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8848)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324442025-03-17T22:41:43.445ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8848)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
32445 ----------------------------------------------------------------
32446 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:20
32447 States: Active Active Active
32448 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32449 9130 Acked Read 10 Done Done Done true
32450 9131 Acked Write 10 Done Done Done true
32451 9132 Acked Read 2 Done Done Done true
32452 9133 Acked Read 5 Done Done Done true
32453 9134 Acked Write 8 Done Done Done true
32454 9135 Acked Write 9 Done Done Done true
32455 9136 Acked Read 6 Done Done Done true
32456 9137 Acked Read 8 Done Done Done true
32457 9138 Acked Read 7 Done Done Done true
32458 9139 Acked Write 7 Done Done Done true
32459 9140 Acked Read 5 Done Done Done true
32460 9141 Acked Read 3 Done Done Done true
32461 9142 Acked Read 8 Done Done Done true
32462 9143 Acked Read 5 Done Done Done true
32463 9144 Acked Write 8 Done Done Done true
32464 9145 Acked Read 1 Done Done Done true
32465 9146 Acked Read 8 Done Done Done true
32466 9147 Acked Read 6 Done Done Done true
32467 9148 Acked Read 3 Done Done Done true
32468 9149 Acked Read 2 Done Done Done true
32469 STATES DS:0 DS:1 DS:2 TOTAL
32470 Sent 0 0 0 0
32471 Done 20 20 20 60
32472 Skipped 0 0 0 0
32473 Error 0 0 0 0
32474 Last Flush: Some(JobId(9129)) Some(JobId(9129)) Some(JobId(9129))
32475 Downstairs last five completed: 9129 9128 9127 9126 9125
32476 Upstairs last five completed: 9149 9148 9147 9146 9145
32477 CLIENT: Up:0 ds:20 act:3
32478 ----------------------------------------------------------------
32479 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32480 Downstairs last five completed: 9150 9149 9148 9147 9146
32481 Upstairs last five completed: 9150 9149 9148 9147 9146
32482 CLIENT: Up:0 ds:0 act:3
32483 Replay: All jobs finished
32484 Main task has work to do, go find it
32485 got dsc Stop(1)
32486 stop 1
32487 Mar 17 22:41:47.714[8820] Got stop action so:false kr:false
32488 INFO request completed, latency_us: 399, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e975810c-10bb-48b8-b465-fd1be4b8f1d6, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
324892025-03-17T22:41:47.715ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324902025-03-17T22:41:47.715ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324912025-03-17T22:41:47.715ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
324922025-03-17T22:41:47.715ZINFOcrucible: client task is exiting = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
32493 Mar 17 22:41:47.715 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/1, method: GET, req_id: c4c90d63-a072-4dbe-bb2e-8033fbe8c30e, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
32494 [8820] Exited with: ExitStatus(unix_wait_status(9))
32495 [8820][1] reports Exit
32496 Mar 17 22:41:51.718 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 37ef2d8b-aa76-4a5e-8316-0f32a50e139a, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
32497 001/300 Write block 4821 len 28672 data: 5 6 4 4 5 4 5
32498 002/300 Read block 1012 len 24576
32499 003/300 Read block 3606 len 12288
32500 004/300 Read block 4883 len 20480
32501 005/300 Read block 1957 len 32768
32502 006/300 Read block 2371 len 12288
32503 007/300 Read block 4901 len 40960
32504 008/300 Write block 3260 len 32768 data: 4 5 6 6 6 5 6 6
32505 009/300 Read block 331 len 28672
32506 010/300 Flush
32507 011/300 Write block 3385 len 24576 data: 8 8 6 9 6 6
32508 012/300 Read block 2660 len 20480
32509 013/300 Write block 953 len 12288 data: 5 4 4
32510 014/300 Write block 2648 len 12288 data: 3 3 3
32511 015/300 Read block 4974 len 40960
32512 016/300 Read block 3239 len 24576
32513 017/300 Read block 3501 len 32768
32514 018/300 Read block 481 len 32768
32515 019/300 Read block 2463 len 8192
32516 020/300 Read block 3901 len 16384
32517 021/300 Read block 1366 len 12288
32518 022/300 Write block 2884 len 8192 data: 7 7
32519 023/300 Read block 2469 len 40960
32520 024/300 Write block 1269 len 32768 data: 5 5 6 5 6 6 6 5
32521 025/300 Write block 3452 len 12288 data: 6 7 5
32522 026/300 Write block 1806 len 20480 data: 6 7 7 7 5
32523 027/300 Write block 2687 len 36864 data: 8 8 7 5 6 5 4 5 6
32524 028/300 Read block 6 len 40960
32525 029/300 Write block 2126 len 8192 data: 4 5
32526 030/300 Write block 1549 len 8192 data: 8 7
32527 031/300 Write block 1322 len 12288 data: 6 6 6
32528 032/300 Read block 508 len 32768
32529 033/300 Read block 3705 len 16384
32530 034/300 Write block 3855 len 24576 data: 7 8 6 5 4 5
32531 035/300 Write block 1973 len 20480 data: 2 2 2 2 2
32532 036/300 Read block 4914 len 20480
32533 037/300 Read block 2562 len 4096
32534 038/300 Write block 279 len 12288 data: 7 5 6
32535 039/300 Read block 636 len 24576
32536 040/300 Read block 1169 len 28672
32537 041/300 Write block 4026 len 40960 data: 8 7 6 4 3 3 4 4 5 6
32538 042/300 Flush
32539 043/300 Write block 1578 len 20480 data: 4 4 3 5 4
32540 044/300 Write block 1320 len 28672 data: 7 6 7 7 7 8 9
32541 045/300 Read block 1706 len 32768
32542 046/300 Read block 1296 len 32768
32543 047/300 Write block 2875 len 36864 data: 4 6 7 6 9 7 8 8 6
32544 048/300 Write block 3500 len 20480 data: 2 2 4 4 5
32545 049/300 Write block 3580 len 20480 data: 4 4 4 3 3
32546 050/300 Read block 613 len 12288
32547 051/300 Read block 2018 len 12288
32548 052/300 Write block 4420 len 24576 data: 5 5 6 6 7 8
32549 053/300 Read block 250 len 8192
32550 054/300 Flush
32551 055/300 Read block 2686 len 28672
32552 056/300 Write block 221 len 40960 data: 4 4 5 4 4 4 4 4 5 8
32553 057/300 Write block 4211 len 36864 data: 4 3 3 4 4 7 7 7 6
32554 058/300 Read block 3927 len 16384
32555 059/300 Flush
32556 060/300 Write block 3433 len 24576 data: 9 10 8 8 6 5
32557 061/300 Read block 3040 len 28672
32558 062/300 Write block 3330 len 8192 data: 4 4
32559 063/300 Write block 4726 len 36864 data: 5 5 6 5 4 4 4 3 3
32560 064/300 Read block 167 len 12288
32561 065/300 Write block 4891 len 4096 data: 3
32562 066/300 Write block 4693 len 32768 data: 7 6 6 7 6 8 7 12
32563 067/300 Flush
32564 068/300 Read block 2798 len 16384
32565 069/300 Read block 4950 len 32768
32566 070/300 Read block 4810 len 32768
32567 071/300 Write block 4769 len 12288 data: 5 5 5
32568 072/300 Write block 1463 len 4096 data: 7
32569 073/300 Read block 2855 len 36864
32570 074/300 Read block 2452 len 40960
32571 075/300 Write block 3956 len 24576 data: 4 4 5 5 5 6
32572 076/300 Flush
32573 077/300 Flush
32574 078/300 Read block 3108 len 36864
32575 079/300 Read block 3412 len 8192
32576 080/300 Flush
32577 081/300 Read block 2859 len 8192
32578 082/300 Write block 1597 len 32768 data: 11 12 10 9 8 6 5 4
32579 083/300 Write block 762 len 36864 data: 5 5 6 6 8 8 7 7 6
32580 084/300 Write block 4641 len 8192 data: 7 8
32581 085/300 Write block 723 len 28672 data: 6 6 7 9 9 10 10
32582 086/300 Read block 2414 len 36864
32583 087/300 Write block 832 len 16384 data: 5 5 5 5
32584 088/300 Read block 1330 len 24576
32585 089/300 Read block 3535 len 12288
32586 090/300 Read block 4985 len 40960
32587 091/300 Read block 3776 len 8192
32588 092/300 Read block 563 len 40960
32589 093/300 Read block 3015 len 12288
32590 094/300 Read block 1359 len 40960
32591 095/300 Read block 4876 len 24576
32592 096/300 Write block 1386 len 32768 data: 4 4 5 5 4 5 6 5
32593 097/300 Write block 1001 len 40960 data: 6 6 6 5 4 4 6 5 5 3
32594 098/300 Write block 2068 len 4096 data: 6
32595 099/300 Read block 893 len 36864
32596 100/300 Flush
32597 101/300 Write block 1736 len 12288 data: 8 6 6
32598 102/300 Flush
32599 103/300 Write block 3719 len 12288 data: 9 8 9
32600 104/300 Write block 2251 len 40960 data: 5 4 4 5 6 5 5 6 6 8
32601 105/300 Read block 2327 len 32768
32602 106/300 Read block 1742 len 12288
32603 107/300 Read block 4012 len 8192
32604 108/300 Write block 3926 len 24576 data: 7 8 7 7 7 8
32605 109/300 Read block 2902 len 20480
32606 110/300 Write block 2433 len 8192 data: 5 4
32607 111/300 Write block 240 len 16384 data: 5 4 5 4
32608 112/300 Write block 1643 len 16384 data: 6 10 8 8
32609 113/300 Read block 2117 len 40960
32610 114/300 Read block 942 len 20480
32611 115/300 Read block 3380 len 20480
32612 116/300 Write block 290 len 28672 data: 6 7 7 9 8 8 8
32613 117/300 Write block 2147 len 40960 data: 8 8 7 4 5 6 5 5 5 6
32614 118/300 Write block 541 len 28672 data: 5 5 5 6 6 6 4
32615 119/300 Read block 580 len 12288
32616 120/300 Write block 3649 len 16384 data: 5 5 5 7
32617 121/300 Read block 2317 len 8192
32618 122/300 Flush
32619 123/300 Read block 2826 len 20480
32620 124/300 Read block 1392 len 36864
32621 125/300 Read block 3201 len 40960
32622 126/300 Read block 2782 len 16384
32623 127/300 Write block 3243 len 4096 data: 5
32624 128/300 Write block 924 len 28672 data: 3 5 6 6 8 7 6
32625 129/300 Write block 836 len 20480 data: 6 6 5 4 4
32626 130/300 Read block 795 len 28672
32627 131/300 Write block 4598 len 36864 data: 6 7 7 7 7 7 8 10 8
32628 132/300 Write block 1671 len 8192 data: 3 6
32629 133/300 Write block 4179 len 8192 data: 7 7
32630 134/300 Write block 2536 len 40960 data: 7 7 7 6 5 6 5 5 4 5
32631 135/300 Read block 3980 len 4096
32632 136/300 Write block 4646 len 36864 data: 8 7 7 6 5 5 5 6 6
32633 137/300 Read block 3961 len 36864
32634 138/300 Read block 3596 len 32768
32635 139/300 Read block 3597 len 24576
32636 140/300 Read block 4784 len 32768
32637 141/300 Read block 549 len 32768
32638 142/300 Read block 546 len 32768
32639 143/300 Read block 2823 len 40960
32640 144/300 Write block 3381 len 36864 data: 6 7 7 9 9 9 7 10 7
32641 145/300 Read block 3574 len 40960
32642 146/300 Read block 4891 len 28672
32643 147/300 Read block 2576 len 28672
32644 148/300 Read block 701 len 28672
32645 149/300 Write block 400 len 32768 data: 3 3 5 5 6 7 5 5
32646 150/300 Write block 4367 len 28672 data: 9 9 6 7 8 8 8
32647 151/300 Read block 2463 len 36864
32648 152/300 Flush
32649 153/300 Write block 1212 len 16384 data: 6 5 5 5
32650 154/300 Write block 725 len 16384 data: 8 10 10 11
32651 155/300 Read block 3721 len 20480
32652 156/300 Read block 1774 len 32768
32653 157/300 Write block 3024 len 12288 data: 7 7 5
32654 158/300 Read block 495 len 8192
32655 159/300 Read block 389 len 32768
32656 160/300 Read block 4486 len 20480
32657 161/300 Flush
32658 162/300 Flush
32659 163/300 Read block 2891 len 12288
32660 164/300 Read block 2734 len 28672
32661 165/300 Read block 662 len 28672
32662 166/300 Read block 787 len 8192
32663 167/300 Read block 871 len 36864
32664 168/300 Read block 2665 len 28672
32665 169/300 Read block 3361 len 20480
32666 170/300 Write block 3189 len 32768 data: 9 7 4 5 6 6 7 7
32667 171/300 Write block 1145 len 32768 data: 4 3 3 3 4 6 6 7
32668 172/300 Read block 2833 len 4096
32669 173/300 Flush
32670 174/300 Read block 1203 len 36864
32671 175/300 Write block 4225 len 36864 data: 6 5 7 7 6 8 9 9 9
32672 176/300 Read block 2675 len 8192
32673 177/300 Read block 4946 len 24576
32674 178/300 Write block 506 len 32768 data: 2 2 3 4 4 4 5 7
32675 179/300 Read block 1077 len 40960
32676 180/300 Flush
32677 181/300 Read block 3612 len 32768
32678 182/300 Write block 1000 len 28672 data: 7 7 7 7 6 5 5
32679 183/300 Write block 509 len 8192 data: 5 5
32680 184/300 Write block 1650 len 12288 data: 6 5 5
32681 185/300 Write block 2999 len 36864 data: 5 5 5 5 4 4 4 4 4
32682 186/300 Write block 2198 len 40960 data: 7 7 5 6 8 8 9 11 10 9
32683 187/300 Read block 3256 len 40960
32684 188/300 Flush
32685 189/300 Write block 1933 len 16384 data: 2 3 3 4
32686 190/300 Write block 2741 len 4096 data: 7
32687 191/300 Read block 1165 len 32768
32688 192/300 Read block 2433 len 12288
32689 193/300 Read block 922 len 20480
32690 194/300 Write block 3115 len 8192 data: 5 5
32691 195/300 Read block 852 len 4096
32692 196/300 Write block 3719 len 36864 data: 10 9 10 10 9 8 7 7 7
32693 197/300 Read block 1574 len 40960
32694 198/300 Read block 414 len 32768
32695 199/300 Write block 208 len 8192 data: 5 5
32696 200/300 Read block 3841 len 4096
32697 201/300 Write block 1506 len 8192 data: 5 7
32698 202/300 Write block 3735 len 24576 data: 7 7 7 6 6 4
32699 203/300 Flush
32700 204/300 Write block 4687 len 4096 data: 4
32701 205/300 Read block 4971 len 32768
32702 206/300 Read block 3212 len 12288
32703 207/300 Flush
32704 208/300 Write block 612 len 36864 data: 6 7 6 7 8 8 7 6 6
32705 209/300 Read block 1695 len 16384
32706 210/300 Write block 4219 len 28672 data: 7 6 6 9 7 6 7
32707 211/300 Write block 2983 len 36864 data: 7 7 8 11 8 9 8 10 9
32708 212/300 Read block 2265 len 24576
32709 213/300 Read block 486 len 36864
32710 214/300 Read block 1578 len 16384
32711 215/300 Write block 3298 len 16384 data: 5 5 5 6
32712 216/300 Read block 838 len 32768
32713 217/300 Write block 2296 len 40960 data: 7 5 6 6 7 5 6 4 3 2
32714 218/300 Write block 2294 len 8192 data: 5 6
32715 219/300 Read block 3733 len 36864
32716 220/300 Read block 142 len 24576
32717 221/300 Write block 2708 len 40960 data: 4 5 7 5 5 4 5 4 4 5
32718 222/300 Read block 250 len 24576
32719 223/300 Write block 59 len 16384 data: 3 5 5 5
32720 224/300 Read block 678 len 28672
32721 225/300 Read block 4079 len 20480
32722 226/300 Read block 167 len 4096
32723 227/300 Write block 4445 len 8192 data: 6 5
32724 228/300 Flush
32725 229/300 Write block 1131 len 28672 data: 3 3 4 4 3 5 6
32726 230/300 Flush
32727 231/300 Write block 783 len 12288 data: 8 8 8
32728 232/300 Write block 3658 len 24576 data: 5 4 4 6 7 5
32729 233/300 Read block 2932 len 16384
32730 234/300 Write block 3546 len 40960 data: 8 7 7 7 5 6 5 5 7 7
32731 235/300 Read block 2730 len 20480
32732 236/300 Write block 4947 len 40960 data: 5 6 4 5 5 6 7 6 5 5
32733 237/300 Read block 1067 len 4096
32734 238/300 Write block 2089 len 16384 data: 9 8 8 7
32735 239/300 Flush
32736 240/300 Write block 3336 len 20480 data: 6 6 7 8 9
32737 241/300 Write block 2248 len 28672 data: 4 4 4 6 5 5 6
32738 242/300 Read block 3817 len 16384
32739 243/300 Write block 1376 len 12288 data: 6 7 7
32740 244/300 Read block 3470 len 16384
32741 245/300 Read block 601 len 16384
32742 246/300 Write block 3250 len 36864 data: 5 6 6 5 4 4 5 5 4
32743 247/300 Write block 2865 len 40960 data: 7 7 6 5 6 6 4 6 5 4
32744 248/300 Read block 2450 len 36864
32745 249/300 Read block 4558 len 4096
32746 250/300 Read block 2341 len 36864
32747 251/300 Write block 4703 len 28672 data: 11 7 8 7 4 3 5
32748 252/300 Read block 4708 len 24576
32749 253/300 Read block 1911 len 40960
32750 254/300 Write block 3424 len 12288 data: 9 9 8
32751 255/300 Write block 4476 len 40960 data: 3 3 2 2 3 3 3 3 4 4
32752 256/300 Write block 4107 len 8192 data: 5 5
32753 257/300 Write block 2754 len 20480 data: 8 8 6 5 4
32754 258/300 Write block 2004 len 28672 data: 6 4 4 5 5 6 5
32755 259/300 Write block 3940 len 28672 data: 6 6 6 7 7 8 9
32756 260/300 Read block 1288 len 12288
32757 261/300 Read block 2483 len 40960
32758 262/300 Read block 1088 len 4096
32759 263/300 Read block 4348 len 36864
32760 264/300 Read block 1332 len 40960
32761 265/300 Write block 2002 len 32768 data: 8 8 7 5 5 6 6 7
32762 266/300 Read block 1893 len 32768
32763 267/300 Write block 1101 len 24576 data: 6 6 5 5 4 4
32764 268/300 Flush
32765 269/300 Read block 4820 len 36864
32766 270/300 Read block 4264 len 20480
32767 271/300 Read block 1370 len 20480
32768 272/300 Write block 1940 len 32768 data: 5 6 5 6 6 7 7 7
32769 273/300 Write block 2238 len 36864 data: 4 4 4 3 3 3 3 3 3
32770 274/300 Read block 903 len 28672
32771 275/300 Flush
32772 276/300 Read block 3425 len 20480
32773 277/300 Read block 332 len 20480
32774 278/300 Write block 2847 len 24576 data: 4 4 4 4 5 5
32775 279/300 Write block 2977 len 12288 data: 10 13 14
32776 280/300 Write block 643 len 24576 data: 7 7 8 8 6 6
32777 281/300 Read block 126 len 16384
32778 282/300 Read block 3261 len 32768
32779 283/300 Read block 4401 len 20480
32780 284/300 Read block 3217 len 12288
32781 285/300 Write block 1055 len 32768 data: 4 4 4 4 5 5 5 3
32782 286/300 Write block 742 len 20480 data: 4 4 5 7 7
32783 287/300 Read block 3457 len 28672
32784 288/300 Read block 564 len 20480
32785 289/300 Read block 1568 len 20480
32786 290/300 Write block 4171 len 32768 data: 6 7 6 7 6 7 6 6
32787 291/300 Write block 354 len 40960 data: 4 2 2 2 2 2 2 3 6 6
32788 292/300 Write block 2265 len 40960 data: 7 6 8 6 5 4 4 5 5 5
32789 293/300 Write block 1516 len 20480 data: 7 7 6 5 4
32790 294/300 Read block 2450 len 8192
32791 295/300 Flush
32792 296/300 Write block 1 len 12288 data: 5 4 4
32793 297/300 Flush
32794 298/300 Write block 1598 len 16384 data: 13 11 10 9
32795 299/300 Read block 3622 len 36864
32796 300/300 Read block 4240 len 28672
32797 301/300 Read block 2269 len 32768
32798 Main task has work to do, go find it
32799 got dsc Start(1)
32800 start 1
32801 Mar 17 22:41:51.978 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/1, method: POST, req_id: 19423121-e7e8-4b29-a682-6cd98b90d4db, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
32802 Replay: started 1, returned:Ok(())
32803 ----------------------------------------------------------------
32804 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32805 States: Active Offline Active
32806 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32807 9151 Acked Write 7 Done Sent Done false
32808 9152 Acked Read 6 Done Sent Done false
32809 9153 Acked Read 3 Done Sent Done false
32810 9154 Acked Read 5 Done Sent Done false
32811 9155 Acked Read 8 Done Sent Done false
32812 9156 Acked Read 3 Done Sent Done false
32813 9157 Acked Read 10 Done Sent Done false
32814 9158 Acked Write 8 Done Sent Done false
32815 9159 Acked Read 7 Done Sent Done false
32816 9160 Acked Flush 0 Done Sent Done false
32817 9161 Acked Write 6 Done Sent Done false
32818 9162 Acked Read 5 Done Sent Done false
32819 9163 Acked Write 3 Done Sent Done false
32820 9164 Acked Write 3 Done Sent Done false
32821 9165 Acked Read 10 Done Sent Done false
32822 9166 Acked Read 6 Done Sent Done false
32823 9167 Acked Read 8 Done Sent Done false
32824 9168 Acked Read 8 Done Sent Done false
32825 9169 Acked Read 2 Done Sent Done false
32826 9170 Acked Read 4 Done Sent Done false
32827 9171 Acked Read 3 Done Sent Done false
32828 9172 Acked Write 2 Done Sent Done false
32829 9173 Acked Read 10 Done Sent Done false
32830 9174 Acked Write 8 Done Sent Done false
32831 9175 Acked Write 3 Done Sent Done false
32832 9176 Acked Write 5 Done Sent Done false
32833 9177 Acked Write 9 Done Sent Done false
32834 9178 Acked Read 10 Done Sent Done false
32835 9179 Acked Write 2 Done Sent Done false
32836 9180 Acked Write 2 Done Sent Done false
32837 9181 Acked Write 3 Done Sent Done false
32838 9182 Acked Read 8 Done Sent Done false
32839 9183 Acked Read 4 Done Sent Done false
32840 9184 Acked Write 6 Done Sent Done false
32841 9185 Acked Write 5 Done Sent Done false
32842 9186 Acked Read 5 Done Sent Done false
32843 9187 Acked Read 1 Done Sent Done false
32844 9188 Acked Write 3 Done Sent Done false
32845 9189 Acked Read 6 Done Sent Done false
32846 9190 Acked Read 7 Done Sent Done false
32847 9191 Acked Write 10 Done Sent Done false
32848 9192 Acked Flush 0 Done Sent Done false
32849 9193 Acked Write 5 Done Sent Done false
32850 9194 Acked Write 7 Done Sent Done false
32851 9195 Acked Read 8 Done Sent Done false
32852 9196 Acked Read 8 Done Sent Done false
32853 9197 Acked Write 9 Done Sent Done false
32854 9198 Acked Write 5 Done Sent Done false
32855 9199 Acked Write 5 Done Sent Done false
32856 9200 Acked Read 3 Done Sent Done false
32857 9201 Acked Read 3 Done Sent Done false
32858 9202 Acked Write 6 Done Sent Done false
32859 9203 Acked Read 2 Done Sent Done false
32860 9204 Acked Flush 0 Done Sent Done false
32861 9205 Acked Read 7 Done Sent Done false
32862 9206 Acked Write 10 Done Sent Done false
32863 9207 Acked Write 9 Done Sent Done false
32864 9208 Acked Read 4 Done Sent Done false
32865 9209 Acked Flush 0 Done Sent Done false
32866 9210 Acked Write 6 Done Sent Done false
32867 9211 Acked Read 7 Done Sent Done false
32868 9212 Acked Write 2 Done Sent Done false
32869 9213 Acked Write 9 Done Sent Done false
32870 9214 Acked Read 3 Done Sent Done false
32871 9215 Acked Write 1 Done Sent Done false
32872 9216 Acked Write 8 Done Sent Done false
32873 9217 Acked Flush 0 Done Sent Done false
32874 9218 Acked Read 4 Done Sent Done false
32875 9219 Acked Read 8 Done Sent Done false
32876 9220 Acked Read 8 Done Sent Done false
32877 9221 Acked Write 3 Done Sent Done false
32878 9222 Acked Write 1 Done Sent Done false
32879 9223 Acked Read 9 Done Sent Done false
32880 9224 Acked Read 10 Done Sent Done false
32881 9225 Acked Write 6 Done Sent Done false
32882 9226 Acked Flush 0 Done Sent Done false
32883 9227 Acked Flush 0 Done Sent Done false
32884 9228 Acked Read 9 Done Sent Done false
32885 9229 Acked Read 2 Done Sent Done false
32886 9230 Acked Flush 0 Done Sent Done false
32887 9231 Acked Read 2 Done Sent Done false
32888 9232 Acked Write 8 Done Sent Done false
32889 9233 Acked Write 9 Done Sent Done false
32890 9234 Acked Write 2 Done Sent Done false
32891 9235 Acked Write 7 Done Sent Done false
32892 9236 Acked Read 9 Done Sent Done false
32893 9237 Acked Write 4 Done Sent Done false
32894 9238 Acked Read 6 Done Sent Done false
32895 9239 Acked Read 3 Done Sent Done false
32896 9240 Acked Read 10 Done Sent Done false
32897 9241 Acked Read 2 Done Sent Done false
32898 9242 Acked Read 10 Done Sent Done false
32899 9243 Acked Read 3 Done Sent Done false
32900 9244 Acked Read 10 Done Sent Done false
32901 9245 Acked Read 6 Done Sent Done false
32902 9246 Acked Write 8 Done Sent Done false
32903 9247 Acked Write 10 Done Sent Done false
32904 9248 Acked Write 1 Done Sent Done false
32905 9249 Acked Read 9 Done Sent Done false
32906 9250 Acked Flush 0 Done Sent Done false
32907 9251 Acked Write 3 Done Sent Done false
32908 9252 Acked Flush 0 Done Sent Done false
32909 9253 Acked Write 3 Done Sent Done false
32910 9254 Acked Write 10 Done Sent Done false
32911 9255 Acked Read 8 Done Sent Done false
32912 9256 Acked Read 3 Done Sent Done false
32913 9257 Acked Read 2 Done Sent Done false
32914 9258 Acked Write 6 Done Sent Done false
32915 9259 Acked Read 5 Done Sent Done false
32916 9260 Acked Write 2 Done Sent Done false
32917 9261 Acked Write 4 Done Sent Done false
32918 9262 Acked Write 4 Done Sent Done false
32919 9263 Acked Read 10 Done Sent Done false
32920 9264 Acked Read 5 Done Sent Done false
32921 9265 Acked Read 5 Done Sent Done false
32922 9266 Acked Write 7 Done Sent Done false
32923 9267 Acked Write 10 Done Sent Done false
32924 9268 Acked Write 7 Done Sent Done false
32925 9269 Acked Read 3 Done Sent Done false
32926 9270 Acked Write 4 Done Sent Done false
32927 9271 Acked Read 2 Done Sent Done false
32928 9272 Acked Flush 0 Done Sent Done false
32929 9273 Acked Read 5 Done Sent Done false
32930 9274 Acked Read 9 Done Sent Done false
32931 9275 Acked Read 10 Done Sent Done false
32932 9276 Acked Read 4 Done Sent Done false
32933 9277 Acked Write 1 Done Sent Done false
32934 9278 Acked Write 7 Done Sent Done false
32935 9279 Acked Write 5 Done Sent Done false
32936 9280 Acked Read 7 Done Sent Done false
32937 9281 Acked Write 9 Done Sent Done false
32938 9282 Acked Write 2 Done Sent Done false
32939 9283 Acked Write 2 Done Sent Done false
32940 9284 Acked Write 10 Done Sent Done false
32941 9285 Acked Read 1 Done Sent Done false
32942 9286 Acked Write 9 Done Sent Done false
32943 9287 Acked Read 9 Done Sent Done false
32944 9288 Acked Read 8 Done Sent Done false
32945 9289 Acked Read 6 Done Sent Done false
32946 9290 Acked Read 8 Done Sent Done false
32947 9291 Acked Read 8 Done Sent Done false
32948 9292 Acked Read 8 Done Sent Done false
32949 9293 Acked Read 10 Done Sent Done false
32950 9294 Acked Write 9 Done Sent Done false
32951 9295 Acked Read 10 Done Sent Done false
32952 9296 Acked Read 7 Done Sent Done false
32953 9297 Acked Read 7 Done Sent Done false
32954 9298 Acked Read 7 Done Sent Done false
32955 9299 Acked Write 8 Done Sent Done false
32956 9300 Acked Write 7 Done Sent Done false
32957 9301 Acked Read 9 Done Sent Done false
32958 9302 Acked Flush 0 Done Sent Done false
32959 9303 Acked Write 4 Done Sent Done false
32960 9304 Acked Write 4 Done Sent Done false
32961 9305 Acked Read 5 Done Sent Done false
32962 9306 Acked Read 8 Done Sent Done false
32963 9307 Acked Write 3 Done Sent Done false
32964 9308 Acked Read 2 Done Sent Done false
32965 9309 Acked Read 8 Done Sent Done false
32966 9310 Acked Read 5 Done Sent Done false
32967 9311 Acked Flush 0 Done Sent Done false
32968 9312 Acked Flush 0 Done Sent Done false
32969 9313 Acked Read 3 Done Sent Done false
32970 9314 Acked Read 7 Done Sent Done false
32971 9315 Acked Read 7 Done Sent Done false
32972 9316 Acked Read 2 Done Sent Done false
32973 9317 Acked Read 9 Done Sent Done false
32974 9318 Acked Read 7 Done Sent Done false
32975 9319 Acked Read 5 Done Sent Done false
32976 9320 Acked Write 8 Done Sent Done false
32977 9321 Acked Write 8 Done Sent Done false
32978 9322 Acked Read 1 Done Sent Done false
32979 9323 Acked Flush 0 Done Sent Done false
32980 9324 Acked Read 9 Done Sent Done false
32981 9325 Acked Write 9 Done Sent Done false
32982 9326 Acked Read 2 Done Sent Done false
32983 9327 Acked Read 6 Done Sent Done false
32984 9328 Acked Write 8 Done Sent Done false
32985 9329 Acked Read 10 Done Sent Done false
32986 9330 Acked Flush 0 Done Sent Done false
32987 9331 Acked Read 8 Done Sent Done false
32988 9332 Acked Write 7 Done Sent Done false
32989 9333 Acked Write 2 Done Sent Done false
32990 9334 Acked Write 3 Done Sent Done false
32991 9335 Acked Write 9 Done Sent Done false
32992 9336 Acked Write 10 Done Sent Done false
32993 9337 Acked Read 10 Done Sent Done false
32994 9338 Acked Flush 0 Done Sent Done false
32995 9339 Acked Write 4 Done Sent Done false
32996 9340 Acked Write 1 Done Sent Done false
32997 9341 Acked Read 8 Done Sent Done false
32998 9342 Acked Read 3 Done Sent Done false
32999 9343 Acked Read 5 Done Sent Done false
33000 9344 Acked Write 2 Done Sent Done false
33001 9345 Acked Read 1 Done Sent Done false
33002 9346 Acked Write 9 Done Sent Done false
33003 9347 Acked Read 10 Done Sent Done false
33004 9348 Acked Read 8 Done Sent Done false
33005 9349 Acked Write 2 Done Sent Done false
33006 9350 Acked Read 1 Done Sent Done false
33007 9351 Acked Write 2 Done Sent Done false
33008 9352 Acked Write 6 Done Sent Done false
33009 9353 Acked Flush 0 Done Sent Done false
33010 9354 Acked Write 1 Done Sent Done false
33011 9355 Acked Read 8 Done Sent Done false
33012 9356 Acked Read 3 Done Sent Done false
33013 9357 Acked Flush 0 Done Sent Done false
33014 9358 Acked Write 9 Done Sent Done false
33015 9359 Acked Read 4 Done Sent Done false
33016 9360 Acked Write 7 Done Sent Done false
33017 9361 Acked Write 9 Done Sent Done false
33018 9362 Acked Read 6 Done Sent Done false
33019 9363 Acked Read 9 Done Sent Done false
33020 9364 Acked Read 4 Done Sent Done false
33021 9365 Acked Write 4 Done Sent Done false
33022 9366 Acked Read 8 Done Sent Done false
33023 9367 Acked Write 10 Done Sent Done false
33024 9368 Acked Write 2 Done Sent Done false
33025 9369 Acked Read 9 Done Sent Done false
33026 9370 Acked Read 6 Done Sent Done false
33027 9371 Acked Write 10 Done Sent Done false
33028 9372 Acked Read 6 Done Sent Done false
33029 9373 Acked Write 4 Done Sent Done false
33030 9374 Acked Read 7 Done Sent Done false
33031 9375 Acked Read 5 Done Sent Done false
33032 9376 Acked Read 1 Done Sent Done false
33033 9377 Acked Write 2 Done Sent Done false
33034 9378 Acked Flush 0 Done Sent Done false
33035 9379 Acked Write 7 Done Sent Done false
33036 9380 Acked Flush 0 Done Sent Done false
33037 9381 Acked Write 3 Done Sent Done false
33038 9382 Acked Write 6 Done Sent Done false
33039 9383 Acked Read 4 Done Sent Done false
33040 9384 Acked Write 10 Done Sent Done false
33041 9385 Acked Read 5 Done Sent Done false
33042 9386 Acked Write 10 Done Sent Done false
33043 9387 Acked Read 1 Done Sent Done false
33044 9388 Acked Write 4 Done Sent Done false
33045 9389 Acked Flush 0 Done Sent Done false
33046 9390 Acked Write 5 Done Sent Done false
33047 9391 Acked Write 7 Done Sent Done false
33048 9392 Acked Read 4 Done Sent Done false
33049 9393 Acked Write 3 Done Sent Done false
33050 9394 Acked Read 4 Done Sent Done false
33051 9395 Acked Read 4 Done Sent Done false
33052 9396 Acked Write 9 Done Sent Done false
33053 9397 Acked Write 10 Done Sent Done false
33054 9398 Acked Read 9 Done Sent Done false
33055 9399 Acked Read 1 Done Sent Done false
33056 9400 Acked Read 9 Done Sent Done false
33057 9401 Acked Write 7 Done Sent Done false
33058 9402 Acked Read 6 Done Sent Done false
33059 9403 Acked Read 10 Done Sent Done false
33060 9404 Acked Write 3 Done Sent Done false
33061 9405 Acked Write 10 Done Sent Done false
33062 9406 Acked Write 2 Done Sent Done false
33063 9407 Acked Write 5 Done Sent Done false
33064 9408 Acked Write 7 Done Sent Done false
33065 9409 Acked Write 7 Done Sent Done false
33066 9410 Acked Read 3 Done Sent Done false
33067 9411 Acked Read 10 Done Sent Done false
33068 9412 Acked Read 1 Done Sent Done false
33069 9413 Acked Read 9 Done Sent Done false
33070 9414 Acked Read 10 Done Sent Done false
33071 9415 Acked Write 8 Done Sent Done false
33072 9416 Acked Read 8 Done Sent Done false
33073 9417 Acked Write 6 Done Sent Done false
33074 9418 Acked Flush 0 Done Sent Done false
33075 9419 Acked Read 9 Done Sent Done false
33076 9420 Acked Read 5 Done Sent Done false
33077 9421 Acked Read 5 Done Sent Done false
33078 9422 Acked Write 8 Done Sent Done false
33079 9423 Acked Write 9 Done Sent Done false
33080 9424 Acked Read 7 Done Sent Done false
33081 9425 Acked Flush 0 Done Sent Done false
33082 9426 Acked Read 5 Done Sent Done false
33083 9427 Acked Read 5 Done Sent Done false
33084 9428 Acked Write 6 Done Sent Done false
33085 9429 Acked Write 3 Done Sent Done false
33086 9430 Acked Write 6 Done Sent Done false
33087 9431 Acked Read 4 Done Sent Done false
33088 9432 Acked Read 8 Done Sent Done false
33089 9433 Acked Read 5 Done Sent Done false
33090 9434 Acked Read 3 Done Sent Done false
33091 9435 Acked Write 8 Done Sent Done false
33092 9436 Acked Write 5 Done Sent Done false
33093 9437 Acked Read 7 Done Sent Done false
33094 9438 Acked Read 5 Done Sent Done false
33095 9439 Acked Read 5 Done Sent Done false
33096 9440 Acked Write 8 Done Sent Done false
33097 9441 Acked Write 10 Done Sent Done false
33098 9442 Acked Write 10 Done Sent Done false
33099 9443 Acked Write 5 Done Sent Done false
33100 9444 Acked Read 2 Done Sent Done false
33101 9445 Acked Flush 0 Done Sent Done false
33102 9446 Acked Write 3 Done Sent Done false
33103 9447 Acked Flush 0 Done Sent Done false
33104 9448 Acked Write 4 Done Sent Done false
33105 9449 Acked Read 9 Done Sent Done false
33106 9450 Acked Read 7 Done Sent Done false
33107 9451 Acked Read 8 Done Sent Done false
33108 STATES DS:0 DS:1 DS:2 TOTAL
33109 Sent 0 301 0 301
33110 Done 301 0 301 602
33111 Skipped 0 0 0 0
33112 Error 0 0 0 0
33113 Last Flush: Some(JobId(9447)) Some(JobId(9150)) Some(JobId(9447))
33114 Downstairs last five completed: 9150 9149 9148 9147 9146
33115 Upstairs last five completed: 9451 9450 9449 9448 9447
33116 CLIENT: Up:0 ds:301 act:2
331172025-03-17T22:41:52.480ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
331182025-03-17T22:41:52.991ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
33119 [8820] got start action
33120 [8820] I am going to restart
33121 Starting downstairs at port 8820
33122 Make output file at "/tmp/dsc/downstairs-8820.txt"
33123 [8820][1] reports Starting
33124 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
33125 [8820][1] reports Running
331262025-03-17T22:41:53.992ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
331272025-03-17T22:41:54.994ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
33128 ----------------------------------------------------------------
33129 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33130 States: Active Offline Active
33131 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33132 9151 Acked Write 7 Done Sent Done false
33133 9152 Acked Read 6 Done Sent Done false
33134 9153 Acked Read 3 Done Sent Done false
33135 9154 Acked Read 5 Done Sent Done false
33136 9155 Acked Read 8 Done Sent Done false
33137 9156 Acked Read 3 Done Sent Done false
33138 9157 Acked Read 10 Done Sent Done false
33139 9158 Acked Write 8 Done Sent Done false
33140 9159 Acked Read 7 Done Sent Done false
33141 9160 Acked Flush 0 Done Sent Done false
33142 9161 Acked Write 6 Done Sent Done false
33143 9162 Acked Read 5 Done Sent Done false
33144 9163 Acked Write 3 Done Sent Done false
33145 9164 Acked Write 3 Done Sent Done false
33146 9165 Acked Read 10 Done Sent Done false
33147 9166 Acked Read 6 Done Sent Done false
33148 9167 Acked Read 8 Done Sent Done false
33149 9168 Acked Read 8 Done Sent Done false
33150 9169 Acked Read 2 Done Sent Done false
33151 9170 Acked Read 4 Done Sent Done false
33152 9171 Acked Read 3 Done Sent Done false
33153 9172 Acked Write 2 Done Sent Done false
33154 9173 Acked Read 10 Done Sent Done false
33155 9174 Acked Write 8 Done Sent Done false
33156 9175 Acked Write 3 Done Sent Done false
33157 9176 Acked Write 5 Done Sent Done false
33158 9177 Acked Write 9 Done Sent Done false
33159 9178 Acked Read 10 Done Sent Done false
33160 9179 Acked Write 2 Done Sent Done false
33161 9180 Acked Write 2 Done Sent Done false
33162 9181 Acked Write 3 Done Sent Done false
33163 9182 Acked Read 8 Done Sent Done false
33164 9183 Acked Read 4 Done Sent Done false
33165 9184 Acked Write 6 Done Sent Done false
33166 9185 Acked Write 5 Done Sent Done false
33167 9186 Acked Read 5 Done Sent Done false
33168 9187 Acked Read 1 Done Sent Done false
33169 9188 Acked Write 3 Done Sent Done false
33170 9189 Acked Read 6 Done Sent Done false
33171 9190 Acked Read 7 Done Sent Done false
33172 9191 Acked Write 10 Done Sent Done false
33173 9192 Acked Flush 0 Done Sent Done false
33174 9193 Acked Write 5 Done Sent Done false
33175 9194 Acked Write 7 Done Sent Done false
33176 9195 Acked Read 8 Done Sent Done false
33177 9196 Acked Read 8 Done Sent Done false
33178 9197 Acked Write 9 Done Sent Done false
33179 9198 Acked Write 5 Done Sent Done false
33180 9199 Acked Write 5 Done Sent Done false
33181 9200 Acked Read 3 Done Sent Done false
33182 9201 Acked Read 3 Done Sent Done false
33183 9202 Acked Write 6 Done Sent Done false
33184 9203 Acked Read 2 Done Sent Done false
33185 9204 Acked Flush 0 Done Sent Done false
33186 9205 Acked Read 7 Done Sent Done false
33187 9206 Acked Write 10 Done Sent Done false
33188 9207 Acked Write 9 Done Sent Done false
33189 9208 Acked Read 4 Done Sent Done false
33190 9209 Acked Flush 0 Done Sent Done false
33191 9210 Acked Write 6 Done Sent Done false
33192 9211 Acked Read 7 Done Sent Done false
33193 9212 Acked Write 2 Done Sent Done false
33194 9213 Acked Write 9 Done Sent Done false
33195 9214 Acked Read 3 Done Sent Done false
33196 9215 Acked Write 1 Done Sent Done false
33197 9216 Acked Write 8 Done Sent Done false
33198 9217 Acked Flush 0 Done Sent Done false
33199 9218 Acked Read 4 Done Sent Done false
33200 9219 Acked Read 8 Done Sent Done false
33201 9220 Acked Read 8 Done Sent Done false
33202 9221 Acked Write 3 Done Sent Done false
33203 9222 Acked Write 1 Done Sent Done false
33204 9223 Acked Read 9 Done Sent Done false
33205 9224 Acked Read 10 Done Sent Done false
33206 9225 Acked Write 6 Done Sent Done false
33207 9226 Acked Flush 0 Done Sent Done false
33208 9227 Acked Flush 0 Done Sent Done false
33209 9228 Acked Read 9 Done Sent Done false
33210 9229 Acked Read 2 Done Sent Done false
33211 9230 Acked Flush 0 Done Sent Done false
33212 9231 Acked Read 2 Done Sent Done false
33213 9232 Acked Write 8 Done Sent Done false
33214 9233 Acked Write 9 Done Sent Done false
33215 9234 Acked Write 2 Done Sent Done false
33216 9235 Acked Write 7 Done Sent Done false
33217 9236 Acked Read 9 Done Sent Done false
33218 9237 Acked Write 4 Done Sent Done false
33219 9238 Acked Read 6 Done Sent Done false
33220 9239 Acked Read 3 Done Sent Done false
33221 9240 Acked Read 10 Done Sent Done false
33222 9241 Acked Read 2 Done Sent Done false
33223 9242 Acked Read 10 Done Sent Done false
33224 9243 Acked Read 3 Done Sent Done false
33225 9244 Acked Read 10 Done Sent Done false
33226 9245 Acked Read 6 Done Sent Done false
33227 9246 Acked Write 8 Done Sent Done false
33228 9247 Acked Write 10 Done Sent Done false
33229 9248 Acked Write 1 Done Sent Done false
33230 9249 Acked Read 9 Done Sent Done false
33231 9250 Acked Flush 0 Done Sent Done false
33232 9251 Acked Write 3 Done Sent Done false
33233 9252 Acked Flush 0 Done Sent Done false
33234 9253 Acked Write 3 Done Sent Done false
33235 9254 Acked Write 10 Done Sent Done false
33236 9255 Acked Read 8 Done Sent Done false
33237 9256 Acked Read 3 Done Sent Done false
33238 9257 Acked Read 2 Done Sent Done false
33239 9258 Acked Write 6 Done Sent Done false
33240 9259 Acked Read 5 Done Sent Done false
33241 9260 Acked Write 2 Done Sent Done false
33242 9261 Acked Write 4 Done Sent Done false
33243 9262 Acked Write 4 Done Sent Done false
33244 9263 Acked Read 10 Done Sent Done false
33245 9264 Acked Read 5 Done Sent Done false
33246 9265 Acked Read 5 Done Sent Done false
33247 9266 Acked Write 7 Done Sent Done false
33248 9267 Acked Write 10 Done Sent Done false
33249 9268 Acked Write 7 Done Sent Done false
33250 9269 Acked Read 3 Done Sent Done false
33251 9270 Acked Write 4 Done Sent Done false
33252 9271 Acked Read 2 Done Sent Done false
33253 9272 Acked Flush 0 Done Sent Done false
33254 9273 Acked Read 5 Done Sent Done false
33255 9274 Acked Read 9 Done Sent Done false
33256 9275 Acked Read 10 Done Sent Done false
33257 9276 Acked Read 4 Done Sent Done false
33258 9277 Acked Write 1 Done Sent Done false
33259 9278 Acked Write 7 Done Sent Done false
33260 9279 Acked Write 5 Done Sent Done false
33261 9280 Acked Read 7 Done Sent Done false
33262 9281 Acked Write 9 Done Sent Done false
33263 9282 Acked Write 2 Done Sent Done false
33264 9283 Acked Write 2 Done Sent Done false
33265 9284 Acked Write 10 Done Sent Done false
33266 9285 Acked Read 1 Done Sent Done false
33267 9286 Acked Write 9 Done Sent Done false
33268 9287 Acked Read 9 Done Sent Done false
33269 9288 Acked Read 8 Done Sent Done false
33270 9289 Acked Read 6 Done Sent Done false
33271 9290 Acked Read 8 Done Sent Done false
33272 9291 Acked Read 8 Done Sent Done false
33273 9292 Acked Read 8 Done Sent Done false
33274 9293 Acked Read 10 Done Sent Done false
33275 9294 Acked Write 9 Done Sent Done false
33276 9295 Acked Read 10 Done Sent Done false
33277 9296 Acked Read 7 Done Sent Done false
33278 9297 Acked Read 7 Done Sent Done false
33279 9298 Acked Read 7 Done Sent Done false
33280 9299 Acked Write 8 Done Sent Done false
33281 9300 Acked Write 7 Done Sent Done false
33282 9301 Acked Read 9 Done Sent Done false
33283 9302 Acked Flush 0 Done Sent Done false
33284 9303 Acked Write 4 Done Sent Done false
33285 9304 Acked Write 4 Done Sent Done false
33286 9305 Acked Read 5 Done Sent Done false
33287 9306 Acked Read 8 Done Sent Done false
33288 9307 Acked Write 3 Done Sent Done false
33289 9308 Acked Read 2 Done Sent Done false
33290 9309 Acked Read 8 Done Sent Done false
33291 9310 Acked Read 5 Done Sent Done false
33292 9311 Acked Flush 0 Done Sent Done false
33293 9312 Acked Flush 0 Done Sent Done false
33294 9313 Acked Read 3 Done Sent Done false
33295 9314 Acked Read 7 Done Sent Done false
33296 9315 Acked Read 7 Done Sent Done false
33297 9316 Acked Read 2 Done Sent Done false
33298 9317 Acked Read 9 Done Sent Done false
33299 9318 Acked Read 7 Done Sent Done false
33300 9319 Acked Read 5 Done Sent Done false
33301 9320 Acked Write 8 Done Sent Done false
33302 9321 Acked Write 8 Done Sent Done false
33303 9322 Acked Read 1 Done Sent Done false
33304 9323 Acked Flush 0 Done Sent Done false
33305 9324 Acked Read 9 Done Sent Done false
33306 9325 Acked Write 9 Done Sent Done false
33307 9326 Acked Read 2 Done Sent Done false
33308 9327 Acked Read 6 Done Sent Done false
33309 9328 Acked Write 8 Done Sent Done false
33310 9329 Acked Read 10 Done Sent Done false
33311 9330 Acked Flush 0 Done Sent Done false
33312 9331 Acked Read 8 Done Sent Done false
33313 9332 Acked Write 7 Done Sent Done false
33314 9333 Acked Write 2 Done Sent Done false
33315 9334 Acked Write 3 Done Sent Done false
33316 9335 Acked Write 9 Done Sent Done false
33317 9336 Acked Write 10 Done Sent Done false
33318 9337 Acked Read 10 Done Sent Done false
33319 9338 Acked Flush 0 Done Sent Done false
33320 9339 Acked Write 4 Done Sent Done false
33321 9340 Acked Write 1 Done Sent Done false
33322 9341 Acked Read 8 Done Sent Done false
33323 9342 Acked Read 3 Done Sent Done false
33324 9343 Acked Read 5 Done Sent Done false
33325 9344 Acked Write 2 Done Sent Done false
33326 9345 Acked Read 1 Done Sent Done false
33327 9346 Acked Write 9 Done Sent Done false
33328 9347 Acked Read 10 Done Sent Done false
33329 9348 Acked Read 8 Done Sent Done false
33330 9349 Acked Write 2 Done Sent Done false
33331 9350 Acked Read 1 Done Sent Done false
33332 9351 Acked Write 2 Done Sent Done false
33333 9352 Acked Write 6 Done Sent Done false
33334 9353 Acked Flush 0 Done Sent Done false
33335 9354 Acked Write 1 Done Sent Done false
33336 9355 Acked Read 8 Done Sent Done false
33337 9356 Acked Read 3 Done Sent Done false
33338 9357 Acked Flush 0 Done Sent Done false
33339 9358 Acked Write 9 Done Sent Done false
33340 9359 Acked Read 4 Done Sent Done false
33341 9360 Acked Write 7 Done Sent Done false
33342 9361 Acked Write 9 Done Sent Done false
33343 9362 Acked Read 6 Done Sent Done false
33344 9363 Acked Read 9 Done Sent Done false
33345 9364 Acked Read 4 Done Sent Done false
33346 9365 Acked Write 4 Done Sent Done false
33347 9366 Acked Read 8 Done Sent Done false
33348 9367 Acked Write 10 Done Sent Done false
33349 9368 Acked Write 2 Done Sent Done false
33350 9369 Acked Read 9 Done Sent Done false
33351 9370 Acked Read 6 Done Sent Done false
33352 9371 Acked Write 10 Done Sent Done false
33353 9372 Acked Read 6 Done Sent Done false
33354 9373 Acked Write 4 Done Sent Done false
33355 9374 Acked Read 7 Done Sent Done false
33356 9375 Acked Read 5 Done Sent Done false
33357 9376 Acked Read 1 Done Sent Done false
33358 9377 Acked Write 2 Done Sent Done false
33359 9378 Acked Flush 0 Done Sent Done false
33360 9379 Acked Write 7 Done Sent Done false
33361 9380 Acked Flush 0 Done Sent Done false
33362 9381 Acked Write 3 Done Sent Done false
33363 9382 Acked Write 6 Done Sent Done false
33364 9383 Acked Read 4 Done Sent Done false
33365 9384 Acked Write 10 Done Sent Done false
33366 9385 Acked Read 5 Done Sent Done false
33367 9386 Acked Write 10 Done Sent Done false
33368 9387 Acked Read 1 Done Sent Done false
33369 9388 Acked Write 4 Done Sent Done false
33370 9389 Acked Flush 0 Done Sent Done false
33371 9390 Acked Write 5 Done Sent Done false
33372 9391 Acked Write 7 Done Sent Done false
33373 9392 Acked Read 4 Done Sent Done false
33374 9393 Acked Write 3 Done Sent Done false
33375 9394 Acked Read 4 Done Sent Done false
33376 9395 Acked Read 4 Done Sent Done false
33377 9396 Acked Write 9 Done Sent Done false
33378 9397 Acked Write 10 Done Sent Done false
33379 9398 Acked Read 9 Done Sent Done false
33380 9399 Acked Read 1 Done Sent Done false
33381 9400 Acked Read 9 Done Sent Done false
33382 9401 Acked Write 7 Done Sent Done false
33383 9402 Acked Read 6 Done Sent Done false
33384 9403 Acked Read 10 Done Sent Done false
33385 9404 Acked Write 3 Done Sent Done false
33386 9405 Acked Write 10 Done Sent Done false
33387 9406 Acked Write 2 Done Sent Done false
33388 9407 Acked Write 5 Done Sent Done false
33389 9408 Acked Write 7 Done Sent Done false
33390 9409 Acked Write 7 Done Sent Done false
33391 9410 Acked Read 3 Done Sent Done false
33392 9411 Acked Read 10 Done Sent Done false
33393 9412 Acked Read 1 Done Sent Done false
33394 9413 Acked Read 9 Done Sent Done false
33395 9414 Acked Read 10 Done Sent Done false
33396 9415 Acked Write 8 Done Sent Done false
33397 9416 Acked Read 8 Done Sent Done false
33398 9417 Acked Write 6 Done Sent Done false
33399 9418 Acked Flush 0 Done Sent Done false
33400 9419 Acked Read 9 Done Sent Done false
33401 9420 Acked Read 5 Done Sent Done false
33402 9421 Acked Read 5 Done Sent Done false
33403 9422 Acked Write 8 Done Sent Done false
33404 9423 Acked Write 9 Done Sent Done false
33405 9424 Acked Read 7 Done Sent Done false
33406 9425 Acked Flush 0 Done Sent Done false
33407 9426 Acked Read 5 Done Sent Done false
33408 9427 Acked Read 5 Done Sent Done false
33409 9428 Acked Write 6 Done Sent Done false
33410 9429 Acked Write 3 Done Sent Done false
33411 9430 Acked Write 6 Done Sent Done false
33412 9431 Acked Read 4 Done Sent Done false
33413 9432 Acked Read 8 Done Sent Done false
33414 9433 Acked Read 5 Done Sent Done false
33415 9434 Acked Read 3 Done Sent Done false
33416 9435 Acked Write 8 Done Sent Done false
33417 9436 Acked Write 5 Done Sent Done false
33418 9437 Acked Read 7 Done Sent Done false
33419 9438 Acked Read 5 Done Sent Done false
33420 9439 Acked Read 5 Done Sent Done false
33421 9440 Acked Write 8 Done Sent Done false
33422 9441 Acked Write 10 Done Sent Done false
33423 9442 Acked Write 10 Done Sent Done false
33424 9443 Acked Write 5 Done Sent Done false
33425 9444 Acked Read 2 Done Sent Done false
33426 9445 Acked Flush 0 Done Sent Done false
33427 9446 Acked Write 3 Done Sent Done false
33428 9447 Acked Flush 0 Done Sent Done false
33429 9448 Acked Write 4 Done Sent Done false
33430 9449 Acked Read 9 Done Sent Done false
33431 9450 Acked Read 7 Done Sent Done false
33432 9451 Acked Read 8 Done Sent Done false
33433 STATES DS:0 DS:1 DS:2 TOTAL
33434 Sent 0 301 0 301
33435 Done 301 0 301 602
33436 Skipped 0 0 0 0
33437 Error 0 0 0 0
33438 Last Flush: Some(JobId(9447)) Some(JobId(9150)) Some(JobId(9447))
33439 Downstairs last five completed: 9150 9149 9148 9147 9146
33440 Upstairs last five completed: 9451 9450 9449 9448 9447
33441 CLIENT: Up:0 ds:301 act:2
334422025-03-17T22:41:56.996ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334432025-03-17T22:41:57.716ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334442025-03-17T22:41:57.716ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54464) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334452025-03-17T22:41:57.718ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334462025-03-17T22:41:57.718ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334472025-03-17T22:41:57.718ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334482025-03-17T22:41:57.718ZINFOcrucible: send last flush ID to this DS: Some(JobId(9150)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334492025-03-17T22:41:57.718ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334502025-03-17T22:41:57.718ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9150)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334512025-03-17T22:41:57.718ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9150)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
33452 ----------------------------------------------------------------
33453 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33454 Downstairs last five completed: 9452 9451 9450 9449 9448
33455 Upstairs last five completed: 9452 9451 9450 9449 9448
33456 CLIENT: Up:0 ds:0 act:3
33457 Replay: All jobs finished
33458 Main task has work to do, go find it
33459 got dsc Stop(2)
33460 stop 2
33461 [8830] Got stop action so:false kr:false
33462 Mar 17 22:41:59.983 INFO request completed, latency_us: 345, response_code: 204, uri: /stop/cid/2, method: POST, req_id: bf2d2462-6ba8-4a16-8399-c85481eeffd8, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
334632025-03-17T22:41:59.983ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334642025-03-17T22:41:59.983ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334652025-03-17T22:41:59.983ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
334662025-03-17T22:41:59.983ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
33467 Mar 17 22:41:59.984 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 355bd502-6c38-4d3c-b151-07057513c950, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
33468 [8830] Exited with: ExitStatus(unix_wait_status(9))
33469 [8830][2] reports Exit
33470 Mar 17 22:42:03.986 INFO request completed, latency_us: 236, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1a1d877e-becf-4b31-9a27-3ed461fc4f0d, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
33471 001/300 Write block 4486 len 20480 data: 6 5 6 5 5
33472 002/300 Read block 1505 len 12288
33473 003/300 Read block 1944 len 24576
33474 004/300 Write block 77 len 12288 data: 5 5 6
33475 005/300 Read block 3234 len 16384
33476 006/300 Read block 3201 len 4096
33477 007/300 Write block 2767 len 24576 data: 6 7 7 7 6 7
33478 008/300 Write block 631 len 8192 data: 2 2
33479 009/300 Write block 3612 len 4096 data: 4
33480 010/300 Read block 214 len 20480
33481 011/300 Write block 1009 len 4096 data: 6
33482 012/300 Read block 1943 len 36864
33483 013/300 Write block 4174 len 16384 data: 8 7 8 7
33484 014/300 Read block 3753 len 32768
33485 015/300 Read block 1762 len 28672
33486 016/300 Write block 3281 len 8192 data: 7 6
33487 017/300 Read block 4130 len 16384
33488 018/300 Write block 2807 len 12288 data: 5 5 5
33489 019/300 Read block 1335 len 12288
33490 020/300 Read block 845 len 8192
33491 021/300 Write block 2981 len 40960 data: 11 9 8 8 9 12 9 10 9 11
33492 022/300 Write block 3434 len 24576 data: 11 9 9 7 6 4
33493 023/300 Read block 303 len 24576
33494 024/300 Write block 3581 len 20480 data: 5 5 4 4 2
33495 025/300 Read block 3706 len 8192
33496 026/300 Read block 3690 len 24576
33497 027/300 Read block 912 len 32768
33498 028/300 Read block 4226 len 36864
33499 029/300 Read block 58 len 24576
33500 030/300 Read block 2258 len 40960
33501 031/300 Write block 2278 len 40960 data: 4 3 4 5 3 4 4 5 4 5
33502 032/300 Write block 2263 len 36864 data: 6 6 8 7 9 7 6 5 5
33503 033/300 Read block 4573 len 36864
33504 034/300 Read block 1183 len 36864
33505 035/300 Write block 2266 len 32768 data: 8 10 8 7 6 6 6 6
33506 036/300 Write block 1892 len 36864 data: 3 4 5 5 6 6 5 4 5
33507 037/300 Read block 3646 len 32768
33508 038/300 Read block 4894 len 32768
33509 039/300 Write block 139 len 12288 data: 6 6 6
33510 040/300 Flush
33511 041/300 Read block 4907 len 36864
33512 042/300 Write block 2431 len 32768 data: 6 6 6 5 3 7 7 6
33513 043/300 Read block 4815 len 24576
33514 044/300 Write block 4470 len 4096 data: 2
33515 045/300 Write block 1145 len 8192 data: 5 4
33516 046/300 Write block 2325 len 12288 data: 3 2 3
33517 047/300 Write block 4879 len 32768 data: 4 4 5 6 6 6 6 5
33518 048/300 Read block 4720 len 40960
33519 049/300 Read block 1476 len 4096
33520 050/300 Write block 3865 len 28672 data: 4 4 4 4 4 5 6
33521 051/300 Flush
33522 052/300 Write block 2457 len 24576 data: 4 3 3 4 4 6
33523 053/300 Read block 4864 len 4096
33524 054/300 Flush
33525 055/300 Read block 3443 len 32768
33526 056/300 Read block 2574 len 24576
33527 057/300 Write block 1296 len 12288 data: 8 8 6
33528 058/300 Read block 3243 len 12288
33529 059/300 Read block 3126 len 8192
33530 060/300 Write block 2871 len 32768 data: 5 7 6 5 5 7 8 7
33531 061/300 Read block 145 len 12288
33532 062/300 Write block 45 len 12288 data: 8 7 7
33533 063/300 Read block 2992 len 8192
33534 064/300 Write block 4706 len 28672 data: 8 5 4 6 5 4 5
33535 065/300 Flush
33536 066/300 Read block 916 len 16384
33537 067/300 Write block 2639 len 8192 data: 4 4
33538 068/300 Write block 1986 len 4096 data: 10
33539 069/300 Write block 871 len 40960 data: 5 6 5 5 6 7 7 5 5 5
33540 070/300 Read block 1567 len 16384
33541 071/300 Flush
33542 072/300 Write block 1947 len 36864 data: 8 8 9 7 4 4 4 4 5
33543 073/300 Write block 3782 len 32768 data: 6 8 5 7 6 6 6 4
33544 074/300 Read block 4863 len 36864
33545 075/300 Read block 2446 len 12288
33546 076/300 Write block 49 len 24576 data: 4 3 2 2 5 6
33547 077/300 Read block 255 len 40960
33548 078/300 Read block 1930 len 24576
33549 079/300 Read block 47 len 28672
33550 080/300 Write block 1083 len 40960 data: 6 6 4 4 6 7 6 7 6 6
33551 081/300 Write block 4522 len 12288 data: 6 5 5
33552 082/300 Write block 4103 len 28672 data: 4 4 3 5 6 6 7
33553 083/300 Write block 370 len 36864 data: 4 5 4 4 3 5 3 3 4
33554 084/300 Read block 2478 len 20480
33555 085/300 Flush
33556 086/300 Write block 706 len 36864 data: 8 7 7 7 6 5 5 5 5
33557 087/300 Flush
33558 088/300 Read block 2103 len 28672
33559 089/300 Write block 3319 len 28672 data: 9 11 9 10 11 10 10
33560 090/300 Write block 3856 len 36864 data: 9 7 6 5 6 3 3 4 4
33561 091/300 Read block 4179 len 20480
33562 092/300 Read block 615 len 32768
33563 093/300 Read block 4944 len 12288
33564 094/300 Read block 293 len 36864
33565 095/300 Read block 2176 len 16384
33566 096/300 Read block 3334 len 16384
33567 097/300 Read block 4302 len 20480
33568 098/300 Write block 2569 len 20480 data: 6 7 8 8 8
33569 099/300 Read block 2215 len 32768
33570 100/300 Write block 3502 len 40960 data: 5 5 6 5 6 6 7 7 5 4
33571 101/300 Read block 3194 len 16384
33572 102/300 Write block 841 len 24576 data: 4 5 4 4 6 7
33573 103/300 Read block 556 len 4096
33574 104/300 Write block 720 len 4096 data: 4
33575 105/300 Read block 2843 len 12288
33576 106/300 Read block 1667 len 24576
33577 107/300 Flush
33578 108/300 Read block 2354 len 16384
33579 109/300 Write block 1875 len 4096 data: 4
33580 110/300 Write block 1623 len 32768 data: 6 7 7 6 6 6 5 3
33581 111/300 Read block 2800 len 20480
33582 112/300 Read block 1949 len 28672
33583 113/300 Write block 1589 len 8192 data: 3 3
33584 114/300 Read block 784 len 16384
33585 115/300 Flush
33586 116/300 Read block 3396 len 12288
33587 117/300 Read block 958 len 8192
33588 118/300 Write block 1113 len 28672 data: 2 2 2 2 2 3 3
33589 119/300 Write block 1874 len 12288 data: 5 5 5
33590 120/300 Write block 2832 len 12288 data: 7 7 8
33591 121/300 Read block 30 len 12288
33592 122/300 Read block 4404 len 40960
33593 123/300 Write block 1647 len 8192 data: 5 6
33594 124/300 Read block 1822 len 40960
33595 125/300 Write block 1957 len 28672 data: 7 7 8 9 8 10 7
33596 126/300 Read block 1154 len 28672
33597 127/300 Write block 3886 len 40960 data: 5 6 5 5 5 7 6 7 7 7
33598 128/300 Read block 408 len 28672
33599 129/300 Read block 4988 len 12288
33600 130/300 Write block 63 len 24576 data: 4 5 6 7 8 8
33601 131/300 Read block 2027 len 40960
33602 132/300 Write block 4774 len 4096 data: 8
33603 133/300 Flush
33604 134/300 Write block 1925 len 8192 data: 7 8
33605 135/300 Read block 332 len 36864
33606 136/300 Read block 600 len 24576
33607 137/300 Read block 4066 len 12288
33608 138/300 Write block 4201 len 28672 data: 6 6 7 7 7 8 8
33609 139/300 Read block 1930 len 4096
33610 140/300 Write block 1673 len 40960 data: 6 7 5 5 6 4 5 4 5 5
33611 141/300 Read block 4422 len 8192
33612 142/300 Write block 4410 len 8192 data: 5 4
33613 143/300 Write block 3980 len 20480 data: 4 4 4 4 4
33614 144/300 Read block 3424 len 16384
33615 145/300 Write block 4852 len 8192 data: 4 4
33616 146/300 Read block 1470 len 32768
33617 147/300 Write block 3234 len 8192 data: 9 9
33618 148/300 Write block 3820 len 36864 data: 4 5 5 5 7 7 5 4 4
33619 149/300 Write block 3534 len 32768 data: 5 4 4 4 4 3 3 4
33620 150/300 Write block 2389 len 28672 data: 3 3 3 3 4 3 3
33621 151/300 Flush
33622 152/300 Write block 3426 len 16384 data: 9 8 7 8
33623 153/300 Flush
33624 154/300 Flush
33625 155/300 Read block 3487 len 16384
33626 156/300 Write block 2842 len 16384 data: 4 3 3 4
33627 157/300 Read block 3214 len 40960
33628 158/300 Read block 1341 len 28672
33629 159/300 Read block 4743 len 20480
33630 160/300 Write block 2508 len 36864 data: 5 5 4 4 4 3 3 5 6
33631 161/300 Write block 399 len 12288 data: 2 4 4
33632 162/300 Write block 4130 len 12288 data: 6 6 6
33633 163/300 Read block 3150 len 24576
33634 164/300 Write block 559 len 16384 data: 7 7 6 8
33635 165/300 Read block 2848 len 8192
33636 166/300 Write block 3984 len 4096 data: 5
33637 167/300 Write block 1636 len 4096 data: 6
33638 168/300 Read block 64 len 24576
33639 169/300 Read block 490 len 32768
33640 170/300 Flush
33641 171/300 Read block 2908 len 4096
33642 172/300 Write block 4016 len 20480 data: 7 8 7 7 7
33643 173/300 Flush
33644 174/300 Read block 238 len 24576
33645 175/300 Write block 4988 len 12288 data: 5 5 4
33646 176/300 Read block 2950 len 4096
33647 177/300 Read block 1043 len 36864
33648 178/300 Read block 1252 len 40960
33649 179/300 Read block 92 len 24576
33650 180/300 Read block 2579 len 12288
33651 181/300 Read block 1818 len 36864
33652 182/300 Write block 2383 len 40960 data: 2 2 3 3 3 2 4 4 4 4
33653 183/300 Write block 3273 len 40960 data: 3 3 4 5 5 6 5 7 8 7
33654 184/300 Write block 4749 len 12288 data: 5 5 4
33655 185/300 Read block 1332 len 12288
33656 186/300 Read block 2317 len 16384
33657 187/300 Read block 4012 len 40960
33658 188/300 Write block 3177 len 20480 data: 7 8 8 8 9
33659 189/300 Read block 2417 len 24576
33660 190/300 Write block 527 len 12288 data: 7 8 7
33661 191/300 Read block 830 len 12288
33662 192/300 Read block 3139 len 40960
33663 193/300 Flush
33664 194/300 Write block 2263 len 32768 data: 7 7 9 9 11 9 8 7
33665 195/300 Write block 3767 len 28672 data: 4 5 4 5 5 4 5
33666 196/300 Write block 1700 len 28672 data: 6 7 8 8 6 6 6
33667 197/300 Read block 1796 len 12288
33668 198/300 Read block 2531 len 36864
33669 199/300 Read block 4954 len 28672
33670 200/300 Read block 1572 len 32768
33671 201/300 Read block 3851 len 28672
33672 202/300 Read block 3525 len 8192
33673 203/300 Read block 3880 len 32768
33674 204/300 Flush
33675 205/300 Read block 2124 len 20480
33676 206/300 Write block 613 len 4096 data: 8
33677 207/300 Read block 4680 len 8192
33678 208/300 Write block 3866 len 32768 data: 5 5 5 5 6 7 7 8
33679 209/300 Write block 4814 len 8192 data: 4 3
33680 210/300 Write block 623 len 4096 data: 3
33681 211/300 Flush
33682 212/300 Read block 884 len 4096
33683 213/300 Read block 3344 len 40960
33684 214/300 Write block 1687 len 4096 data: 7
33685 215/300 Write block 3252 len 20480 data: 7 6 5 5 6
33686 216/300 Read block 851 len 4096
33687 217/300 Write block 1230 len 40960 data: 4 5 5 6 6 6 5 4 5 4
33688 218/300 Write block 2834 len 8192 data: 9 8
33689 219/300 Read block 3139 len 16384
33690 220/300 Write block 2744 len 4096 data: 4
33691 221/300 Read block 3641 len 20480
33692 222/300 Flush
33693 223/300 Write block 2465 len 32768 data: 6 7 6 9 8 9 9 9
33694 224/300 Write block 4340 len 4096 data: 8
33695 225/300 Read block 661 len 28672
33696 226/300 Write block 2190 len 12288 data: 4 3 3
33697 227/300 Read block 4169 len 8192
33698 228/300 Read block 3388 len 28672
33699 229/300 Read block 3129 len 4096
33700 230/300 Flush
33701 231/300 Write block 415 len 8192 data: 8 8
33702 232/300 Read block 2026 len 40960
33703 233/300 Read block 1758 len 24576
33704 234/300 Write block 1361 len 32768 data: 9 9 9 9 7 5 5 5
33705 235/300 Read block 4825 len 8192
33706 236/300 Write block 4015 len 24576 data: 8 8 9 8 8 8
33707 237/300 Read block 1936 len 16384
33708 238/300 Read block 2164 len 32768
33709 239/300 Read block 186 len 36864
33710 240/300 Flush
33711 241/300 Flush
33712 242/300 Write block 244 len 40960 data: 5 4 5 5 4 5 7 6 5 7
33713 243/300 Flush
33714 244/300 Write block 4548 len 32768 data: 4 7 8 7 7 7 8 7
33715 245/300 Flush
33716 246/300 Read block 3306 len 32768
33717 247/300 Read block 4524 len 20480
33718 248/300 Write block 4415 len 8192 data: 5 4
33719 249/300 Write block 1216 len 20480 data: 5 6 5 6 6
33720 250/300 Read block 2943 len 16384
33721 251/300 Read block 1427 len 40960
33722 252/300 Write block 3668 len 24576 data: 6 7 5 5 4 5
33723 253/300 Write block 2561 len 40960 data: 6 7 8 6 5 5 4 6 7 8
33724 254/300 Read block 427 len 28672
33725 255/300 Read block 1662 len 36864
33726 256/300 Flush
33727 257/300 Write block 2112 len 32768 data: 5 5 6 7 7 6 5 6
33728 258/300 Read block 577 len 36864
33729 259/300 Flush
33730 260/300 Read block 1013 len 36864
33731 261/300 Read block 3272 len 28672
33732 262/300 Write block 3491 len 16384 data: 3 3 4 4
33733 263/300 Write block 2642 len 28672 data: 4 3 3 3 3 3 4
33734 264/300 Write block 2449 len 20480 data: 6 5 5 7 6
33735 265/300 Read block 401 len 36864
33736 266/300 Read block 769 len 24576
33737 267/300 Read block 915 len 16384
33738 268/300 Write block 1052 len 16384 data: 4 3 4 5
33739 269/300 Write block 3423 len 16384 data: 9 10 10 10
33740 270/300 Write block 2960 len 12288 data: 3 4 5
33741 271/300 Read block 3395 len 16384
33742 272/300 Flush
33743 273/300 Read block 4107 len 8192
33744 274/300 Flush
33745 275/300 Write block 2333 len 32768 data: 4 4 6 5 5 5 6 5
33746 276/300 Read block 2004 len 12288
33747 277/300 Read block 790 len 24576
33748 278/300 Read block 926 len 20480
33749 279/300 Read block 2444 len 8192
33750 280/300 Read block 4878 len 16384
33751 281/300 Read block 720 len 28672
33752 282/300 Write block 367 len 28672 data: 5 4 5 5 6 5 5
33753 283/300 Read block 4755 len 36864
33754 284/300 Read block 1535 len 28672
33755 285/300 Write block 45 len 8192 data: 9 8
33756 286/300 Write block 1969 len 28672 data: 4 3 3 3 3 3 3
33757 287/300 Read block 2328 len 36864
33758 288/300 Write block 3722 len 4096 data: 11
33759 289/300 Write block 4546 len 24576 data: 5 5 5 8 9 8
33760 290/300 Read block 2408 len 12288
33761 291/300 Read block 2090 len 36864
33762 292/300 Write block 2038 len 16384 data: 9 9 10 9
33763 293/300 Read block 2283 len 4096
33764 294/300 Write block 3780 len 24576 data: 8 6 7 9 6 8
33765 295/300 Write block 1401 len 4096 data: 7
33766 296/300 Write block 970 len 32768 data: 7 7 5 5 7 8 7 6
33767 297/300 Read block 204 len 8192
33768 298/300 Write block 44 len 32768 data: 9 10 9 8 6 5 4 3
33769 299/300 Read block 672 len 8192
33770 300/300 Write block 62 len 28672 data: 6 5 6 7 8 9 9
33771 301/300 Read block 3847 len 36864
33772 Main task has work to do, go find it
33773 got dsc Start(2)
33774 start 2
33775 Mar 17 22:42:04.232 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/2, method: POST, req_id: 43e10749-d01c-4c6a-8ac5-04ab36c57317, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
33776 Replay: started 2, returned:Ok(())
33777 ----------------------------------------------------------------
33778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33779 States: Active Active Offline
33780 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33781 9453 Acked Write 5 Done Done Sent false
33782 9454 Acked Read 3 Done Done Sent false
33783 9455 Acked Read 6 Done Done Sent false
33784 9456 Acked Write 3 Done Done Sent false
33785 9457 Acked Read 4 Done Done Sent false
33786 9458 Acked Read 1 Done Done Sent false
33787 9459 Acked Write 6 Done Done Sent false
33788 9460 Acked Write 2 Done Done Sent false
33789 9461 Acked Write 1 Done Done Sent false
33790 9462 Acked Read 5 Done Done Sent false
33791 9463 Acked Write 1 Done Done Sent false
33792 9464 Acked Read 9 Done Done Sent false
33793 9465 Acked Write 4 Done Done Sent false
33794 9466 Acked Read 8 Done Done Sent false
33795 9467 Acked Read 7 Done Done Sent false
33796 9468 Acked Write 2 Done Done Sent false
33797 9469 Acked Read 4 Done Done Sent false
33798 9470 Acked Write 3 Done Done Sent false
33799 9471 Acked Read 3 Done Done Sent false
33800 9472 Acked Read 2 Done Done Sent false
33801 9473 Acked Write 10 Done Done Sent false
33802 9474 Acked Write 6 Done Done Sent false
33803 9475 Acked Read 6 Done Done Sent false
33804 9476 Acked Write 5 Done Done Sent false
33805 9477 Acked Read 2 Done Done Sent false
33806 9478 Acked Read 6 Done Done Sent false
33807 9479 Acked Read 8 Done Done Sent false
33808 9480 Acked Read 9 Done Done Sent false
33809 9481 Acked Read 6 Done Done Sent false
33810 9482 Acked Read 10 Done Done Sent false
33811 9483 Acked Write 10 Done Done Sent false
33812 9484 Acked Write 9 Done Done Sent false
33813 9485 Acked Read 9 Done Done Sent false
33814 9486 Acked Read 9 Done Done Sent false
33815 9487 Acked Write 8 Done Done Sent false
33816 9488 Acked Write 9 Done Done Sent false
33817 9489 Acked Read 8 Done Done Sent false
33818 9490 Acked Read 8 Done Done Sent false
33819 9491 Acked Write 3 Done Done Sent false
33820 9492 Acked Flush 0 Done Done Sent false
33821 9493 Acked Read 9 Done Done Sent false
33822 9494 Acked Write 8 Done Done Sent false
33823 9495 Acked Read 6 Done Done Sent false
33824 9496 Acked Write 1 Done Done Sent false
33825 9497 Acked Write 2 Done Done Sent false
33826 9498 Acked Write 3 Done Done Sent false
33827 9499 Acked Write 8 Done Done Sent false
33828 9500 Acked Read 10 Done Done Sent false
33829 9501 Acked Read 1 Done Done Sent false
33830 9502 Acked Write 7 Done Done Sent false
33831 9503 Acked Flush 0 Done Done Sent false
33832 9504 Acked Write 6 Done Done Sent false
33833 9505 Acked Read 1 Done Done Sent false
33834 9506 Acked Flush 0 Done Done Sent false
33835 9507 Acked Read 8 Done Done Sent false
33836 9508 Acked Read 6 Done Done Sent false
33837 9509 Acked Write 3 Done Done Sent false
33838 9510 Acked Read 3 Done Done Sent false
33839 9511 Acked Read 2 Done Done Sent false
33840 9512 Acked Write 8 Done Done Sent false
33841 9513 Acked Read 3 Done Done Sent false
33842 9514 Acked Write 3 Done Done Sent false
33843 9515 Acked Read 2 Done Done Sent false
33844 9516 Acked Write 7 Done Done Sent false
33845 9517 Acked Flush 0 Done Done Sent false
33846 9518 Acked Read 4 Done Done Sent false
33847 9519 Acked Write 2 Done Done Sent false
33848 9520 Acked Write 1 Done Done Sent false
33849 9521 Acked Write 10 Done Done Sent false
33850 9522 Acked Read 4 Done Done Sent false
33851 9523 Acked Flush 0 Done Done Sent false
33852 9524 Acked Write 9 Done Done Sent false
33853 9525 Acked Write 8 Done Done Sent false
33854 9526 Acked Read 9 Done Done Sent false
33855 9527 Acked Read 3 Done Done Sent false
33856 9528 Acked Write 6 Done Done Sent false
33857 9529 Acked Read 10 Done Done Sent false
33858 9530 Acked Read 6 Done Done Sent false
33859 9531 Acked Read 7 Done Done Sent false
33860 9532 Acked Write 10 Done Done Sent false
33861 9533 Acked Write 3 Done Done Sent false
33862 9534 Acked Write 7 Done Done Sent false
33863 9535 Acked Write 9 Done Done Sent false
33864 9536 Acked Read 5 Done Done Sent false
33865 9537 Acked Flush 0 Done Done Sent false
33866 9538 Acked Write 9 Done Done Sent false
33867 9539 Acked Flush 0 Done Done Sent false
33868 9540 Acked Read 7 Done Done Sent false
33869 9541 Acked Write 7 Done Done Sent false
33870 9542 Acked Write 9 Done Done Sent false
33871 9543 Acked Read 5 Done Done Sent false
33872 9544 Acked Read 8 Done Done Sent false
33873 9545 Acked Read 3 Done Done Sent false
33874 9546 Acked Read 9 Done Done Sent false
33875 9547 Acked Read 4 Done Done Sent false
33876 9548 Acked Read 4 Done Done Sent false
33877 9549 Acked Read 5 Done Done Sent false
33878 9550 Acked Write 5 Done Done Sent false
33879 9551 Acked Read 8 Done Done Sent false
33880 9552 Acked Write 10 Done Done Sent false
33881 9553 Acked Read 4 Done Done Sent false
33882 9554 Acked Write 6 Done Done Sent false
33883 9555 Acked Read 1 Done Done Sent false
33884 9556 Acked Write 1 Done Done Sent false
33885 9557 Acked Read 3 Done Done Sent false
33886 9558 Acked Read 6 Done Done Sent false
33887 9559 Acked Flush 0 Done Done Sent false
33888 9560 Acked Read 4 Done Done Sent false
33889 9561 Acked Write 1 Done Done Sent false
33890 9562 Acked Write 8 Done Done Sent false
33891 9563 Acked Read 5 Done Done Sent false
33892 9564 Acked Read 7 Done Done Sent false
33893 9565 Acked Write 2 Done Done Sent false
33894 9566 Acked Read 4 Done Done Sent false
33895 9567 Acked Flush 0 Done Done Sent false
33896 9568 Acked Read 3 Done Done Sent false
33897 9569 Acked Read 2 Done Done Sent false
33898 9570 Acked Write 7 Done Done Sent false
33899 9571 Acked Write 3 Done Done Sent false
33900 9572 Acked Write 3 Done Done Sent false
33901 9573 Acked Read 3 Done Done Sent false
33902 9574 Acked Read 10 Done Done Sent false
33903 9575 Acked Write 2 Done Done Sent false
33904 9576 Acked Read 10 Done Done Sent false
33905 9577 Acked Write 7 Done Done Sent false
33906 9578 Acked Read 7 Done Done Sent false
33907 9579 Acked Write 10 Done Done Sent false
33908 9580 Acked Read 7 Done Done Sent false
33909 9581 Acked Read 3 Done Done Sent false
33910 9582 Acked Write 6 Done Done Sent false
33911 9583 Acked Read 10 Done Done Sent false
33912 9584 Acked Write 1 Done Done Sent false
33913 9585 Acked Flush 0 Done Done Sent false
33914 9586 Acked Write 2 Done Done Sent false
33915 9587 Acked Read 9 Done Done Sent false
33916 9588 Acked Read 6 Done Done Sent false
33917 9589 Acked Read 3 Done Done Sent false
33918 9590 Acked Write 7 Done Done Sent false
33919 9591 Acked Read 1 Done Done Sent false
33920 9592 Acked Write 10 Done Done Sent false
33921 9593 Acked Read 2 Done Done Sent false
33922 9594 Acked Write 2 Done Done Sent false
33923 9595 Acked Write 5 Done Done Sent false
33924 9596 Acked Read 4 Done Done Sent false
33925 9597 Acked Write 2 Done Done Sent false
33926 9598 Acked Read 8 Done Done Sent false
33927 9599 Acked Write 2 Done Done Sent false
33928 9600 Acked Write 9 Done Done Sent false
33929 9601 Acked Write 8 Done Done Sent false
33930 9602 Acked Write 7 Done Done Sent false
33931 9603 Acked Flush 0 Done Done Sent false
33932 9604 Acked Write 4 Done Done Sent false
33933 9605 Acked Flush 0 Done Done Sent false
33934 9606 Acked Flush 0 Done Done Sent false
33935 9607 Acked Read 4 Done Done Sent false
33936 9608 Acked Write 4 Done Done Sent false
33937 9609 Acked Read 10 Done Done Sent false
33938 9610 Acked Read 7 Done Done Sent false
33939 9611 Acked Read 5 Done Done Sent false
33940 9612 Acked Write 9 Done Done Sent false
33941 9613 Acked Write 3 Done Done Sent false
33942 9614 Acked Write 3 Done Done Sent false
33943 9615 Acked Read 6 Done Done Sent false
33944 9616 Acked Write 4 Done Done Sent false
33945 9617 Acked Read 2 Done Done Sent false
33946 9618 Acked Write 1 Done Done Sent false
33947 9619 Acked Write 1 Done Done Sent false
33948 9620 Acked Read 6 Done Done Sent false
33949 9621 Acked Read 8 Done Done Sent false
33950 9622 Acked Flush 0 Done Done Sent false
33951 9623 Acked Read 1 Done Done Sent false
33952 9624 Acked Write 5 Done Done Sent false
33953 9625 Acked Flush 0 Done Done Sent false
33954 9626 Acked Read 6 Done Done Sent false
33955 9627 Acked Write 3 Done Done Sent false
33956 9628 Acked Read 1 Done Done Sent false
33957 9629 Acked Read 9 Done Done Sent false
33958 9630 Acked Read 10 Done Done Sent false
33959 9631 Acked Read 6 Done Done Sent false
33960 9632 Acked Read 3 Done Done Sent false
33961 9633 Acked Read 9 Done Done Sent false
33962 9634 Acked Write 10 Done Done Sent false
33963 9635 Acked Write 10 Done Done Sent false
33964 9636 Acked Write 3 Done Done Sent false
33965 9637 Acked Read 3 Done Done Sent false
33966 9638 Acked Read 4 Done Done Sent false
33967 9639 Acked Read 10 Done Done Sent false
33968 9640 Acked Write 5 Done Done Sent false
33969 9641 Acked Read 6 Done Done Sent false
33970 9642 Acked Write 3 Done Done Sent false
33971 9643 Acked Read 3 Done Done Sent false
33972 9644 Acked Read 10 Done Done Sent false
33973 9645 Acked Flush 0 Done Done Sent false
33974 9646 Acked Write 8 Done Done Sent false
33975 9647 Acked Write 7 Done Done Sent false
33976 9648 Acked Write 7 Done Done Sent false
33977 9649 Acked Read 3 Done Done Sent false
33978 9650 Acked Read 9 Done Done Sent false
33979 9651 Acked Read 7 Done Done Sent false
33980 9652 Acked Read 8 Done Done Sent false
33981 9653 Acked Read 7 Done Done Sent false
33982 9654 Acked Read 2 Done Done Sent false
33983 9655 Acked Read 8 Done Done Sent false
33984 9656 Acked Flush 0 Done Done Sent false
33985 9657 Acked Read 5 Done Done Sent false
33986 9658 Acked Write 1 Done Done Sent false
33987 9659 Acked Read 2 Done Done Sent false
33988 9660 Acked Write 8 Done Done Sent false
33989 9661 Acked Write 2 Done Done Sent false
33990 9662 Acked Write 1 Done Done Sent false
33991 9663 Acked Flush 0 Done Done Sent false
33992 9664 Acked Read 1 Done Done Sent false
33993 9665 Acked Read 10 Done Done Sent false
33994 9666 Acked Write 1 Done Done Sent false
33995 9667 Acked Write 5 Done Done Sent false
33996 9668 Acked Read 1 Done Done Sent false
33997 9669 Acked Write 10 Done Done Sent false
33998 9670 Acked Write 2 Done Done Sent false
33999 9671 Acked Read 4 Done Done Sent false
34000 9672 Acked Write 1 Done Done Sent false
34001 9673 Acked Read 5 Done Done Sent false
34002 9674 Acked Flush 0 Done Done Sent false
34003 9675 Acked Write 8 Done Done Sent false
34004 9676 Acked Write 1 Done Done Sent false
34005 9677 Acked Read 7 Done Done Sent false
34006 9678 Acked Write 3 Done Done Sent false
34007 9679 Acked Read 2 Done Done Sent false
34008 9680 Acked Read 7 Done Done Sent false
34009 9681 Acked Read 1 Done Done Sent false
34010 9682 Acked Flush 0 Done Done Sent false
34011 9683 Acked Write 2 Done Done Sent false
34012 9684 Acked Read 10 Done Done Sent false
34013 9685 Acked Read 6 Done Done Sent false
34014 9686 Acked Write 8 Done Done Sent false
34015 9687 Acked Read 2 Done Done Sent false
34016 9688 Acked Write 6 Done Done Sent false
34017 9689 Acked Read 4 Done Done Sent false
34018 9690 Acked Read 8 Done Done Sent false
34019 9691 Acked Read 9 Done Done Sent false
34020 9692 Acked Flush 0 Done Done Sent false
34021 9693 Acked Flush 0 Done Done Sent false
34022 9694 Acked Write 10 Done Done Sent false
34023 9695 Acked Flush 0 Done Done Sent false
34024 9696 Acked Write 8 Done Done Sent false
34025 9697 Acked Flush 0 Done Done Sent false
34026 9698 Acked Read 8 Done Done Sent false
34027 9699 Acked Read 5 Done Done Sent false
34028 9700 Acked Write 2 Done Done Sent false
34029 9701 Acked Write 5 Done Done Sent false
34030 9702 Acked Read 4 Done Done Sent false
34031 9703 Acked Read 10 Done Done Sent false
34032 9704 Acked Write 6 Done Done Sent false
34033 9705 Acked Write 10 Done Done Sent false
34034 9706 Acked Read 7 Done Done Sent false
34035 9707 Acked Read 9 Done Done Sent false
34036 9708 Acked Flush 0 Done Done Sent false
34037 9709 Acked Write 8 Done Done Sent false
34038 9710 Acked Read 9 Done Done Sent false
34039 9711 Acked Flush 0 Done Done Sent false
34040 9712 Acked Read 9 Done Done Sent false
34041 9713 Acked Read 7 Done Done Sent false
34042 9714 Acked Write 4 Done Done Sent false
34043 9715 Acked Write 7 Done Done Sent false
34044 9716 Acked Write 5 Done Done Sent false
34045 9717 Acked Read 9 Done Done Sent false
34046 9718 Acked Read 6 Done Done Sent false
34047 9719 Acked Read 4 Done Done Sent false
34048 9720 Acked Write 4 Done Done Sent false
34049 9721 Acked Write 4 Done Done Sent false
34050 9722 Acked Write 3 Done Done Sent false
34051 9723 Acked Read 4 Done Done Sent false
34052 9724 Acked Flush 0 Done Done Sent false
34053 9725 Acked Read 2 Done Done Sent false
34054 9726 Acked Flush 0 Done Done Sent false
34055 9727 Acked Write 8 Done Done Sent false
34056 9728 Acked Read 3 Done Done Sent false
34057 9729 Acked Read 6 Done Done Sent false
34058 9730 Acked Read 5 Done Done Sent false
34059 9731 Acked Read 2 Done Done Sent false
34060 9732 Acked Read 4 Done Done Sent false
34061 9733 Acked Read 7 Done Done Sent false
34062 9734 Acked Write 7 Done Done Sent false
34063 9735 Acked Read 9 Done Done Sent false
34064 9736 Acked Read 7 Done Done Sent false
34065 9737 Acked Write 2 Done Done Sent false
34066 9738 Acked Write 7 Done Done Sent false
34067 9739 Acked Read 9 Done Done Sent false
34068 9740 Acked Write 1 Done Done Sent false
34069 9741 Acked Write 6 Done Done Sent false
34070 9742 Acked Read 3 Done Done Sent false
34071 9743 Acked Read 9 Done Done Sent false
34072 9744 Acked Write 4 Done Done Sent false
34073 9745 Acked Read 1 Done Done Sent false
34074 9746 Acked Write 6 Done Done Sent false
34075 9747 Acked Write 1 Done Done Sent false
34076 9748 Acked Write 8 Done Done Sent false
34077 9749 Acked Read 2 Done Done Sent false
34078 9750 Acked Write 8 Done Done Sent false
34079 9751 Acked Read 2 Done Done Sent false
34080 9752 Acked Write 7 Done Done Sent false
34081 9753 Acked Read 9 Done Done Sent false
34082 STATES DS:0 DS:1 DS:2 TOTAL
34083 Sent 0 0 301 301
34084 Done 301 301 0 602
34085 Skipped 0 0 0 0
34086 Error 0 0 0 0
34087 Last Flush: Some(JobId(9726)) Some(JobId(9726)) Some(JobId(9452))
34088 Downstairs last five completed: 9452 9451 9450 9449 9448
34089 Upstairs last five completed: 9753 9752 9751 9750 9749
34090 CLIENT: Up:0 ds:301 act:2
340912025-03-17T22:42:05.006ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
340922025-03-17T22:42:06.007ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
340932025-03-17T22:42:07.009ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
340942025-03-17T22:42:08.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34095 ----------------------------------------------------------------
34096 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34097 States: Active Active Offline
34098 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34099 9453 Acked Write 5 Done Done Sent false
34100 9454 Acked Read 3 Done Done Sent false
34101 9455 Acked Read 6 Done Done Sent false
34102 9456 Acked Write 3 Done Done Sent false
34103 9457 Acked Read 4 Done Done Sent false
34104 9458 Acked Read 1 Done Done Sent false
34105 9459 Acked Write 6 Done Done Sent false
34106 9460 Acked Write 2 Done Done Sent false
34107 9461 Acked Write 1 Done Done Sent false
34108 9462 Acked Read 5 Done Done Sent false
34109 9463 Acked Write 1 Done Done Sent false
34110 9464 Acked Read 9 Done Done Sent false
34111 9465 Acked Write 4 Done Done Sent false
34112 9466 Acked Read 8 Done Done Sent false
34113 9467 Acked Read 7 Done Done Sent false
34114 9468 Acked Write 2 Done Done Sent false
34115 9469 Acked Read 4 Done Done Sent false
34116 9470 Acked Write 3 Done Done Sent false
34117 9471 Acked Read 3 Done Done Sent false
34118 9472 Acked Read 2 Done Done Sent false
34119 9473 Acked Write 10 Done Done Sent false
34120 9474 Acked Write 6 Done Done Sent false
34121 9475 Acked Read 6 Done Done Sent false
34122 9476 Acked Write 5 Done Done Sent false
34123 9477 Acked Read 2 Done Done Sent false
34124 9478 Acked Read 6 Done Done Sent false
34125 9479 Acked Read 8 Done Done Sent false
34126 9480 Acked Read 9 Done Done Sent false
34127 9481 Acked Read 6 Done Done Sent false
34128 9482 Acked Read 10 Done Done Sent false
34129 9483 Acked Write 10 Done Done Sent false
34130 9484 Acked Write 9 Done Done Sent false
34131 9485 Acked Read 9 Done Done Sent false
34132 9486 Acked Read 9 Done Done Sent false
34133 9487 Acked Write 8 Done Done Sent false
34134 9488 Acked Write 9 Done Done Sent false
34135 9489 Acked Read 8 Done Done Sent false
34136 9490 Acked Read 8 Done Done Sent false
34137 9491 Acked Write 3 Done Done Sent false
34138 9492 Acked Flush 0 Done Done Sent false
34139 9493 Acked Read 9 Done Done Sent false
34140 9494 Acked Write 8 Done Done Sent false
34141 9495 Acked Read 6 Done Done Sent false
34142 9496 Acked Write 1 Done Done Sent false
34143 9497 Acked Write 2 Done Done Sent false
34144 9498 Acked Write 3 Done Done Sent false
34145 9499 Acked Write 8 Done Done Sent false
34146 9500 Acked Read 10 Done Done Sent false
34147 9501 Acked Read 1 Done Done Sent false
34148 9502 Acked Write 7 Done Done Sent false
34149 9503 Acked Flush 0 Done Done Sent false
34150 9504 Acked Write 6 Done Done Sent false
34151 9505 Acked Read 1 Done Done Sent false
34152 9506 Acked Flush 0 Done Done Sent false
34153 9507 Acked Read 8 Done Done Sent false
34154 9508 Acked Read 6 Done Done Sent false
34155 9509 Acked Write 3 Done Done Sent false
34156 9510 Acked Read 3 Done Done Sent false
34157 9511 Acked Read 2 Done Done Sent false
34158 9512 Acked Write 8 Done Done Sent false
34159 9513 Acked Read 3 Done Done Sent false
34160 9514 Acked Write 3 Done Done Sent false
34161 9515 Acked Read 2 Done Done Sent false
34162 9516 Acked Write 7 Done Done Sent false
34163 9517 Acked Flush 0 Done Done Sent false
34164 9518 Acked Read 4 Done Done Sent false
34165 9519 Acked Write 2 Done Done Sent false
34166 9520 Acked Write 1 Done Done Sent false
34167 9521 Acked Write 10 Done Done Sent false
34168 9522 Acked Read 4 Done Done Sent false
34169 9523 Acked Flush 0 Done Done Sent false
34170 9524 Acked Write 9 Done Done Sent false
34171 9525 Acked Write 8 Done Done Sent false
34172 9526 Acked Read 9 Done Done Sent false
34173 9527 Acked Read 3 Done Done Sent false
34174 9528 Acked Write 6 Done Done Sent false
34175 9529 Acked Read 10 Done Done Sent false
34176 9530 Acked Read 6 Done Done Sent false
34177 9531 Acked Read 7 Done Done Sent false
34178 9532 Acked Write 10 Done Done Sent false
34179 9533 Acked Write 3 Done Done Sent false
34180 9534 Acked Write 7 Done Done Sent false
34181 9535 Acked Write 9 Done Done Sent false
34182 9536 Acked Read 5 Done Done Sent false
34183 9537 Acked Flush 0 Done Done Sent false
34184 9538 Acked Write 9 Done Done Sent false
34185 9539 Acked Flush 0 Done Done Sent false
34186 9540 Acked Read 7 Done Done Sent false
34187 9541 Acked Write 7 Done Done Sent false
34188 9542 Acked Write 9 Done Done Sent false
34189 9543 Acked Read 5 Done Done Sent false
34190 9544 Acked Read 8 Done Done Sent false
34191 9545 Acked Read 3 Done Done Sent false
34192 9546 Acked Read 9 Done Done Sent false
34193 9547 Acked Read 4 Done Done Sent false
34194 9548 Acked Read 4 Done Done Sent false
34195 9549 Acked Read 5 Done Done Sent false
34196 9550 Acked Write 5 Done Done Sent false
34197 9551 Acked Read 8 Done Done Sent false
34198 9552 Acked Write 10 Done Done Sent false
34199 9553 Acked Read 4 Done Done Sent false
34200 9554 Acked Write 6 Done Done Sent false
34201 9555 Acked Read 1 Done Done Sent false
34202 9556 Acked Write 1 Done Done Sent false
34203 9557 Acked Read 3 Done Done Sent false
34204 9558 Acked Read 6 Done Done Sent false
34205 9559 Acked Flush 0 Done Done Sent false
34206 9560 Acked Read 4 Done Done Sent false
34207 9561 Acked Write 1 Done Done Sent false
34208 9562 Acked Write 8 Done Done Sent false
34209 9563 Acked Read 5 Done Done Sent false
34210 9564 Acked Read 7 Done Done Sent false
34211 9565 Acked Write 2 Done Done Sent false
34212 9566 Acked Read 4 Done Done Sent false
34213 9567 Acked Flush 0 Done Done Sent false
34214 9568 Acked Read 3 Done Done Sent false
34215 9569 Acked Read 2 Done Done Sent false
34216 9570 Acked Write 7 Done Done Sent false
34217 9571 Acked Write 3 Done Done Sent false
34218 9572 Acked Write 3 Done Done Sent false
34219 9573 Acked Read 3 Done Done Sent false
34220 9574 Acked Read 10 Done Done Sent false
34221 9575 Acked Write 2 Done Done Sent false
34222 9576 Acked Read 10 Done Done Sent false
34223 9577 Acked Write 7 Done Done Sent false
34224 9578 Acked Read 7 Done Done Sent false
34225 9579 Acked Write 10 Done Done Sent false
34226 9580 Acked Read 7 Done Done Sent false
34227 9581 Acked Read 3 Done Done Sent false
34228 9582 Acked Write 6 Done Done Sent false
34229 9583 Acked Read 10 Done Done Sent false
34230 9584 Acked Write 1 Done Done Sent false
34231 9585 Acked Flush 0 Done Done Sent false
34232 9586 Acked Write 2 Done Done Sent false
34233 9587 Acked Read 9 Done Done Sent false
34234 9588 Acked Read 6 Done Done Sent false
34235 9589 Acked Read 3 Done Done Sent false
34236 9590 Acked Write 7 Done Done Sent false
34237 9591 Acked Read 1 Done Done Sent false
34238 9592 Acked Write 10 Done Done Sent false
34239 9593 Acked Read 2 Done Done Sent false
34240 9594 Acked Write 2 Done Done Sent false
34241 9595 Acked Write 5 Done Done Sent false
34242 9596 Acked Read 4 Done Done Sent false
34243 9597 Acked Write 2 Done Done Sent false
34244 9598 Acked Read 8 Done Done Sent false
34245 9599 Acked Write 2 Done Done Sent false
34246 9600 Acked Write 9 Done Done Sent false
34247 9601 Acked Write 8 Done Done Sent false
34248 9602 Acked Write 7 Done Done Sent false
34249 9603 Acked Flush 0 Done Done Sent false
34250 9604 Acked Write 4 Done Done Sent false
34251 9605 Acked Flush 0 Done Done Sent false
34252 9606 Acked Flush 0 Done Done Sent false
34253 9607 Acked Read 4 Done Done Sent false
34254 9608 Acked Write 4 Done Done Sent false
34255 9609 Acked Read 10 Done Done Sent false
34256 9610 Acked Read 7 Done Done Sent false
34257 9611 Acked Read 5 Done Done Sent false
34258 9612 Acked Write 9 Done Done Sent false
34259 9613 Acked Write 3 Done Done Sent false
34260 9614 Acked Write 3 Done Done Sent false
34261 9615 Acked Read 6 Done Done Sent false
34262 9616 Acked Write 4 Done Done Sent false
34263 9617 Acked Read 2 Done Done Sent false
34264 9618 Acked Write 1 Done Done Sent false
34265 9619 Acked Write 1 Done Done Sent false
34266 9620 Acked Read 6 Done Done Sent false
34267 9621 Acked Read 8 Done Done Sent false
34268 9622 Acked Flush 0 Done Done Sent false
34269 9623 Acked Read 1 Done Done Sent false
34270 9624 Acked Write 5 Done Done Sent false
34271 9625 Acked Flush 0 Done Done Sent false
34272 9626 Acked Read 6 Done Done Sent false
34273 9627 Acked Write 3 Done Done Sent false
34274 9628 Acked Read 1 Done Done Sent false
34275 9629 Acked Read 9 Done Done Sent false
34276 9630 Acked Read 10 Done Done Sent false
34277 9631 Acked Read 6 Done Done Sent false
34278 9632 Acked Read 3 Done Done Sent false
34279 9633 Acked Read 9 Done Done Sent false
34280 9634 Acked Write 10 Done Done Sent false
34281 9635 Acked Write 10 Done Done Sent false
34282 9636 Acked Write 3 Done Done Sent false
34283 9637 Acked Read 3 Done Done Sent false
34284 9638 Acked Read 4 Done Done Sent false
34285 9639 Acked Read 10 Done Done Sent false
34286 9640 Acked Write 5 Done Done Sent false
34287 9641 Acked Read 6 Done Done Sent false
34288 9642 Acked Write 3 Done Done Sent false
34289 9643 Acked Read 3 Done Done Sent false
34290 9644 Acked Read 10 Done Done Sent false
34291 9645 Acked Flush 0 Done Done Sent false
34292 9646 Acked Write 8 Done Done Sent false
34293 9647 Acked Write 7 Done Done Sent false
34294 9648 Acked Write 7 Done Done Sent false
34295 9649 Acked Read 3 Done Done Sent false
34296 9650 Acked Read 9 Done Done Sent false
34297 9651 Acked Read 7 Done Done Sent false
34298 9652 Acked Read 8 Done Done Sent false
34299 9653 Acked Read 7 Done Done Sent false
34300 9654 Acked Read 2 Done Done Sent false
34301 9655 Acked Read 8 Done Done Sent false
34302 9656 Acked Flush 0 Done Done Sent false
34303 9657 Acked Read 5 Done Done Sent false
34304 9658 Acked Write 1 Done Done Sent false
34305 9659 Acked Read 2 Done Done Sent false
34306 9660 Acked Write 8 Done Done Sent false
34307 9661 Acked Write 2 Done Done Sent false
34308 9662 Acked Write 1 Done Done Sent false
34309 9663 Acked Flush 0 Done Done Sent false
34310 9664 Acked Read 1 Done Done Sent false
34311 9665 Acked Read 10 Done Done Sent false
34312 9666 Acked Write 1 Done Done Sent false
34313 9667 Acked Write 5 Done Done Sent false
34314 9668 Acked Read 1 Done Done Sent false
34315 9669 Acked Write 10 Done Done Sent false
34316 9670 Acked Write 2 Done Done Sent false
34317 9671 Acked Read 4 Done Done Sent false
34318 9672 Acked Write 1 Done Done Sent false
34319 9673 Acked Read 5 Done Done Sent false
34320 9674 Acked Flush 0 Done Done Sent false
34321 9675 Acked Write 8 Done Done Sent false
34322 9676 Acked Write 1 Done Done Sent false
34323 9677 Acked Read 7 Done Done Sent false
34324 9678 Acked Write 3 Done Done Sent false
34325 9679 Acked Read 2 Done Done Sent false
34326 9680 Acked Read 7 Done Done Sent false
34327 9681 Acked Read 1 Done Done Sent false
34328 9682 Acked Flush 0 Done Done Sent false
34329 9683 Acked Write 2 Done Done Sent false
34330 9684 Acked Read 10 Done Done Sent false
34331 9685 Acked Read 6 Done Done Sent false
34332 9686 Acked Write 8 Done Done Sent false
34333 9687 Acked Read 2 Done Done Sent false
34334 9688 Acked Write 6 Done Done Sent false
34335 9689 Acked Read 4 Done Done Sent false
34336 9690 Acked Read 8 Done Done Sent false
34337 9691 Acked Read 9 Done Done Sent false
34338 9692 Acked Flush 0 Done Done Sent false
34339 9693 Acked Flush 0 Done Done Sent false
34340 9694 Acked Write 10 Done Done Sent false
34341 9695 Acked Flush 0 Done Done Sent false
34342 9696 Acked Write 8 Done Done Sent false
34343 9697 Acked Flush 0 Done Done Sent false
34344 9698 Acked Read 8 Done Done Sent false
34345 9699 Acked Read 5 Done Done Sent false
34346 9700 Acked Write 2 Done Done Sent false
34347 9701 Acked Write 5 Done Done Sent false
34348 9702 Acked Read 4 Done Done Sent false
34349 9703 Acked Read 10 Done Done Sent false
34350 9704 Acked Write 6 Done Done Sent false
34351 9705 Acked Write 10 Done Done Sent false
34352 9706 Acked Read 7 Done Done Sent false
34353 9707 Acked Read 9 Done Done Sent false
34354 9708 Acked Flush 0 Done Done Sent false
34355 9709 Acked Write 8 Done Done Sent false
34356 9710 Acked Read 9 Done Done Sent false
34357 9711 Acked Flush 0 Done Done Sent false
34358 9712 Acked Read 9 Done Done Sent false
34359 9713 Acked Read 7 Done Done Sent false
34360 9714 Acked Write 4 Done Done Sent false
34361 9715 Acked Write 7 Done Done Sent false
34362 9716 Acked Write 5 Done Done Sent false
34363 9717 Acked Read 9 Done Done Sent false
34364 9718 Acked Read 6 Done Done Sent false
34365 9719 Acked Read 4 Done Done Sent false
34366 9720 Acked Write 4 Done Done Sent false
34367 9721 Acked Write 4 Done Done Sent false
34368 9722 Acked Write 3 Done Done Sent false
34369 9723 Acked Read 4 Done Done Sent false
34370 9724 Acked Flush 0 Done Done Sent false
34371 9725 Acked Read 2 Done Done Sent false
34372 9726 Acked Flush 0 Done Done Sent false
34373 9727 Acked Write 8 Done Done Sent false
34374 9728 Acked Read 3 Done Done Sent false
34375 9729 Acked Read 6 Done Done Sent false
34376 9730 Acked Read 5 Done Done Sent false
34377 9731 Acked Read 2 Done Done Sent false
34378 9732 Acked Read 4 Done Done Sent false
34379 9733 Acked Read 7 Done Done Sent false
34380 9734 Acked Write 7 Done Done Sent false
34381 9735 Acked Read 9 Done Done Sent false
34382 9736 Acked Read 7 Done Done Sent false
34383 9737 Acked Write 2 Done Done Sent false
34384 9738 Acked Write 7 Done Done Sent false
34385 9739 Acked Read 9 Done Done Sent false
34386 9740 Acked Write 1 Done Done Sent false
34387 9741 Acked Write 6 Done Done Sent false
34388 9742 Acked Read 3 Done Done Sent false
34389 9743 Acked Read 9 Done Done Sent false
34390 9744 Acked Write 4 Done Done Sent false
34391 9745 Acked Read 1 Done Done Sent false
34392 9746 Acked Write 6 Done Done Sent false
34393 9747 Acked Write 1 Done Done Sent false
34394 9748 Acked Write 8 Done Done Sent false
34395 9749 Acked Read 2 Done Done Sent false
34396 9750 Acked Write 8 Done Done Sent false
34397 9751 Acked Read 2 Done Done Sent false
34398 9752 Acked Write 7 Done Done Sent false
34399 9753 Acked Read 9 Done Done Sent false
34400 STATES DS:0 DS:1 DS:2 TOTAL
34401 Sent 0 0 301 301
34402 Done 301 301 0 602
34403 Skipped 0 0 0 0
34404 Error 0 0 0 0
34405 Last Flush: Some(JobId(9726)) Some(JobId(9726)) Some(JobId(9452))
34406 Downstairs last five completed: 9452 9451 9450 9449 9448
34407 Upstairs last five completed: 9753 9752 9751 9750 9749
34408 CLIENT: Up:0 ds:301 act:2
344092025-03-17T22:42:09.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344102025-03-17T22:42:09.985ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344112025-03-17T22:42:09.985ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344122025-03-17T22:42:09.985ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344132025-03-17T22:42:09.985ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344142025-03-17T22:42:09.985ZINFOcrucible: client task is exiting = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34415 [8830] got start action
34416 [8830] I am going to restart
34417 Starting downstairs at port 8830
34418 Make output file at "/tmp/dsc/downstairs-8830.txt"
34419 [8830][2] reports Starting
34420 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1237)
34421 [8830][2] reports Running
344222025-03-17T22:42:10.485ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344232025-03-17T22:42:11.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
344242025-03-17T22:42:12.017ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34425 ----------------------------------------------------------------
34426 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34427 States: Active Active Offline
34428 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34429 9453 Acked Write 5 Done Done Sent false
34430 9454 Acked Read 3 Done Done Sent false
34431 9455 Acked Read 6 Done Done Sent false
34432 9456 Acked Write 3 Done Done Sent false
34433 9457 Acked Read 4 Done Done Sent false
34434 9458 Acked Read 1 Done Done Sent false
34435 9459 Acked Write 6 Done Done Sent false
34436 9460 Acked Write 2 Done Done Sent false
34437 9461 Acked Write 1 Done Done Sent false
34438 9462 Acked Read 5 Done Done Sent false
34439 9463 Acked Write 1 Done Done Sent false
34440 9464 Acked Read 9 Done Done Sent false
34441 9465 Acked Write 4 Done Done Sent false
34442 9466 Acked Read 8 Done Done Sent false
34443 9467 Acked Read 7 Done Done Sent false
34444 9468 Acked Write 2 Done Done Sent false
34445 9469 Acked Read 4 Done Done Sent false
34446 9470 Acked Write 3 Done Done Sent false
34447 9471 Acked Read 3 Done Done Sent false
34448 9472 Acked Read 2 Done Done Sent false
34449 9473 Acked Write 10 Done Done Sent false
34450 9474 Acked Write 6 Done Done Sent false
34451 9475 Acked Read 6 Done Done Sent false
34452 9476 Acked Write 5 Done Done Sent false
34453 9477 Acked Read 2 Done Done Sent false
34454 9478 Acked Read 6 Done Done Sent false
34455 9479 Acked Read 8 Done Done Sent false
34456 9480 Acked Read 9 Done Done Sent false
34457 9481 Acked Read 6 Done Done Sent false
34458 9482 Acked Read 10 Done Done Sent false
34459 9483 Acked Write 10 Done Done Sent false
34460 9484 Acked Write 9 Done Done Sent false
34461 9485 Acked Read 9 Done Done Sent false
34462 9486 Acked Read 9 Done Done Sent false
34463 9487 Acked Write 8 Done Done Sent false
34464 9488 Acked Write 9 Done Done Sent false
34465 9489 Acked Read 8 Done Done Sent false
34466 9490 Acked Read 8 Done Done Sent false
34467 9491 Acked Write 3 Done Done Sent false
34468 9492 Acked Flush 0 Done Done Sent false
34469 9493 Acked Read 9 Done Done Sent false
34470 9494 Acked Write 8 Done Done Sent false
34471 9495 Acked Read 6 Done Done Sent false
34472 9496 Acked Write 1 Done Done Sent false
34473 9497 Acked Write 2 Done Done Sent false
34474 9498 Acked Write 3 Done Done Sent false
34475 9499 Acked Write 8 Done Done Sent false
34476 9500 Acked Read 10 Done Done Sent false
34477 9501 Acked Read 1 Done Done Sent false
34478 9502 Acked Write 7 Done Done Sent false
34479 9503 Acked Flush 0 Done Done Sent false
34480 9504 Acked Write 6 Done Done Sent false
34481 9505 Acked Read 1 Done Done Sent false
34482 9506 Acked Flush 0 Done Done Sent false
34483 9507 Acked Read 8 Done Done Sent false
34484 9508 Acked Read 6 Done Done Sent false
34485 9509 Acked Write 3 Done Done Sent false
34486 9510 Acked Read 3 Done Done Sent false
34487 9511 Acked Read 2 Done Done Sent false
34488 9512 Acked Write 8 Done Done Sent false
34489 9513 Acked Read 3 Done Done Sent false
34490 9514 Acked Write 3 Done Done Sent false
34491 9515 Acked Read 2 Done Done Sent false
34492 9516 Acked Write 7 Done Done Sent false
34493 9517 Acked Flush 0 Done Done Sent false
34494 9518 Acked Read 4 Done Done Sent false
34495 9519 Acked Write 2 Done Done Sent false
34496 9520 Acked Write 1 Done Done Sent false
34497 9521 Acked Write 10 Done Done Sent false
34498 9522 Acked Read 4 Done Done Sent false
34499 9523 Acked Flush 0 Done Done Sent false
34500 9524 Acked Write 9 Done Done Sent false
34501 9525 Acked Write 8 Done Done Sent false
34502 9526 Acked Read 9 Done Done Sent false
34503 9527 Acked Read 3 Done Done Sent false
34504 9528 Acked Write 6 Done Done Sent false
34505 9529 Acked Read 10 Done Done Sent false
34506 9530 Acked Read 6 Done Done Sent false
34507 9531 Acked Read 7 Done Done Sent false
34508 9532 Acked Write 10 Done Done Sent false
34509 9533 Acked Write 3 Done Done Sent false
34510 9534 Acked Write 7 Done Done Sent false
34511 9535 Acked Write 9 Done Done Sent false
34512 9536 Acked Read 5 Done Done Sent false
34513 9537 Acked Flush 0 Done Done Sent false
34514 9538 Acked Write 9 Done Done Sent false
34515 9539 Acked Flush 0 Done Done Sent false
34516 9540 Acked Read 7 Done Done Sent false
34517 9541 Acked Write 7 Done Done Sent false
34518 9542 Acked Write 9 Done Done Sent false
34519 9543 Acked Read 5 Done Done Sent false
34520 9544 Acked Read 8 Done Done Sent false
34521 9545 Acked Read 3 Done Done Sent false
34522 9546 Acked Read 9 Done Done Sent false
34523 9547 Acked Read 4 Done Done Sent false
34524 9548 Acked Read 4 Done Done Sent false
34525 9549 Acked Read 5 Done Done Sent false
34526 9550 Acked Write 5 Done Done Sent false
34527 9551 Acked Read 8 Done Done Sent false
34528 9552 Acked Write 10 Done Done Sent false
34529 9553 Acked Read 4 Done Done Sent false
34530 9554 Acked Write 6 Done Done Sent false
34531 9555 Acked Read 1 Done Done Sent false
34532 9556 Acked Write 1 Done Done Sent false
34533 9557 Acked Read 3 Done Done Sent false
34534 9558 Acked Read 6 Done Done Sent false
34535 9559 Acked Flush 0 Done Done Sent false
34536 9560 Acked Read 4 Done Done Sent false
34537 9561 Acked Write 1 Done Done Sent false
34538 9562 Acked Write 8 Done Done Sent false
34539 9563 Acked Read 5 Done Done Sent false
34540 9564 Acked Read 7 Done Done Sent false
34541 9565 Acked Write 2 Done Done Sent false
34542 9566 Acked Read 4 Done Done Sent false
34543 9567 Acked Flush 0 Done Done Sent false
34544 9568 Acked Read 3 Done Done Sent false
34545 9569 Acked Read 2 Done Done Sent false
34546 9570 Acked Write 7 Done Done Sent false
34547 9571 Acked Write 3 Done Done Sent false
34548 9572 Acked Write 3 Done Done Sent false
34549 9573 Acked Read 3 Done Done Sent false
34550 9574 Acked Read 10 Done Done Sent false
34551 9575 Acked Write 2 Done Done Sent false
34552 9576 Acked Read 10 Done Done Sent false
34553 9577 Acked Write 7 Done Done Sent false
34554 9578 Acked Read 7 Done Done Sent false
34555 9579 Acked Write 10 Done Done Sent false
34556 9580 Acked Read 7 Done Done Sent false
34557 9581 Acked Read 3 Done Done Sent false
34558 9582 Acked Write 6 Done Done Sent false
34559 9583 Acked Read 10 Done Done Sent false
34560 9584 Acked Write 1 Done Done Sent false
34561 9585 Acked Flush 0 Done Done Sent false
34562 9586 Acked Write 2 Done Done Sent false
34563 9587 Acked Read 9 Done Done Sent false
34564 9588 Acked Read 6 Done Done Sent false
34565 9589 Acked Read 3 Done Done Sent false
34566 9590 Acked Write 7 Done Done Sent false
34567 9591 Acked Read 1 Done Done Sent false
34568 9592 Acked Write 10 Done Done Sent false
34569 9593 Acked Read 2 Done Done Sent false
34570 9594 Acked Write 2 Done Done Sent false
34571 9595 Acked Write 5 Done Done Sent false
34572 9596 Acked Read 4 Done Done Sent false
34573 9597 Acked Write 2 Done Done Sent false
34574 9598 Acked Read 8 Done Done Sent false
34575 9599 Acked Write 2 Done Done Sent false
34576 9600 Acked Write 9 Done Done Sent false
34577 9601 Acked Write 8 Done Done Sent false
34578 9602 Acked Write 7 Done Done Sent false
34579 9603 Acked Flush 0 Done Done Sent false
34580 9604 Acked Write 4 Done Done Sent false
34581 9605 Acked Flush 0 Done Done Sent false
34582 9606 Acked Flush 0 Done Done Sent false
34583 9607 Acked Read 4 Done Done Sent false
34584 9608 Acked Write 4 Done Done Sent false
34585 9609 Acked Read 10 Done Done Sent false
34586 9610 Acked Read 7 Done Done Sent false
34587 9611 Acked Read 5 Done Done Sent false
34588 9612 Acked Write 9 Done Done Sent false
34589 9613 Acked Write 3 Done Done Sent false
34590 9614 Acked Write 3 Done Done Sent false
34591 9615 Acked Read 6 Done Done Sent false
34592 9616 Acked Write 4 Done Done Sent false
34593 9617 Acked Read 2 Done Done Sent false
34594 9618 Acked Write 1 Done Done Sent false
34595 9619 Acked Write 1 Done Done Sent false
34596 9620 Acked Read 6 Done Done Sent false
34597 9621 Acked Read 8 Done Done Sent false
34598 9622 Acked Flush 0 Done Done Sent false
34599 9623 Acked Read 1 Done Done Sent false
34600 9624 Acked Write 5 Done Done Sent false
34601 9625 Acked Flush 0 Done Done Sent false
34602 9626 Acked Read 6 Done Done Sent false
34603 9627 Acked Write 3 Done Done Sent false
34604 9628 Acked Read 1 Done Done Sent false
34605 9629 Acked Read 9 Done Done Sent false
34606 9630 Acked Read 10 Done Done Sent false
34607 9631 Acked Read 6 Done Done Sent false
34608 9632 Acked Read 3 Done Done Sent false
34609 9633 Acked Read 9 Done Done Sent false
34610 9634 Acked Write 10 Done Done Sent false
34611 9635 Acked Write 10 Done Done Sent false
34612 9636 Acked Write 3 Done Done Sent false
34613 9637 Acked Read 3 Done Done Sent false
34614 9638 Acked Read 4 Done Done Sent false
34615 9639 Acked Read 10 Done Done Sent false
34616 9640 Acked Write 5 Done Done Sent false
34617 9641 Acked Read 6 Done Done Sent false
34618 9642 Acked Write 3 Done Done Sent false
34619 9643 Acked Read 3 Done Done Sent false
34620 9644 Acked Read 10 Done Done Sent false
34621 9645 Acked Flush 0 Done Done Sent false
34622 9646 Acked Write 8 Done Done Sent false
34623 9647 Acked Write 7 Done Done Sent false
34624 9648 Acked Write 7 Done Done Sent false
34625 9649 Acked Read 3 Done Done Sent false
34626 9650 Acked Read 9 Done Done Sent false
34627 9651 Acked Read 7 Done Done Sent false
34628 9652 Acked Read 8 Done Done Sent false
34629 9653 Acked Read 7 Done Done Sent false
34630 9654 Acked Read 2 Done Done Sent false
34631 9655 Acked Read 8 Done Done Sent false
34632 9656 Acked Flush 0 Done Done Sent false
34633 9657 Acked Read 5 Done Done Sent false
34634 9658 Acked Write 1 Done Done Sent false
34635 9659 Acked Read 2 Done Done Sent false
34636 9660 Acked Write 8 Done Done Sent false
34637 9661 Acked Write 2 Done Done Sent false
34638 9662 Acked Write 1 Done Done Sent false
34639 9663 Acked Flush 0 Done Done Sent false
34640 9664 Acked Read 1 Done Done Sent false
34641 9665 Acked Read 10 Done Done Sent false
34642 9666 Acked Write 1 Done Done Sent false
34643 9667 Acked Write 5 Done Done Sent false
34644 9668 Acked Read 1 Done Done Sent false
34645 9669 Acked Write 10 Done Done Sent false
34646 9670 Acked Write 2 Done Done Sent false
34647 9671 Acked Read 4 Done Done Sent false
34648 9672 Acked Write 1 Done Done Sent false
34649 9673 Acked Read 5 Done Done Sent false
34650 9674 Acked Flush 0 Done Done Sent false
34651 9675 Acked Write 8 Done Done Sent false
34652 9676 Acked Write 1 Done Done Sent false
34653 9677 Acked Read 7 Done Done Sent false
34654 9678 Acked Write 3 Done Done Sent false
34655 9679 Acked Read 2 Done Done Sent false
34656 9680 Acked Read 7 Done Done Sent false
34657 9681 Acked Read 1 Done Done Sent false
34658 9682 Acked Flush 0 Done Done Sent false
34659 9683 Acked Write 2 Done Done Sent false
34660 9684 Acked Read 10 Done Done Sent false
34661 9685 Acked Read 6 Done Done Sent false
34662 9686 Acked Write 8 Done Done Sent false
34663 9687 Acked Read 2 Done Done Sent false
34664 9688 Acked Write 6 Done Done Sent false
34665 9689 Acked Read 4 Done Done Sent false
34666 9690 Acked Read 8 Done Done Sent false
34667 9691 Acked Read 9 Done Done Sent false
34668 9692 Acked Flush 0 Done Done Sent false
34669 9693 Acked Flush 0 Done Done Sent false
34670 9694 Acked Write 10 Done Done Sent false
34671 9695 Acked Flush 0 Done Done Sent false
34672 9696 Acked Write 8 Done Done Sent false
34673 9697 Acked Flush 0 Done Done Sent false
34674 9698 Acked Read 8 Done Done Sent false
34675 9699 Acked Read 5 Done Done Sent false
34676 9700 Acked Write 2 Done Done Sent false
34677 9701 Acked Write 5 Done Done Sent false
34678 9702 Acked Read 4 Done Done Sent false
34679 9703 Acked Read 10 Done Done Sent false
34680 9704 Acked Write 6 Done Done Sent false
34681 9705 Acked Write 10 Done Done Sent false
34682 9706 Acked Read 7 Done Done Sent false
34683 9707 Acked Read 9 Done Done Sent false
34684 9708 Acked Flush 0 Done Done Sent false
34685 9709 Acked Write 8 Done Done Sent false
34686 9710 Acked Read 9 Done Done Sent false
34687 9711 Acked Flush 0 Done Done Sent false
34688 9712 Acked Read 9 Done Done Sent false
34689 9713 Acked Read 7 Done Done Sent false
34690 9714 Acked Write 4 Done Done Sent false
34691 9715 Acked Write 7 Done Done Sent false
34692 9716 Acked Write 5 Done Done Sent false
34693 9717 Acked Read 9 Done Done Sent false
34694 9718 Acked Read 6 Done Done Sent false
34695 9719 Acked Read 4 Done Done Sent false
34696 9720 Acked Write 4 Done Done Sent false
34697 9721 Acked Write 4 Done Done Sent false
34698 9722 Acked Write 3 Done Done Sent false
34699 9723 Acked Read 4 Done Done Sent false
34700 9724 Acked Flush 0 Done Done Sent false
34701 9725 Acked Read 2 Done Done Sent false
34702 9726 Acked Flush 0 Done Done Sent false
34703 9727 Acked Write 8 Done Done Sent false
34704 9728 Acked Read 3 Done Done Sent false
34705 9729 Acked Read 6 Done Done Sent false
34706 9730 Acked Read 5 Done Done Sent false
34707 9731 Acked Read 2 Done Done Sent false
34708 9732 Acked Read 4 Done Done Sent false
34709 9733 Acked Read 7 Done Done Sent false
34710 9734 Acked Write 7 Done Done Sent false
34711 9735 Acked Read 9 Done Done Sent false
34712 9736 Acked Read 7 Done Done Sent false
34713 9737 Acked Write 2 Done Done Sent false
34714 9738 Acked Write 7 Done Done Sent false
34715 9739 Acked Read 9 Done Done Sent false
34716 9740 Acked Write 1 Done Done Sent false
34717 9741 Acked Write 6 Done Done Sent false
34718 9742 Acked Read 3 Done Done Sent false
34719 9743 Acked Read 9 Done Done Sent false
34720 9744 Acked Write 4 Done Done Sent false
34721 9745 Acked Read 1 Done Done Sent false
34722 9746 Acked Write 6 Done Done Sent false
34723 9747 Acked Write 1 Done Done Sent false
34724 9748 Acked Write 8 Done Done Sent false
34725 9749 Acked Read 2 Done Done Sent false
34726 9750 Acked Write 8 Done Done Sent false
34727 9751 Acked Read 2 Done Done Sent false
34728 9752 Acked Write 7 Done Done Sent false
34729 9753 Acked Read 9 Done Done Sent false
34730 STATES DS:0 DS:1 DS:2 TOTAL
34731 Sent 0 0 301 301
34732 Done 301 301 0 602
34733 Skipped 0 0 0 0
34734 Error 0 0 0 0
34735 Last Flush: Some(JobId(9726)) Some(JobId(9726)) Some(JobId(9452))
34736 Downstairs last five completed: 9452 9451 9450 9449 9448
34737 Upstairs last five completed: 9753 9752 9751 9750 9749
34738 CLIENT: Up:0 ds:301 act:2
347392025-03-17T22:42:13.223ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
347402025-03-17T22:42:14.021ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
347412025-03-17T22:42:15.023ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
347422025-03-17T22:42:16.025ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
34743 ----------------------------------------------------------------
34744 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34745 States: Active Active Offline
34746 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34747 9453 Acked Write 5 Done Done Sent false
34748 9454 Acked Read 3 Done Done Sent false
34749 9455 Acked Read 6 Done Done Sent false
34750 9456 Acked Write 3 Done Done Sent false
34751 9457 Acked Read 4 Done Done Sent false
34752 9458 Acked Read 1 Done Done Sent false
34753 9459 Acked Write 6 Done Done Sent false
34754 9460 Acked Write 2 Done Done Sent false
34755 9461 Acked Write 1 Done Done Sent false
34756 9462 Acked Read 5 Done Done Sent false
34757 9463 Acked Write 1 Done Done Sent false
34758 9464 Acked Read 9 Done Done Sent false
34759 9465 Acked Write 4 Done Done Sent false
34760 9466 Acked Read 8 Done Done Sent false
34761 9467 Acked Read 7 Done Done Sent false
34762 9468 Acked Write 2 Done Done Sent false
34763 9469 Acked Read 4 Done Done Sent false
34764 9470 Acked Write 3 Done Done Sent false
34765 9471 Acked Read 3 Done Done Sent false
34766 9472 Acked Read 2 Done Done Sent false
34767 9473 Acked Write 10 Done Done Sent false
34768 9474 Acked Write 6 Done Done Sent false
34769 9475 Acked Read 6 Done Done Sent false
34770 9476 Acked Write 5 Done Done Sent false
34771 9477 Acked Read 2 Done Done Sent false
34772 9478 Acked Read 6 Done Done Sent false
34773 9479 Acked Read 8 Done Done Sent false
34774 9480 Acked Read 9 Done Done Sent false
34775 9481 Acked Read 6 Done Done Sent false
34776 9482 Acked Read 10 Done Done Sent false
34777 9483 Acked Write 10 Done Done Sent false
34778 9484 Acked Write 9 Done Done Sent false
34779 9485 Acked Read 9 Done Done Sent false
34780 9486 Acked Read 9 Done Done Sent false
34781 9487 Acked Write 8 Done Done Sent false
34782 9488 Acked Write 9 Done Done Sent false
34783 9489 Acked Read 8 Done Done Sent false
34784 9490 Acked Read 8 Done Done Sent false
34785 9491 Acked Write 3 Done Done Sent false
34786 9492 Acked Flush 0 Done Done Sent false
34787 9493 Acked Read 9 Done Done Sent false
34788 9494 Acked Write 8 Done Done Sent false
34789 9495 Acked Read 6 Done Done Sent false
34790 9496 Acked Write 1 Done Done Sent false
34791 9497 Acked Write 2 Done Done Sent false
34792 9498 Acked Write 3 Done Done Sent false
34793 9499 Acked Write 8 Done Done Sent false
34794 9500 Acked Read 10 Done Done Sent false
34795 9501 Acked Read 1 Done Done Sent false
34796 9502 Acked Write 7 Done Done Sent false
34797 9503 Acked Flush 0 Done Done Sent false
34798 9504 Acked Write 6 Done Done Sent false
34799 9505 Acked Read 1 Done Done Sent false
34800 9506 Acked Flush 0 Done Done Sent false
34801 9507 Acked Read 8 Done Done Sent false
34802 9508 Acked Read 6 Done Done Sent false
34803 9509 Acked Write 3 Done Done Sent false
34804 9510 Acked Read 3 Done Done Sent false
34805 9511 Acked Read 2 Done Done Sent false
34806 9512 Acked Write 8 Done Done Sent false
34807 9513 Acked Read 3 Done Done Sent false
34808 9514 Acked Write 3 Done Done Sent false
34809 9515 Acked Read 2 Done Done Sent false
34810 9516 Acked Write 7 Done Done Sent false
34811 9517 Acked Flush 0 Done Done Sent false
34812 9518 Acked Read 4 Done Done Sent false
34813 9519 Acked Write 2 Done Done Sent false
34814 9520 Acked Write 1 Done Done Sent false
34815 9521 Acked Write 10 Done Done Sent false
34816 9522 Acked Read 4 Done Done Sent false
34817 9523 Acked Flush 0 Done Done Sent false
34818 9524 Acked Write 9 Done Done Sent false
34819 9525 Acked Write 8 Done Done Sent false
34820 9526 Acked Read 9 Done Done Sent false
34821 9527 Acked Read 3 Done Done Sent false
34822 9528 Acked Write 6 Done Done Sent false
34823 9529 Acked Read 10 Done Done Sent false
34824 9530 Acked Read 6 Done Done Sent false
34825 9531 Acked Read 7 Done Done Sent false
34826 9532 Acked Write 10 Done Done Sent false
34827 9533 Acked Write 3 Done Done Sent false
34828 9534 Acked Write 7 Done Done Sent false
34829 9535 Acked Write 9 Done Done Sent false
34830 9536 Acked Read 5 Done Done Sent false
34831 9537 Acked Flush 0 Done Done Sent false
34832 9538 Acked Write 9 Done Done Sent false
34833 9539 Acked Flush 0 Done Done Sent false
34834 9540 Acked Read 7 Done Done Sent false
34835 9541 Acked Write 7 Done Done Sent false
34836 9542 Acked Write 9 Done Done Sent false
34837 9543 Acked Read 5 Done Done Sent false
34838 9544 Acked Read 8 Done Done Sent false
34839 9545 Acked Read 3 Done Done Sent false
34840 9546 Acked Read 9 Done Done Sent false
34841 9547 Acked Read 4 Done Done Sent false
34842 9548 Acked Read 4 Done Done Sent false
34843 9549 Acked Read 5 Done Done Sent false
34844 9550 Acked Write 5 Done Done Sent false
34845 9551 Acked Read 8 Done Done Sent false
34846 9552 Acked Write 10 Done Done Sent false
34847 9553 Acked Read 4 Done Done Sent false
34848 9554 Acked Write 6 Done Done Sent false
34849 9555 Acked Read 1 Done Done Sent false
34850 9556 Acked Write 1 Done Done Sent false
34851 9557 Acked Read 3 Done Done Sent false
34852 9558 Acked Read 6 Done Done Sent false
34853 9559 Acked Flush 0 Done Done Sent false
34854 9560 Acked Read 4 Done Done Sent false
34855 9561 Acked Write 1 Done Done Sent false
34856 9562 Acked Write 8 Done Done Sent false
34857 9563 Acked Read 5 Done Done Sent false
34858 9564 Acked Read 7 Done Done Sent false
34859 9565 Acked Write 2 Done Done Sent false
34860 9566 Acked Read 4 Done Done Sent false
34861 9567 Acked Flush 0 Done Done Sent false
34862 9568 Acked Read 3 Done Done Sent false
34863 9569 Acked Read 2 Done Done Sent false
34864 9570 Acked Write 7 Done Done Sent false
34865 9571 Acked Write 3 Done Done Sent false
34866 9572 Acked Write 3 Done Done Sent false
34867 9573 Acked Read 3 Done Done Sent false
34868 9574 Acked Read 10 Done Done Sent false
34869 9575 Acked Write 2 Done Done Sent false
34870 9576 Acked Read 10 Done Done Sent false
34871 9577 Acked Write 7 Done Done Sent false
34872 9578 Acked Read 7 Done Done Sent false
34873 9579 Acked Write 10 Done Done Sent false
34874 9580 Acked Read 7 Done Done Sent false
34875 9581 Acked Read 3 Done Done Sent false
34876 9582 Acked Write 6 Done Done Sent false
34877 9583 Acked Read 10 Done Done Sent false
34878 9584 Acked Write 1 Done Done Sent false
34879 9585 Acked Flush 0 Done Done Sent false
34880 9586 Acked Write 2 Done Done Sent false
34881 9587 Acked Read 9 Done Done Sent false
34882 9588 Acked Read 6 Done Done Sent false
34883 9589 Acked Read 3 Done Done Sent false
34884 9590 Acked Write 7 Done Done Sent false
34885 9591 Acked Read 1 Done Done Sent false
34886 9592 Acked Write 10 Done Done Sent false
34887 9593 Acked Read 2 Done Done Sent false
34888 9594 Acked Write 2 Done Done Sent false
34889 9595 Acked Write 5 Done Done Sent false
34890 9596 Acked Read 4 Done Done Sent false
34891 9597 Acked Write 2 Done Done Sent false
34892 9598 Acked Read 8 Done Done Sent false
34893 9599 Acked Write 2 Done Done Sent false
34894 9600 Acked Write 9 Done Done Sent false
34895 9601 Acked Write 8 Done Done Sent false
34896 9602 Acked Write 7 Done Done Sent false
34897 9603 Acked Flush 0 Done Done Sent false
34898 9604 Acked Write 4 Done Done Sent false
34899 9605 Acked Flush 0 Done Done Sent false
34900 9606 Acked Flush 0 Done Done Sent false
34901 9607 Acked Read 4 Done Done Sent false
34902 9608 Acked Write 4 Done Done Sent false
34903 9609 Acked Read 10 Done Done Sent false
34904 9610 Acked Read 7 Done Done Sent false
34905 9611 Acked Read 5 Done Done Sent false
34906 9612 Acked Write 9 Done Done Sent false
34907 9613 Acked Write 3 Done Done Sent false
34908 9614 Acked Write 3 Done Done Sent false
34909 9615 Acked Read 6 Done Done Sent false
34910 9616 Acked Write 4 Done Done Sent false
34911 9617 Acked Read 2 Done Done Sent false
34912 9618 Acked Write 1 Done Done Sent false
34913 9619 Acked Write 1 Done Done Sent false
34914 9620 Acked Read 6 Done Done Sent false
34915 9621 Acked Read 8 Done Done Sent false
34916 9622 Acked Flush 0 Done Done Sent false
34917 9623 Acked Read 1 Done Done Sent false
34918 9624 Acked Write 5 Done Done Sent false
34919 9625 Acked Flush 0 Done Done Sent false
34920 9626 Acked Read 6 Done Done Sent false
34921 9627 Acked Write 3 Done Done Sent false
34922 9628 Acked Read 1 Done Done Sent false
34923 9629 Acked Read 9 Done Done Sent false
34924 9630 Acked Read 10 Done Done Sent false
34925 9631 Acked Read 6 Done Done Sent false
34926 9632 Acked Read 3 Done Done Sent false
34927 9633 Acked Read 9 Done Done Sent false
34928 9634 Acked Write 10 Done Done Sent false
34929 9635 Acked Write 10 Done Done Sent false
34930 9636 Acked Write 3 Done Done Sent false
34931 9637 Acked Read 3 Done Done Sent false
34932 9638 Acked Read 4 Done Done Sent false
34933 9639 Acked Read 10 Done Done Sent false
34934 9640 Acked Write 5 Done Done Sent false
34935 9641 Acked Read 6 Done Done Sent false
34936 9642 Acked Write 3 Done Done Sent false
34937 9643 Acked Read 3 Done Done Sent false
34938 9644 Acked Read 10 Done Done Sent false
34939 9645 Acked Flush 0 Done Done Sent false
34940 9646 Acked Write 8 Done Done Sent false
34941 9647 Acked Write 7 Done Done Sent false
34942 9648 Acked Write 7 Done Done Sent false
34943 9649 Acked Read 3 Done Done Sent false
34944 9650 Acked Read 9 Done Done Sent false
34945 9651 Acked Read 7 Done Done Sent false
34946 9652 Acked Read 8 Done Done Sent false
34947 9653 Acked Read 7 Done Done Sent false
34948 9654 Acked Read 2 Done Done Sent false
34949 9655 Acked Read 8 Done Done Sent false
34950 9656 Acked Flush 0 Done Done Sent false
34951 9657 Acked Read 5 Done Done Sent false
34952 9658 Acked Write 1 Done Done Sent false
34953 9659 Acked Read 2 Done Done Sent false
34954 9660 Acked Write 8 Done Done Sent false
34955 9661 Acked Write 2 Done Done Sent false
34956 9662 Acked Write 1 Done Done Sent false
34957 9663 Acked Flush 0 Done Done Sent false
34958 9664 Acked Read 1 Done Done Sent false
34959 9665 Acked Read 10 Done Done Sent false
34960 9666 Acked Write 1 Done Done Sent false
34961 9667 Acked Write 5 Done Done Sent false
34962 9668 Acked Read 1 Done Done Sent false
34963 9669 Acked Write 10 Done Done Sent false
34964 9670 Acked Write 2 Done Done Sent false
34965 9671 Acked Read 4 Done Done Sent false
34966 9672 Acked Write 1 Done Done Sent false
34967 9673 Acked Read 5 Done Done Sent false
34968 9674 Acked Flush 0 Done Done Sent false
34969 9675 Acked Write 8 Done Done Sent false
34970 9676 Acked Write 1 Done Done Sent false
34971 9677 Acked Read 7 Done Done Sent false
34972 9678 Acked Write 3 Done Done Sent false
34973 9679 Acked Read 2 Done Done Sent false
34974 9680 Acked Read 7 Done Done Sent false
34975 9681 Acked Read 1 Done Done Sent false
34976 9682 Acked Flush 0 Done Done Sent false
34977 9683 Acked Write 2 Done Done Sent false
34978 9684 Acked Read 10 Done Done Sent false
34979 9685 Acked Read 6 Done Done Sent false
34980 9686 Acked Write 8 Done Done Sent false
34981 9687 Acked Read 2 Done Done Sent false
34982 9688 Acked Write 6 Done Done Sent false
34983 9689 Acked Read 4 Done Done Sent false
34984 9690 Acked Read 8 Done Done Sent false
34985 9691 Acked Read 9 Done Done Sent false
34986 9692 Acked Flush 0 Done Done Sent false
34987 9693 Acked Flush 0 Done Done Sent false
34988 9694 Acked Write 10 Done Done Sent false
34989 9695 Acked Flush 0 Done Done Sent false
34990 9696 Acked Write 8 Done Done Sent false
34991 9697 Acked Flush 0 Done Done Sent false
34992 9698 Acked Read 8 Done Done Sent false
34993 9699 Acked Read 5 Done Done Sent false
34994 9700 Acked Write 2 Done Done Sent false
34995 9701 Acked Write 5 Done Done Sent false
34996 9702 Acked Read 4 Done Done Sent false
34997 9703 Acked Read 10 Done Done Sent false
34998 9704 Acked Write 6 Done Done Sent false
34999 9705 Acked Write 10 Done Done Sent false
35000 9706 Acked Read 7 Done Done Sent false
35001 9707 Acked Read 9 Done Done Sent false
35002 9708 Acked Flush 0 Done Done Sent false
35003 9709 Acked Write 8 Done Done Sent false
35004 9710 Acked Read 9 Done Done Sent false
35005 9711 Acked Flush 0 Done Done Sent false
35006 9712 Acked Read 9 Done Done Sent false
35007 9713 Acked Read 7 Done Done Sent false
35008 9714 Acked Write 4 Done Done Sent false
35009 9715 Acked Write 7 Done Done Sent false
35010 9716 Acked Write 5 Done Done Sent false
35011 9717 Acked Read 9 Done Done Sent false
35012 9718 Acked Read 6 Done Done Sent false
35013 9719 Acked Read 4 Done Done Sent false
35014 9720 Acked Write 4 Done Done Sent false
35015 9721 Acked Write 4 Done Done Sent false
35016 9722 Acked Write 3 Done Done Sent false
35017 9723 Acked Read 4 Done Done Sent false
35018 9724 Acked Flush 0 Done Done Sent false
35019 9725 Acked Read 2 Done Done Sent false
35020 9726 Acked Flush 0 Done Done Sent false
35021 9727 Acked Write 8 Done Done Sent false
35022 9728 Acked Read 3 Done Done Sent false
35023 9729 Acked Read 6 Done Done Sent false
35024 9730 Acked Read 5 Done Done Sent false
35025 9731 Acked Read 2 Done Done Sent false
35026 9732 Acked Read 4 Done Done Sent false
35027 9733 Acked Read 7 Done Done Sent false
35028 9734 Acked Write 7 Done Done Sent false
35029 9735 Acked Read 9 Done Done Sent false
35030 9736 Acked Read 7 Done Done Sent false
35031 9737 Acked Write 2 Done Done Sent false
35032 9738 Acked Write 7 Done Done Sent false
35033 9739 Acked Read 9 Done Done Sent false
35034 9740 Acked Write 1 Done Done Sent false
35035 9741 Acked Write 6 Done Done Sent false
35036 9742 Acked Read 3 Done Done Sent false
35037 9743 Acked Read 9 Done Done Sent false
35038 9744 Acked Write 4 Done Done Sent false
35039 9745 Acked Read 1 Done Done Sent false
35040 9746 Acked Write 6 Done Done Sent false
35041 9747 Acked Write 1 Done Done Sent false
35042 9748 Acked Write 8 Done Done Sent false
35043 9749 Acked Read 2 Done Done Sent false
35044 9750 Acked Write 8 Done Done Sent false
35045 9751 Acked Read 2 Done Done Sent false
35046 9752 Acked Write 7 Done Done Sent false
35047 9753 Acked Read 9 Done Done Sent false
35048 STATES DS:0 DS:1 DS:2 TOTAL
35049 Sent 0 0 301 301
35050 Done 301 301 0 602
35051 Skipped 0 0 0 0
35052 Error 0 0 0 0
35053 Last Flush: Some(JobId(9726)) Some(JobId(9726)) Some(JobId(9452))
35054 Downstairs last five completed: 9452 9451 9450 9449 9448
35055 Upstairs last five completed: 9753 9752 9751 9750 9749
35056 CLIENT: Up:0 ds:301 act:2
350572025-03-17T22:42:17.027ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350582025-03-17T22:42:18.223ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350592025-03-17T22:42:19.031ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350602025-03-17T22:42:19.986ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350612025-03-17T22:42:19.986ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41416) = io task client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350622025-03-17T22:42:19.987ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350632025-03-17T22:42:19.988ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350642025-03-17T22:42:19.988ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350652025-03-17T22:42:19.988ZINFOcrucible: send last flush ID to this DS: Some(JobId(9452)) = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350662025-03-17T22:42:19.988ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350672025-03-17T22:42:19.988ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9452)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
350682025-03-17T22:42:19.988ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9452)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
35069 ----------------------------------------------------------------
35070 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:27
35071 States: Active Active Active
35072 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35073 9727 Acked Write 8 Done Done Done true
35074 9728 Acked Read 3 Done Done Done true
35075 9729 Acked Read 6 Done Done Done true
35076 9730 Acked Read 5 Done Done Done true
35077 9731 Acked Read 2 Done Done Done true
35078 9732 Acked Read 4 Done Done Done true
35079 9733 Acked Read 7 Done Done Done true
35080 9734 Acked Write 7 Done Done Done true
35081 9735 Acked Read 9 Done Done Done true
35082 9736 Acked Read 7 Done Done Done true
35083 9737 Acked Write 2 Done Done Done true
35084 9738 Acked Write 7 Done Done Done true
35085 9739 Acked Read 9 Done Done Done true
35086 9740 Acked Write 1 Done Done Done true
35087 9741 Acked Write 6 Done Done Done true
35088 9742 Acked Read 3 Done Done Done true
35089 9743 Acked Read 9 Done Done Done true
35090 9744 Acked Write 4 Done Done Done true
35091 9745 Acked Read 1 Done Done Done true
35092 9746 Acked Write 6 Done Done Done true
35093 9747 Acked Write 1 Done Done Done true
35094 9748 Acked Write 8 Done Done Done true
35095 9749 Acked Read 2 Done Done Done true
35096 9750 Acked Write 8 Done Done Done true
35097 9751 Acked Read 2 Done Done Done true
35098 9752 Acked Write 7 Done Done Done true
35099 9753 Acked Read 9 Done Done Done true
35100 STATES DS:0 DS:1 DS:2 TOTAL
35101 Sent 0 0 0 0
35102 Done 27 27 27 81
35103 Skipped 0 0 0 0
35104 Error 0 0 0 0
35105 Last Flush: Some(JobId(9726)) Some(JobId(9726)) Some(JobId(9726))
35106 Downstairs last five completed: 9726 9725 9724 9723 9722
35107 Upstairs last five completed: 9753 9752 9751 9750 9749
35108 CLIENT: Up:0 ds:27 act:3
35109 ----------------------------------------------------------------
35110 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35111 Downstairs last five completed: 9754 9753 9752 9751 9750
35112 Upstairs last five completed: 9754 9753 9752 9751 9750
35113 CLIENT: Up:0 ds:0 act:3
35114 Replay: All jobs finished
35115 Main task has work to do, go find it
35116 got dsc Stop(0)
35117 stop 0
35118 Mar 17 22:42:24.243 INFO request completed, latency_us: 365, response_code[8810] Got stop action so:false kr:false
35119 : 204, uri: /stop/cid/0, method: POST, req_id: 27c987d5-274c-4eda-aba3-6632e402558d, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
35120 Mar 17 22:42:24.243 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/0, method: GET, req_id: f8da6571-723d-4c6d-a3ed-4e2bcbf12145, remote_addr: 127.0.0.1:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:42:24.243969483Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
35121 35180, local_addr: 127.0.0.1:9998
351222025-03-17T22:42:24.244ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
351232025-03-17T22:42:24.244ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
351242025-03-17T22:42:24.244ZINFOcrucible: client task is exiting = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
35125 [8810] Exited with: ExitStatus(unix_wait_status(9))
35126 [8810][0] reports Exit
35127 Mar 17 22:42:28.245 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/0, method: GET, req_id: cde04fbf-469c-4c8c-abbf-095fcf42f274, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
35128 001/300 Write block 4623 len 8192 data: 4 5
35129 002/300 Read block 2046 len 16384
35130 003/300 Read block 1276 len 20480
35131 004/300 Read block 4457 len 8192
35132 005/300 Write block 4327 len 20480 data: 6 5 4 5 4
35133 006/300 Write block 4597 len 12288 data: 5 7 8
35134 007/300 Read block 2141 len 20480
35135 008/300 Read block 3589 len 20480
35136 009/300 Write block 2150 len 12288 data: 5 6 7
35137 010/300 Write block 1385 len 36864 data: 4 5 5 6 6 5 6 7 6
35138 011/300 Read block 161 len 24576
35139 012/300 Write block 1809 len 12288 data: 8 6 6
35140 013/300 Read block 2549 len 24576
35141 014/300 Read block 4485 len 36864
35142 015/300 Flush
35143 016/300 Read block 2684 len 16384
35144 017/300 Write block 4619 len 4096 data: 4
35145 018/300 Write block 4445 len 12288 data: 7 6 5
35146 019/300 Read block 1388 len 32768
35147 020/300 Write block 1251 len 20480 data: 7 7 7 7 7
35148 021/300 Read block 1515 len 24576
35149 022/300 Write block 3772 len 12288 data: 5 6 5
35150 023/300 Read block 4908 len 32768
35151 024/300 Read block 258 len 24576
35152 025/300 Write block 3379 len 24576 data: 5 6 7 8 8 10
35153 026/300 Read block 26 len 24576
35154 027/300 Flush
35155 028/300 Write block 3866 len 24576 data: 6 6 6 6 7 8
35156 029/300 Flush
35157 030/300 Read block 2373 len 28672
35158 031/300 Read block 2459 len 28672
35159 032/300 Read block 1025 len 40960
35160 033/300 Read block 462 len 16384
35161 034/300 Write block 81 len 12288 data: 8 8 12
35162 035/300 Write block 2488 len 4096 data: 7
35163 036/300 Read block 2433 len 40960
35164 037/300 Flush
35165 038/300 Read block 3863 len 8192
35166 039/300 Read block 4614 len 16384
35167 040/300 Read block 3528 len 28672
35168 041/300 Read block 2513 len 16384
35169 042/300 Read block 4862 len 8192
35170 043/300 Read block 913 len 20480
35171 044/300 Read block 4437 len 40960
35172 045/300 Write block 405 len 16384 data: 8 6 6 7
35173 046/300 Read block 1399 len 12288
35174 047/300 Read block 1567 len 8192
35175 048/300 Write block 4722 len 12288 data: 2 3 4
35176 049/300 Read block 4004 len 36864
35177 050/300 Read block 2281 len 28672
35178 051/300 Write block 3933 len 24576 data: 7 7 7 6 6 5
35179 052/300 Write block 1640 len 36864 data: 9 7 6 7 11 9 9 6 7
35180 053/300 Flush
35181 054/300 Read block 3835 len 20480
35182 055/300 Flush
35183 056/300 Read block 1746 len 28672
35184 057/300 Write block 367 len 12288 data: 6 5 6
35185 058/300 Write block 4119 len 36864 data: 9 8 7 7 6 5 3 3 3
35186 059/300 Read block 2405 len 16384
35187 060/300 Write block 1395 len 8192 data: 6 7
35188 061/300 Read block 4103 len 24576
35189 062/300 Write block 2190 len 12288 data: 5 4 4
35190 063/300 Read block 4363 len 12288
35191 064/300 Read block 1498 len 4096
35192 065/300 Write block 2674 len 32768 data: 4 4 4 4 3 3 3 5
35193 066/300 Write block 917 len 40960 data: 2 3 4 3 4 3 4 4 6 7
35194 067/300 Read block 3256 len 40960
35195 068/300 Flush
35196 069/300 Read block 57 len 32768
35197 070/300 Read block 3788 len 16384
35198 071/300 Write block 4155 len 16384 data: 4 6 7 7
35199 072/300 Write block 3932 len 40960 data: 9 8 8 8 7 7 6 5 7 7
35200 073/300 Read block 4142 len 36864
35201 074/300 Read block 2875 len 20480
35202 075/300 Write block 4044 len 24576 data: 7 4 4 4 2 4
35203 076/300 Write block 411 len 12288 data: 2 3 3
35204 077/300 Read block 3741 len 36864
35205 078/300 Write block 3920 len 28672 data: 7 8 8 6 5 6 8
35206 079/300 Write block 4331 len 24576 data: 5 4 5 5 3 4
35207 080/300 Read block 4173 len 16384
35208 081/300 Read block 397 len 28672
35209 082/300 Write block 1472 len 8192 data: 4 4
35210 083/300 Read block 797 len 24576
35211 084/300 Read block 4427 len 8192
35212 085/300 Write block 721 len 20480 data: 5 5 7 7 9
35213 086/300 Read block 3384 len 4096
35214 087/300 Read block 4539 len 20480
35215 088/300 Read block 215 len 8192
35216 089/300 Write block 2683 len 12288 data: 6 7 7
35217 090/300 Read block 1011 len 24576
35218 091/300 Read block 2212 len 40960
35219 092/300 Write block 1546 len 12288 data: 6 8 9
35220 093/300 Read block 900 len 32768
35221 094/300 Read block 4133 len 16384
35222 095/300 Read block 1445 len 36864
35223 096/300 Write block 4009 len 40960 data: 7 7 7 9 10 10 9 9 10 9
35224 097/300 Write block 3621 len 24576 data: 7 7 8 8 9 9
35225 098/300 Write block 4833 len 24576 data: 6 6 6 5 5 6
35226 099/300 Read block 3051 len 8192
35227 100/300 Read block 834 len 12288
35228 101/300 Read block 1436 len 16384
35229 102/300 Read block 558 len 28672
35230 103/300 Read block 3607 len 20480
35231 104/300 Read block 1786 len 32768
35232 105/300 Read block 4903 len 24576
35233 106/300 Write block 1879 len 28672 data: 7 8 7 7 5 5 6
35234 107/300 Write block 669 len 8192 data: 6 6
35235 108/300 Read block 1593 len 4096
35236 109/300 Write block 3387 len 40960 data: 8 11 8 7 6 6 6 6 7 7
35237 110/300 Read block 3268 len 24576
35238 111/300 Read block 1049 len 4096
35239 112/300 Write block 3195 len 36864 data: 8 8 5 5 4 4 3 4 5
35240 113/300 Read block 2195 len 20480
35241 114/300 Read block 1732 len 16384
35242 115/300 Read block 3354 len 4096
35243 116/300 Write block 4445 len 28672 data: 8 7 6 3 3 3 2
35244 117/300 Flush
35245 118/300 Write block 3121 len 32768 data: 4 5 5 4 4 4 5 7
35246 119/300 Read block 92 len 24576
35247 120/300 Write block 2753 len 40960 data: 10 9 9 7 6 5 4 4 5 3
35248 121/300 Read block 838 len 20480
35249 122/300 Write block 2897 len 12288 data: 2 2 2
35250 123/300 Write block 3404 len 24576 data: 6 7 7 7 7 8
35251 124/300 Read block 24 len 8192
35252 125/300 Flush
35253 126/300 Write block 4947 len 24576 data: 6 7 5 6 6 7
35254 127/300 Write block 1942 len 4096 data: 6
35255 128/300 Read block 1412 len 16384
35256 129/300 Write block 3912 len 20480 data: 7 7 7 7 6
35257 130/300 Read block 3893 len 12288
35258 131/300 Write block 4622 len 8192 data: 3 5
35259 132/300 Write block 1188 len 24576 data: 6 6 7 7 7 8
35260 133/300 Write block 4922 len 20480 data: 7 8 8 6 6
35261 134/300 Write block 2313 len 40960 data: 4 4 2 2 2 3 4 4 4 4
35262 135/300 Read block 1440 len 40960
35263 136/300 Read block 3721 len 20480
35264 137/300 Write block 2308 len 40960 data: 3 3 3 3 4 5 5 3 3 3
35265 138/300 Write block 4021 len 24576 data: 7 8 9 8 9 9
35266 139/300 Read block 4132 len 12288
35267 140/300 Write block 3639 len 40960 data: 3 4 4 4 4 3 3 3 3 3
35268 141/300 Write block 4332 len 16384 data: 5 6 6 4
35269 142/300 Read block 3756 len 8192
35270 143/300 Read block 2870 len 16384
35271 144/300 Flush
35272 145/300 Write block 2265 len 8192 data: 10 10
35273 146/300 Read block 1387 len 8192
35274 147/300 Write block 392 len 20480 data: 6 6 4 3 2
35275 148/300 Write block 1610 len 16384 data: 6 6 4 4
35276 149/300 Read block 2544 len 8192
35277 150/300 Flush
35278 151/300 Read block 963 len 40960
35279 152/300 Write block 4449 len 36864 data: 4 4 3 3 3 3 3 4 3
35280 153/300 Write block 1905 len 4096 data: 5
35281 154/300 Write block 1865 len 12288 data: 7 8 6
35282 155/300 Flush
35283 156/300 Read block 4081 len 8192
35284 157/300 Read block 4070 len 16384
35285 158/300 Read block 173 len 28672
35286 159/300 Read block 598 len 20480
35287 160/300 Read block 3773 len 32768
35288 161/300 Write block 3591 len 8192 data: 5 6
35289 162/300 Flush
35290 163/300 Write block 2832 len 4096 data: 8
35291 164/300 Write block 2489 len 20480 data: 5 3 3 5 7
35292 165/300 Read block 602 len 4096
35293 166/300 Read block 821 len 24576
35294 167/300 Write block 1331 len 12288 data: 5 6 6
35295 168/300 Write block 3073 len 24576 data: 5 4 4 3 4 4
35296 169/300 Read block 2480 len 16384
35297 170/300 Read block 3161 len 12288
35298 171/300 Read block 4569 len 40960
35299 172/300 Write block 2425 len 28672 data: 7 7 7 7 6 7 7
35300 173/300 Write block 137 len 12288 data: 7 8 7
35301 174/300 Write block 3456 len 24576 data: 3 3 3 4 4 4
35302 175/300 Read block 4671 len 32768
35303 176/300 Write block 2349 len 28672 data: 4 5 7 4 5 4 4
35304 177/300 Write block 3340 len 40960 data: 10 8 8 7 7 5 5 5 4 4
35305 178/300 Write block 3084 len 28672 data: 5 5 5 4 6 6 5
35306 179/300 Read block 2760 len 4096
35307 180/300 Flush
35308 181/300 Write block 3175 len 32768 data: 7 7 8 9 9 9 10 8
35309 182/300 Flush
35310 183/300 Read block 2475 len 28672
35311 184/300 Read block 2240 len 20480
35312 185/300 Read block 1170 len 20480
35313 186/300 Write block 280 len 12288 data: 6 7 3
35314 187/300 Read block 2264 len 20480
35315 188/300 Write block 1282 len 24576 data: 5 6 6 5 5 4
35316 189/300 Write block 4440 len 4096 data: 6
35317 190/300 Write block 190 len 12288 data: 6 6 7
35318 191/300 Write block 3619 len 20480 data: 4 5 8 8 9
35319 192/300 Read block 2002 len 24576
35320 193/300 Read block 1466 len 40960
35321 194/300 Read block 1705 len 8192
35322 195/300 Read block 3469 len 32768
35323 196/300 Flush
35324 197/300 Write block 4182 len 16384 data: 7 7 5 7
35325 198/300 Write block 3484 len 28672 data: 4 5 5 5 4 3 3
35326 199/300 Write block 4435 len 12288 data: 6 6 6
35327 200/300 Read block 1249 len 20480
35328 201/300 Read block 2648 len 36864
35329 202/300 Write block 75 len 24576 data: 3 4 6 6 7 7
35330 203/300 Read block 2574 len 32768
35331 204/300 Write block 2018 len 40960 data: 5 5 5 4 3 3 3 3 4 7
35332 205/300 Read block 199 len 20480
35333 206/300 Write block 860 len 8192 data: 5 5
35334 207/300 Read block 4358 len 32768
35335 208/300 Flush
35336 209/300 Read block 2415 len 4096
35337 210/300 Write block 4932 len 20480 data: 7 9 11 10 8
35338 211/300 Flush
35339 212/300 Read block 1492 len 32768
35340 213/300 Flush
35341 214/300 Read block 1578 len 32768
35342 215/300 Write block 3957 len 36864 data: 5 6 6 6 7 7 8 7 7
35343 216/300 Write block 420 len 24576 data: 8 8 8 8 7 8
35344 217/300 Read block 3908 len 16384
35345 218/300 Write block 2566 len 12288 data: 6 5 7
35346 219/300 Read block 4187 len 24576
35347 220/300 Flush
35348 221/300 Flush
35349 222/300 Write block 4862 len 4096 data: 3
35350 223/300 Write block 4655 len 36864 data: 6 5 3 4 8 7 7 6 6
35351 224/300 Flush
35352 225/300 Write block 1 len 4096 data: 6
35353 226/300 Write block 59 len 40960 data: 4 6 6 7 6 7 8 9 10 10
35354 227/300 Read block 4005 len 24576
35355 228/300 Read block 3905 len 16384
35356 229/300 Read block 3540 len 20480
35357 230/300 Write block 3498 len 36864 data: 2 2 3 3 6 6 7 6 7
35358 231/300 Read block 1013 len 4096
35359 232/300 Read block 1201 len 8192
35360 233/300 Write block 3050 len 24576 data: 4 5 5 6 6 7
35361 234/300 Read block 3050 len 8192
35362 235/300 Read block 3645 len 20480
35363 236/300 Read block 4474 len 40960
35364 237/300 Read block 2770 len 24576
35365 238/300 Read block 4045 len 12288
35366 239/300 Read block 2209 len 12288
35367 240/300 Read block 4203 len 12288
35368 241/300 Read block 4130 len 12288
35369 242/300 Read block 4946 len 12288
35370 243/300 Flush
35371 244/300 Read block 665 len 8192
35372 245/300 Read block 836 len 24576
35373 246/300 Write block 2004 len 16384 data: 8 6 6 7
35374 247/300 Read block 2077 len 36864
35375 248/300 Read block 2966 len 24576
35376 249/300 Read block 4255 len 4096
35377 250/300 Read block 4499 len 16384
35378 251/300 Write block 4239 len 20480 data: 6 6 5 4 5
35379 252/300 Read block 4682 len 28672
35380 253/300 Write block 3316 len 4096 data: 4
35381 254/300 Read block 1188 len 40960
35382 255/300 Write block 710 len 8192 data: 7 6
35383 256/300 Read block 3519 len 20480
35384 257/300 Flush
35385 258/300 Flush
35386 259/300 Read block 681 len 36864
35387 260/300 Read block 1459 len 8192
35388 261/300 Read block 4327 len 28672
35389 262/300 Write block 3052 len 8192 data: 6 7
35390 263/300 Read block 3669 len 36864
35391 264/300 Read block 4811 len 4096
35392 265/300 Read block 47 len 8192
35393 266/300 Write block 4522 len 32768 data: 7 6 6 6 4 7 8 8
35394 267/300 Write block 4529 len 36864 data: 9 8 8 7 8 6 8 8 8
35395 268/300 Flush
35396 269/300 Flush
35397 270/300 Write block 1376 len 4096 data: 7
35398 271/300 Write block 2783 len 24576 data: 5 4 6 5 5 7
35399 272/300 Read block 3553 len 40960
35400 273/300 Write block 1276 len 36864 data: 6 8 6 7 8 6 6 7 7
35401 274/300 Read block 4803 len 4096
35402 275/300 Flush
35403 276/300 Write block 194 len 24576 data: 7 7 5 5 4 5
35404 277/300 Flush
35405 278/300 Flush
35406 279/300 Write block 297 len 12288 data: 9 6 6
35407 280/300 Read block 313 len 28672
35408 281/300 Read block 893 len 36864
35409 282/300 Write block 1336 len 40960 data: 5 5 5 5 6 6 4 5 5 4
35410 283/300 Write block 1040 len 36864 data: 5 5 6 5 3 3 4 4 5
35411 284/300 Read block 3657 len 8192
35412 285/300 Read block 3250 len 28672
35413 286/300 Write block 899 len 12288 data: 5 5 7
35414 287/300 Write block 4208 len 4096 data: 9
35415 288/300 Read block 1047 len 4096
35416 289/300 Read block 3141 len 40960
35417 290/300 Read block 1524 len 28672
35418 291/300 Read block 1341 len 8192
35419 292/300 Read block 2193 len 40960
35420 293/300 Write block 2174 len 24576 data: 5 5 5 7 6 6
35421 294/300 Read block 4313 len 24576
35422 295/300 Flush
35423 296/300 Write block 4918 len 36864 data: 7 8 7 8 8 9 9 7 7
35424 297/300 Write block 4142 len 28672 data: 9 8 7 8 6 4 4
35425 298/300 Write block 1395 len 20480 data: 7 8 7 7 6
35426 299/300 Read block 479 len 12288
35427 300/300 Read block 4762 len 28672
35428 301/300 Write block 3112 len 12288 data: 4 4 5
35429 Main task has work to do, go find it
35430 got dsc Start(0)
35431 start 0
35432 Mar 17 22:42:28.487 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 85500d64-bca4-4585-a95f-762cf72962ac, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
35433 Replay: started 0, returned:Ok(())
35434 ----------------------------------------------------------------
35435 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35436 States: Offline Active Active
35437 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35438 9755 Acked Write 2 Sent Done Done false
35439 9756 Acked Read 4 Sent Done Done false
35440 9757 Acked Read 5 Sent Done Done false
35441 9758 Acked Read 2 Sent Done Done false
35442 9759 Acked Write 5 Sent Done Done false
35443 9760 Acked Write 3 Sent Done Done false
35444 9761 Acked Read 5 Sent Done Done false
35445 9762 Acked Read 5 Sent Done Done false
35446 9763 Acked Write 3 Sent Done Done false
35447 9764 Acked Write 9 Sent Done Done false
35448 9765 Acked Read 6 Sent Done Done false
35449 9766 Acked Write 3 Sent Done Done false
35450 9767 Acked Read 6 Sent Done Done false
35451 9768 Acked Read 9 Sent Done Done false
35452 9769 Acked Flush 0 Sent Done Done false
35453 9770 Acked Read 4 Sent Done Done false
35454 9771 Acked Write 1 Sent Done Done false
35455 9772 Acked Write 3 Sent Done Done false
35456 9773 Acked Read 8 Sent Done Done false
35457 9774 Acked Write 5 Sent Done Done false
35458 9775 Acked Read 6 Sent Done Done false
35459 9776 Acked Write 3 Sent Done Done false
35460 9777 Acked Read 8 Sent Done Done false
35461 9778 Acked Read 6 Sent Done Done false
35462 9779 Acked Write 6 Sent Done Done false
35463 9780 Acked Read 6 Sent Done Done false
35464 9781 Acked Flush 0 Sent Done Done false
35465 9782 Acked Write 6 Sent Done Done false
35466 9783 Acked Flush 0 Sent Done Done false
35467 9784 Acked Read 7 Sent Done Done false
35468 9785 Acked Read 7 Sent Done Done false
35469 9786 Acked Read 10 Sent Done Done false
35470 9787 Acked Read 4 Sent Done Done false
35471 9788 Acked Write 3 Sent Done Done false
35472 9789 Acked Write 1 Sent Done Done false
35473 9790 Acked Read 10 Sent Done Done false
35474 9791 Acked Flush 0 Sent Done Done false
35475 9792 Acked Read 2 Sent Done Done false
35476 9793 Acked Read 4 Sent Done Done false
35477 9794 Acked Read 7 Sent Done Done false
35478 9795 Acked Read 4 Sent Done Done false
35479 9796 Acked Read 2 Sent Done Done false
35480 9797 Acked Read 5 Sent Done Done false
35481 9798 Acked Read 10 Sent Done Done false
35482 9799 Acked Write 4 Sent Done Done false
35483 9800 Acked Read 3 Sent Done Done false
35484 9801 Acked Read 2 Sent Done Done false
35485 9802 Acked Write 3 Sent Done Done false
35486 9803 Acked Read 9 Sent Done Done false
35487 9804 Acked Read 7 Sent Done Done false
35488 9805 Acked Write 6 Sent Done Done false
35489 9806 Acked Write 9 Sent Done Done false
35490 9807 Acked Flush 0 Sent Done Done false
35491 9808 Acked Read 5 Sent Done Done false
35492 9809 Acked Flush 0 Sent Done Done false
35493 9810 Acked Read 7 Sent Done Done false
35494 9811 Acked Write 3 Sent Done Done false
35495 9812 Acked Write 9 Sent Done Done false
35496 9813 Acked Read 4 Sent Done Done false
35497 9814 Acked Write 2 Sent Done Done false
35498 9815 Acked Read 6 Sent Done Done false
35499 9816 Acked Write 3 Sent Done Done false
35500 9817 Acked Read 3 Sent Done Done false
35501 9818 Acked Read 1 Sent Done Done false
35502 9819 Acked Write 8 Sent Done Done false
35503 9820 Acked Write 10 Sent Done Done false
35504 9821 Acked Read 10 Sent Done Done false
35505 9822 Acked Flush 0 Sent Done Done false
35506 9823 Acked Read 8 Sent Done Done false
35507 9824 Acked Read 4 Sent Done Done false
35508 9825 Acked Write 4 Sent Done Done false
35509 9826 Acked Write 10 Sent Done Done false
35510 9827 Acked Read 9 Sent Done Done false
35511 9828 Acked Read 5 Sent Done Done false
35512 9829 Acked Write 6 Sent Done Done false
35513 9830 Acked Write 3 Sent Done Done false
35514 9831 Acked Read 9 Sent Done Done false
35515 9832 Acked Write 7 Sent Done Done false
35516 9833 Acked Write 6 Sent Done Done false
35517 9834 Acked Read 4 Sent Done Done false
35518 9835 Acked Read 7 Sent Done Done false
35519 9836 Acked Write 2 Sent Done Done false
35520 9837 Acked Read 6 Sent Done Done false
35521 9838 Acked Read 2 Sent Done Done false
35522 9839 Acked Write 5 Sent Done Done false
35523 9840 Acked Read 1 Sent Done Done false
35524 9841 Acked Read 5 Sent Done Done false
35525 9842 Acked Read 2 Sent Done Done false
35526 9843 Acked Write 3 Sent Done Done false
35527 9844 Acked Read 6 Sent Done Done false
35528 9845 Acked Read 10 Sent Done Done false
35529 9846 Acked Write 3 Sent Done Done false
35530 9847 Acked Read 8 Sent Done Done false
35531 9848 Acked Read 4 Sent Done Done false
35532 9849 Acked Read 9 Sent Done Done false
35533 9850 Acked Write 10 Sent Done Done false
35534 9851 Acked Write 6 Sent Done Done false
35535 9852 Acked Write 6 Sent Done Done false
35536 9853 Acked Read 2 Sent Done Done false
35537 9854 Acked Read 3 Sent Done Done false
35538 9855 Acked Read 4 Sent Done Done false
35539 9856 Acked Read 7 Sent Done Done false
35540 9857 Acked Read 5 Sent Done Done false
35541 9858 Acked Read 8 Sent Done Done false
35542 9859 Acked Read 6 Sent Done Done false
35543 9860 Acked Write 7 Sent Done Done false
35544 9861 Acked Write 2 Sent Done Done false
35545 9862 Acked Read 1 Sent Done Done false
35546 9863 Acked Write 10 Sent Done Done false
35547 9864 Acked Read 6 Sent Done Done false
35548 9865 Acked Read 1 Sent Done Done false
35549 9866 Acked Write 9 Sent Done Done false
35550 9867 Acked Read 5 Sent Done Done false
35551 9868 Acked Read 4 Sent Done Done false
35552 9869 Acked Read 1 Sent Done Done false
35553 9870 Acked Write 7 Sent Done Done false
35554 9871 Acked Flush 0 Sent Done Done false
35555 9872 Acked Write 8 Sent Done Done false
35556 9873 Acked Read 6 Sent Done Done false
35557 9874 Acked Write 10 Sent Done Done false
35558 9875 Acked Read 5 Sent Done Done false
35559 9876 Acked Write 3 Sent Done Done false
35560 9877 Acked Write 6 Sent Done Done false
35561 9878 Acked Read 2 Sent Done Done false
35562 9879 Acked Flush 0 Sent Done Done false
35563 9880 Acked Write 6 Sent Done Done false
35564 9881 Acked Write 1 Sent Done Done false
35565 9882 Acked Read 4 Sent Done Done false
35566 9883 Acked Write 5 Sent Done Done false
35567 9884 Acked Read 3 Sent Done Done false
35568 9885 Acked Write 2 Sent Done Done false
35569 9886 Acked Write 6 Sent Done Done false
35570 9887 Acked Write 5 Sent Done Done false
35571 9888 Acked Write 10 Sent Done Done false
35572 9889 Acked Read 10 Sent Done Done false
35573 9890 Acked Read 5 Sent Done Done false
35574 9891 Acked Write 10 Sent Done Done false
35575 9892 Acked Write 6 Sent Done Done false
35576 9893 Acked Read 3 Sent Done Done false
35577 9894 Acked Write 10 Sent Done Done false
35578 9895 Acked Write 4 Sent Done Done false
35579 9896 Acked Read 2 Sent Done Done false
35580 9897 Acked Read 4 Sent Done Done false
35581 9898 Acked Flush 0 Sent Done Done false
35582 9899 Acked Write 2 Sent Done Done false
35583 9900 Acked Read 2 Sent Done Done false
35584 9901 Acked Write 5 Sent Done Done false
35585 9902 Acked Write 4 Sent Done Done false
35586 9903 Acked Read 2 Sent Done Done false
35587 9904 Acked Flush 0 Sent Done Done false
35588 9905 Acked Read 10 Sent Done Done false
35589 9906 Acked Write 9 Sent Done Done false
35590 9907 Acked Write 1 Sent Done Done false
35591 9908 Acked Write 3 Sent Done Done false
35592 9909 Acked Flush 0 Sent Done Done false
35593 9910 Acked Read 2 Sent Done Done false
35594 9911 Acked Read 4 Sent Done Done false
35595 9912 Acked Read 7 Sent Done Done false
35596 9913 Acked Read 5 Sent Done Done false
35597 9914 Acked Read 8 Sent Done Done false
35598 9915 Acked Write 2 Sent Done Done false
35599 9916 Acked Flush 0 Sent Done Done false
35600 9917 Acked Write 1 Sent Done Done false
35601 9918 Acked Write 5 Sent Done Done false
35602 9919 Acked Read 1 Sent Done Done false
35603 9920 Acked Read 6 Sent Done Done false
35604 9921 Acked Write 3 Sent Done Done false
35605 9922 Acked Write 6 Sent Done Done false
35606 9923 Acked Read 4 Sent Done Done false
35607 9924 Acked Read 3 Sent Done Done false
35608 9925 Acked Read 10 Sent Done Done false
35609 9926 Acked Write 7 Sent Done Done false
35610 9927 Acked Write 3 Sent Done Done false
35611 9928 Acked Write 6 Sent Done Done false
35612 9929 Acked Read 8 Sent Done Done false
35613 9930 Acked Write 7 Sent Done Done false
35614 9931 Acked Write 10 Sent Done Done false
35615 9932 Acked Write 7 Sent Done Done false
35616 9933 Acked Read 1 Sent Done Done false
35617 9934 Acked Flush 0 Sent Done Done false
35618 9935 Acked Write 8 Sent Done Done false
35619 9936 Acked Flush 0 Sent Done Done false
35620 9937 Acked Read 7 Sent Done Done false
35621 9938 Acked Read 5 Sent Done Done false
35622 9939 Acked Read 5 Sent Done Done false
35623 9940 Acked Write 3 Sent Done Done false
35624 9941 Acked Read 5 Sent Done Done false
35625 9942 Acked Write 6 Sent Done Done false
35626 9943 Acked Write 1 Sent Done Done false
35627 9944 Acked Write 3 Sent Done Done false
35628 9945 Acked Write 5 Sent Done Done false
35629 9946 Acked Read 6 Sent Done Done false
35630 9947 Acked Read 10 Sent Done Done false
35631 9948 Acked Read 2 Sent Done Done false
35632 9949 Acked Read 8 Sent Done Done false
35633 9950 Acked Flush 0 Sent Done Done false
35634 9951 Acked Write 4 Sent Done Done false
35635 9952 Acked Write 7 Sent Done Done false
35636 9953 Acked Write 3 Sent Done Done false
35637 9954 Acked Read 5 Sent Done Done false
35638 9955 Acked Read 9 Sent Done Done false
35639 9956 Acked Write 6 Sent Done Done false
35640 9957 Acked Read 8 Sent Done Done false
35641 9958 Acked Write 10 Sent Done Done false
35642 9959 Acked Read 5 Sent Done Done false
35643 9960 Acked Write 2 Sent Done Done false
35644 9961 Acked Read 8 Sent Done Done false
35645 9962 Acked Flush 0 Sent Done Done false
35646 9963 Acked Read 1 Sent Done Done false
35647 9964 Acked Write 5 Sent Done Done false
35648 9965 Acked Flush 0 Sent Done Done false
35649 9966 Acked Read 8 Sent Done Done false
35650 9967 Acked Flush 0 Sent Done Done false
35651 9968 Acked Read 8 Sent Done Done false
35652 9969 Acked Write 9 Sent Done Done false
35653 9970 Acked Write 6 Sent Done Done false
35654 9971 Acked Read 4 Sent Done Done false
35655 9972 Acked Write 3 Sent Done Done false
35656 9973 Acked Read 6 Sent Done Done false
35657 9974 Acked Flush 0 Sent Done Done false
35658 9975 Acked Flush 0 Sent Done Done false
35659 9976 Acked Write 1 Sent Done Done false
35660 9977 Acked Write 9 Sent Done Done false
35661 9978 Acked Flush 0 Sent Done Done false
35662 9979 Acked Write 1 Sent Done Done false
35663 9980 Acked Write 10 Sent Done Done false
35664 9981 Acked Read 6 Sent Done Done false
35665 9982 Acked Read 4 Sent Done Done false
35666 9983 Acked Read 5 Sent Done Done false
35667 9984 Acked Write 9 Sent Done Done false
35668 9985 Acked Read 1 Sent Done Done false
35669 9986 Acked Read 2 Sent Done Done false
35670 9987 Acked Write 6 Sent Done Done false
35671 9988 Acked Read 2 Sent Done Done false
35672 9989 Acked Read 5 Sent Done Done false
35673 9990 Acked Read 10 Sent Done Done false
35674 9991 Acked Read 6 Sent Done Done false
35675 9992 Acked Read 3 Sent Done Done false
35676 9993 Acked Read 3 Sent Done Done false
35677 9994 Acked Read 3 Sent Done Done false
35678 9995 Acked Read 3 Sent Done Done false
35679 9996 Acked Read 3 Sent Done Done false
35680 9997 Acked Flush 0 Sent Done Done false
35681 9998 Acked Read 2 Sent Done Done false
35682 9999 Acked Read 6 Sent Done Done false
35683 10000 Acked Write 4 Sent Done Done false
35684 10001 Acked Read 9 Sent Done Done false
35685 10002 Acked Read 6 Sent Done Done false
35686 10003 Acked Read 1 Sent Done Done false
35687 10004 Acked Read 4 Sent Done Done false
35688 10005 Acked Write 5 Sent Done Done false
35689 10006 Acked Read 7 Sent Done Done false
35690 10007 Acked Write 1 Sent Done Done false
35691 10008 Acked Read 10 Sent Done Done false
35692 10009 Acked Write 2 Sent Done Done false
35693 10010 Acked Read 5 Sent Done Done false
35694 10011 Acked Flush 0 Sent Done Done false
35695 10012 Acked Flush 0 Sent Done Done false
35696 10013 Acked Read 9 Sent Done Done false
35697 10014 Acked Read 2 Sent Done Done false
35698 10015 Acked Read 7 Sent Done Done false
35699 10016 Acked Write 2 Sent Done Done false
35700 10017 Acked Read 9 Sent Done Done false
35701 10018 Acked Read 1 Sent Done Done false
35702 10019 Acked Read 2 Sent Done Done false
35703 10020 Acked Write 8 Sent Done Done false
35704 10021 Acked Write 9 Sent Done Done false
35705 10022 Acked Flush 0 Sent Done Done false
35706 10023 Acked Flush 0 Sent Done Done false
35707 10024 Acked Write 1 Sent Done Done false
35708 10025 Acked Write 6 Sent Done Done false
35709 10026 Acked Read 10 Sent Done Done false
35710 10027 Acked Write 9 Sent Done Done false
35711 10028 Acked Read 1 Sent Done Done false
35712 10029 Acked Flush 0 Sent Done Done false
35713 10030 Acked Write 6 Sent Done Done false
35714 10031 Acked Flush 0 Sent Done Done false
35715 10032 Acked Flush 0 Sent Done Done false
35716 10033 Acked Write 3 Sent Done Done false
35717 10034 Acked Read 7 Sent Done Done false
35718 10035 Acked Read 9 Sent Done Done false
35719 10036 Acked Write 10 Sent Done Done false
35720 10037 Acked Write 9 Sent Done Done false
35721 10038 Acked Read 2 Sent Done Done false
35722 10039 Acked Read 7 Sent Done Done false
35723 10040 Acked Write 3 Sent Done Done false
35724 10041 Acked Write 1 Sent Done Done false
35725 10042 Acked Read 1 Sent Done Done false
35726 10043 Acked Read 10 Sent Done Done false
35727 10044 Acked Read 7 Sent Done Done false
35728 10045 Acked Read 2 Sent Done Done false
35729 10046 Acked Read 10 Sent Done Done false
35730 10047 Acked Write 6 Sent Done Done false
35731 10048 Acked Read 6 Sent Done Done false
35732 10049 Acked Flush 0 Sent Done Done false
35733 10050 Acked Write 9 Sent Done Done false
35734 10051 Acked Write 7 Sent Done Done false
35735 10052 Acked Write 5 Sent Done Done false
35736 10053 Acked Read 3 Sent Done Done false
35737 10054 Acked Read 7 Sent Done Done false
35738 10055 Acked Write 3 Sent Done Done false
35739 STATES DS:0 DS:1 DS:2 TOTAL
35740 Sent 301 0 0 301
35741 Done 0 301 301 602
35742 Skipped 0 0 0 0
35743 Error 0 0 0 0
35744 Last Flush: Some(JobId(9754)) Some(JobId(10049)) Some(JobId(10049))
35745 Downstairs last five completed: 9754 9753 9752 9751 9750
35746 Upstairs last five completed: 10055 10054 10053 10052 10051
35747 CLIENT: Up:0 ds:301 act:2
357482025-03-17T22:42:29.050ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
357492025-03-17T22:42:30.490ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
357502025-03-17T22:42:31.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
357512025-03-17T22:42:32.056ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
35752 [8810] got start action
35753 [8810] I am going to restart
35754 Starting downstairs at port 8810
35755 Make output file at "/tmp/dsc/downstairs-8810.txt"
35756 [8810][0] reports Starting
35757 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1238)
35758 [8810][0] reports Running
35759 ----------------------------------------------------------------
35760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35761 States: Offline Active Active
35762 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35763 9755 Acked Write 2 Sent Done Done false
35764 9756 Acked Read 4 Sent Done Done false
35765 9757 Acked Read 5 Sent Done Done false
35766 9758 Acked Read 2 Sent Done Done false
35767 9759 Acked Write 5 Sent Done Done false
35768 9760 Acked Write 3 Sent Done Done false
35769 9761 Acked Read 5 Sent Done Done false
35770 9762 Acked Read 5 Sent Done Done false
35771 9763 Acked Write 3 Sent Done Done false
35772 9764 Acked Write 9 Sent Done Done false
35773 9765 Acked Read 6 Sent Done Done false
35774 9766 Acked Write 3 Sent Done Done false
35775 9767 Acked Read 6 Sent Done Done false
35776 9768 Acked Read 9 Sent Done Done false
35777 9769 Acked Flush 0 Sent Done Done false
35778 9770 Acked Read 4 Sent Done Done false
35779 9771 Acked Write 1 Sent Done Done false
35780 9772 Acked Write 3 Sent Done Done false
35781 9773 Acked Read 8 Sent Done Done false
35782 9774 Acked Write 5 Sent Done Done false
35783 9775 Acked Read 6 Sent Done Done false
35784 9776 Acked Write 3 Sent Done Done false
35785 9777 Acked Read 8 Sent Done Done false
35786 9778 Acked Read 6 Sent Done Done false
35787 9779 Acked Write 6 Sent Done Done false
35788 9780 Acked Read 6 Sent Done Done false
35789 9781 Acked Flush 0 Sent Done Done false
35790 9782 Acked Write 6 Sent Done Done false
35791 9783 Acked Flush 0 Sent Done Done false
35792 9784 Acked Read 7 Sent Done Done false
35793 9785 Acked Read 7 Sent Done Done false
35794 9786 Acked Read 10 Sent Done Done false
35795 9787 Acked Read 4 Sent Done Done false
35796 9788 Acked Write 3 Sent Done Done false
35797 9789 Acked Write 1 Sent Done Done false
35798 9790 Acked Read 10 Sent Done Done false
35799 9791 Acked Flush 0 Sent Done Done false
35800 9792 Acked Read 2 Sent Done Done false
35801 9793 Acked Read 4 Sent Done Done false
35802 9794 Acked Read 7 Sent Done Done false
35803 9795 Acked Read 4 Sent Done Done false
35804 9796 Acked Read 2 Sent Done Done false
35805 9797 Acked Read 5 Sent Done Done false
35806 9798 Acked Read 10 Sent Done Done false
35807 9799 Acked Write 4 Sent Done Done false
35808 9800 Acked Read 3 Sent Done Done false
35809 9801 Acked Read 2 Sent Done Done false
35810 9802 Acked Write 3 Sent Done Done false
35811 9803 Acked Read 9 Sent Done Done false
35812 9804 Acked Read 7 Sent Done Done false
35813 9805 Acked Write 6 Sent Done Done false
35814 9806 Acked Write 9 Sent Done Done false
35815 9807 Acked Flush 0 Sent Done Done false
35816 9808 Acked Read 5 Sent Done Done false
35817 9809 Acked Flush 0 Sent Done Done false
35818 9810 Acked Read 7 Sent Done Done false
35819 9811 Acked Write 3 Sent Done Done false
35820 9812 Acked Write 9 Sent Done Done false
35821 9813 Acked Read 4 Sent Done Done false
35822 9814 Acked Write 2 Sent Done Done false
35823 9815 Acked Read 6 Sent Done Done false
35824 9816 Acked Write 3 Sent Done Done false
35825 9817 Acked Read 3 Sent Done Done false
35826 9818 Acked Read 1 Sent Done Done false
35827 9819 Acked Write 8 Sent Done Done false
35828 9820 Acked Write 10 Sent Done Done false
35829 9821 Acked Read 10 Sent Done Done false
35830 9822 Acked Flush 0 Sent Done Done false
35831 9823 Acked Read 8 Sent Done Done false
35832 9824 Acked Read 4 Sent Done Done false
35833 9825 Acked Write 4 Sent Done Done false
35834 9826 Acked Write 10 Sent Done Done false
35835 9827 Acked Read 9 Sent Done Done false
35836 9828 Acked Read 5 Sent Done Done false
35837 9829 Acked Write 6 Sent Done Done false
35838 9830 Acked Write 3 Sent Done Done false
35839 9831 Acked Read 9 Sent Done Done false
35840 9832 Acked Write 7 Sent Done Done false
35841 9833 Acked Write 6 Sent Done Done false
35842 9834 Acked Read 4 Sent Done Done false
35843 9835 Acked Read 7 Sent Done Done false
35844 9836 Acked Write 2 Sent Done Done false
35845 9837 Acked Read 6 Sent Done Done false
35846 9838 Acked Read 2 Sent Done Done false
35847 9839 Acked Write 5 Sent Done Done false
35848 9840 Acked Read 1 Sent Done Done false
35849 9841 Acked Read 5 Sent Done Done false
35850 9842 Acked Read 2 Sent Done Done false
35851 9843 Acked Write 3 Sent Done Done false
35852 9844 Acked Read 6 Sent Done Done false
35853 9845 Acked Read 10 Sent Done Done false
35854 9846 Acked Write 3 Sent Done Done false
35855 9847 Acked Read 8 Sent Done Done false
35856 9848 Acked Read 4 Sent Done Done false
35857 9849 Acked Read 9 Sent Done Done false
35858 9850 Acked Write 10 Sent Done Done false
35859 9851 Acked Write 6 Sent Done Done false
35860 9852 Acked Write 6 Sent Done Done false
35861 9853 Acked Read 2 Sent Done Done false
35862 9854 Acked Read 3 Sent Done Done false
35863 9855 Acked Read 4 Sent Done Done false
35864 9856 Acked Read 7 Sent Done Done false
35865 9857 Acked Read 5 Sent Done Done false
35866 9858 Acked Read 8 Sent Done Done false
35867 9859 Acked Read 6 Sent Done Done false
35868 9860 Acked Write 7 Sent Done Done false
35869 9861 Acked Write 2 Sent Done Done false
35870 9862 Acked Read 1 Sent Done Done false
35871 9863 Acked Write 10 Sent Done Done false
35872 9864 Acked Read 6 Sent Done Done false
35873 9865 Acked Read 1 Sent Done Done false
35874 9866 Acked Write 9 Sent Done Done false
35875 9867 Acked Read 5 Sent Done Done false
35876 9868 Acked Read 4 Sent Done Done false
35877 9869 Acked Read 1 Sent Done Done false
35878 9870 Acked Write 7 Sent Done Done false
35879 9871 Acked Flush 0 Sent Done Done false
35880 9872 Acked Write 8 Sent Done Done false
35881 9873 Acked Read 6 Sent Done Done false
35882 9874 Acked Write 10 Sent Done Done false
35883 9875 Acked Read 5 Sent Done Done false
35884 9876 Acked Write 3 Sent Done Done false
35885 9877 Acked Write 6 Sent Done Done false
35886 9878 Acked Read 2 Sent Done Done false
35887 9879 Acked Flush 0 Sent Done Done false
35888 9880 Acked Write 6 Sent Done Done false
35889 9881 Acked Write 1 Sent Done Done false
35890 9882 Acked Read 4 Sent Done Done false
35891 9883 Acked Write 5 Sent Done Done false
35892 9884 Acked Read 3 Sent Done Done false
35893 9885 Acked Write 2 Sent Done Done false
35894 9886 Acked Write 6 Sent Done Done false
35895 9887 Acked Write 5 Sent Done Done false
35896 9888 Acked Write 10 Sent Done Done false
35897 9889 Acked Read 10 Sent Done Done false
35898 9890 Acked Read 5 Sent Done Done false
35899 9891 Acked Write 10 Sent Done Done false
35900 9892 Acked Write 6 Sent Done Done false
35901 9893 Acked Read 3 Sent Done Done false
35902 9894 Acked Write 10 Sent Done Done false
35903 9895 Acked Write 4 Sent Done Done false
35904 9896 Acked Read 2 Sent Done Done false
35905 9897 Acked Read 4 Sent Done Done false
35906 9898 Acked Flush 0 Sent Done Done false
35907 9899 Acked Write 2 Sent Done Done false
35908 9900 Acked Read 2 Sent Done Done false
35909 9901 Acked Write 5 Sent Done Done false
35910 9902 Acked Write 4 Sent Done Done false
35911 9903 Acked Read 2 Sent Done Done false
35912 9904 Acked Flush 0 Sent Done Done false
35913 9905 Acked Read 10 Sent Done Done false
35914 9906 Acked Write 9 Sent Done Done false
35915 9907 Acked Write 1 Sent Done Done false
35916 9908 Acked Write 3 Sent Done Done false
35917 9909 Acked Flush 0 Sent Done Done false
35918 9910 Acked Read 2 Sent Done Done false
35919 9911 Acked Read 4 Sent Done Done false
35920 9912 Acked Read 7 Sent Done Done false
35921 9913 Acked Read 5 Sent Done Done false
35922 9914 Acked Read 8 Sent Done Done false
35923 9915 Acked Write 2 Sent Done Done false
35924 9916 Acked Flush 0 Sent Done Done false
35925 9917 Acked Write 1 Sent Done Done false
35926 9918 Acked Write 5 Sent Done Done false
35927 9919 Acked Read 1 Sent Done Done false
35928 9920 Acked Read 6 Sent Done Done false
35929 9921 Acked Write 3 Sent Done Done false
35930 9922 Acked Write 6 Sent Done Done false
35931 9923 Acked Read 4 Sent Done Done false
35932 9924 Acked Read 3 Sent Done Done false
35933 9925 Acked Read 10 Sent Done Done false
35934 9926 Acked Write 7 Sent Done Done false
35935 9927 Acked Write 3 Sent Done Done false
35936 9928 Acked Write 6 Sent Done Done false
35937 9929 Acked Read 8 Sent Done Done false
35938 9930 Acked Write 7 Sent Done Done false
35939 9931 Acked Write 10 Sent Done Done false
35940 9932 Acked Write 7 Sent Done Done false
35941 9933 Acked Read 1 Sent Done Done false
35942 9934 Acked Flush 0 Sent Done Done false
35943 9935 Acked Write 8 Sent Done Done false
35944 9936 Acked Flush 0 Sent Done Done false
35945 9937 Acked Read 7 Sent Done Done false
35946 9938 Acked Read 5 Sent Done Done false
35947 9939 Acked Read 5 Sent Done Done false
35948 9940 Acked Write 3 Sent Done Done false
35949 9941 Acked Read 5 Sent Done Done false
35950 9942 Acked Write 6 Sent Done Done false
35951 9943 Acked Write 1 Sent Done Done false
35952 9944 Acked Write 3 Sent Done Done false
35953 9945 Acked Write 5 Sent Done Done false
35954 9946 Acked Read 6 Sent Done Done false
35955 9947 Acked Read 10 Sent Done Done false
35956 9948 Acked Read 2 Sent Done Done false
35957 9949 Acked Read 8 Sent Done Done false
35958 9950 Acked Flush 0 Sent Done Done false
35959 9951 Acked Write 4 Sent Done Done false
35960 9952 Acked Write 7 Sent Done Done false
35961 9953 Acked Write 3 Sent Done Done false
35962 9954 Acked Read 5 Sent Done Done false
35963 9955 Acked Read 9 Sent Done Done false
35964 9956 Acked Write 6 Sent Done Done false
35965 9957 Acked Read 8 Sent Done Done false
35966 9958 Acked Write 10 Sent Done Done false
35967 9959 Acked Read 5 Sent Done Done false
35968 9960 Acked Write 2 Sent Done Done false
35969 9961 Acked Read 8 Sent Done Done false
35970 9962 Acked Flush 0 Sent Done Done false
35971 9963 Acked Read 1 Sent Done Done false
35972 9964 Acked Write 5 Sent Done Done false
35973 9965 Acked Flush 0 Sent Done Done false
35974 9966 Acked Read 8 Sent Done Done false
35975 9967 Acked Flush 0 Sent Done Done false
35976 9968 Acked Read 8 Sent Done Done false
35977 9969 Acked Write 9 Sent Done Done false
35978 9970 Acked Write 6 Sent Done Done false
35979 9971 Acked Read 4 Sent Done Done false
35980 9972 Acked Write 3 Sent Done Done false
35981 9973 Acked Read 6 Sent Done Done false
35982 9974 Acked Flush 0 Sent Done Done false
35983 9975 Acked Flush 0 Sent Done Done false
35984 9976 Acked Write 1 Sent Done Done false
35985 9977 Acked Write 9 Sent Done Done false
35986 9978 Acked Flush 0 Sent Done Done false
35987 9979 Acked Write 1 Sent Done Done false
35988 9980 Acked Write 10 Sent Done Done false
35989 9981 Acked Read 6 Sent Done Done false
35990 9982 Acked Read 4 Sent Done Done false
35991 9983 Acked Read 5 Sent Done Done false
35992 9984 Acked Write 9 Sent Done Done false
35993 9985 Acked Read 1 Sent Done Done false
35994 9986 Acked Read 2 Sent Done Done false
35995 9987 Acked Write 6 Sent Done Done false
35996 9988 Acked Read 2 Sent Done Done false
35997 9989 Acked Read 5 Sent Done Done false
35998 9990 Acked Read 10 Sent Done Done false
35999 9991 Acked Read 6 Sent Done Done false
36000 9992 Acked Read 3 Sent Done Done false
36001 9993 Acked Read 3 Sent Done Done false
36002 9994 Acked Read 3 Sent Done Done false
36003 9995 Acked Read 3 Sent Done Done false
36004 9996 Acked Read 3 Sent Done Done false
36005 9997 Acked Flush 0 Sent Done Done false
36006 9998 Acked Read 2 Sent Done Done false
36007 9999 Acked Read 6 Sent Done Done false
36008 10000 Acked Write 4 Sent Done Done false
36009 10001 Acked Read 9 Sent Done Done false
36010 10002 Acked Read 6 Sent Done Done false
36011 10003 Acked Read 1 Sent Done Done false
36012 10004 Acked Read 4 Sent Done Done false
36013 10005 Acked Write 5 Sent Done Done false
36014 10006 Acked Read 7 Sent Done Done false
36015 10007 Acked Write 1 Sent Done Done false
36016 10008 Acked Read 10 Sent Done Done false
36017 10009 Acked Write 2 Sent Done Done false
36018 10010 Acked Read 5 Sent Done Done false
36019 10011 Acked Flush 0 Sent Done Done false
36020 10012 Acked Flush 0 Sent Done Done false
36021 10013 Acked Read 9 Sent Done Done false
36022 10014 Acked Read 2 Sent Done Done false
36023 10015 Acked Read 7 Sent Done Done false
36024 10016 Acked Write 2 Sent Done Done false
36025 10017 Acked Read 9 Sent Done Done false
36026 10018 Acked Read 1 Sent Done Done false
36027 10019 Acked Read 2 Sent Done Done false
36028 10020 Acked Write 8 Sent Done Done false
36029 10021 Acked Write 9 Sent Done Done false
36030 10022 Acked Flush 0 Sent Done Done false
36031 10023 Acked Flush 0 Sent Done Done false
36032 10024 Acked Write 1 Sent Done Done false
36033 10025 Acked Write 6 Sent Done Done false
36034 10026 Acked Read 10 Sent Done Done false
36035 10027 Acked Write 9 Sent Done Done false
36036 10028 Acked Read 1 Sent Done Done false
36037 10029 Acked Flush 0 Sent Done Done false
36038 10030 Acked Write 6 Sent Done Done false
36039 10031 Acked Flush 0 Sent Done Done false
36040 10032 Acked Flush 0 Sent Done Done false
36041 10033 Acked Write 3 Sent Done Done false
36042 10034 Acked Read 7 Sent Done Done false
36043 10035 Acked Read 9 Sent Done Done false
36044 10036 Acked Write 10 Sent Done Done false
36045 10037 Acked Write 9 Sent Done Done false
36046 10038 Acked Read 2 Sent Done Done false
36047 10039 Acked Read 7 Sent Done Done false
36048 10040 Acked Write 3 Sent Done Done false
36049 10041 Acked Write 1 Sent Done Done false
36050 10042 Acked Read 1 Sent Done Done false
36051 10043 Acked Read 10 Sent Done Done false
36052 10044 Acked Read 7 Sent Done Done false
36053 10045 Acked Read 2 Sent Done Done false
36054 10046 Acked Read 10 Sent Done Done false
36055 10047 Acked Write 6 Sent Done Done false
36056 10048 Acked Read 6 Sent Done Done false
36057 10049 Acked Flush 0 Sent Done Done false
36058 10050 Acked Write 9 Sent Done Done false
36059 10051 Acked Write 7 Sent Done Done false
36060 10052 Acked Write 5 Sent Done Done false
36061 10053 Acked Read 3 Sent Done Done false
36062 10054 Acked Read 7 Sent Done Done false
36063 10055 Acked Write 3 Sent Done Done false
36064 STATES DS:0 DS:1 DS:2 TOTAL
36065 Sent 301 0 0 301
36066 Done 0 301 301 602
36067 Skipped 0 0 0 0
36068 Error 0 0 0 0
36069 Last Flush: Some(JobId(9754)) Some(JobId(10049)) Some(JobId(10049))
36070 Downstairs last five completed: 9754 9753 9752 9751 9750
36071 Upstairs last five completed: 10055 10054 10053 10052 10051
36072 CLIENT: Up:0 ds:301 act:2
360732025-03-17T22:42:33.227ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360742025-03-17T22:42:34.059ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360752025-03-17T22:42:34.244ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360762025-03-17T22:42:34.245ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39927) = io task client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360772025-03-17T22:42:34.245ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360782025-03-17T22:42:34.246ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360792025-03-17T22:42:34.246ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360802025-03-17T22:42:34.246ZINFOcrucible: send last flush ID to this DS: Some(JobId(9754)) = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360812025-03-17T22:42:34.246ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360822025-03-17T22:42:34.246ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9754)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360832025-03-17T22:42:34.246ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9754)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
36084 ----------------------------------------------------------------
36085 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36086 Downstairs last five completed: 10056 10055 10054 10053 10052
36087 Upstairs last five completed: 10056 10055 10054 10053 10052
36088 CLIENT: Up:0 ds:0 act:3
36089 Replay: All jobs finished
36090 Main task has work to do, go find it
36091 got dsc Stop(1)
36092 stop 1
36093 Mar 17 22:42:36.492 INFO request completed, latency_us: 449[8820] Got stop action so:false kr:false
36094 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: f97bac99-c811-4531-b43e-42d4a07a821b, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
360952025-03-17T22:42:36.493ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360962025-03-17T22:42:36.493ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
360972025-03-17T22:42:36.493ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
36098 Mar 17 22:42:36.493 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/1, method: GET, req_id: a2451dac-59ff-44de-9c25-d5280f24b63b, remote_addr: 127.0.0.1:35180{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:42:36.493799726Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"34ae9076-d4ea-416d-9168-6f1a5fc19802"}
36099 , local_addr: 127.0.0.1:9998
36100 [8820] Exited with: ExitStatus(unix_wait_status(9))
36101 [8820][1] reports Exit
36102 Mar 17 22:42:40.495 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 32bc9f26-a11a-4985-85c4-3f7589abb6b8, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
36103 001/300 Write block 1756 len 4096 data: 5
36104 002/300 Flush
36105 003/300 Flush
36106 004/300 Write block 2562 len 8192 data: 8 9
36107 005/300 Write block 1520 len 4096 data: 5
36108 006/300 Read block 802 len 40960
36109 007/300 Write block 4329 len 4096 data: 5
36110 008/300 Read block 576 len 4096
36111 009/300 Write block 2073 len 8192 data: 8 9
36112 010/300 Write block 1266 len 40960 data: 7 8 4 6 6 7 6 7 7 7
36113 011/300 Read block 3189 len 12288
36114 012/300 Read block 573 len 8192
36115 013/300 Read block 3688 len 20480
36116 014/300 Read block 187 len 28672
36117 015/300 Read block 2219 len 32768
36118 016/300 Read block 2659 len 24576
36119 017/300 Read block 2944 len 32768
36120 018/300 Write block 4062 len 16384 data: 4 5 6 6
36121 019/300 Read block 4479 len 32768
36122 020/300 Read block 4292 len 8192
36123 021/300 Read block 1540 len 32768
36124 022/300 Read block 3044 len 24576
36125 023/300 Read block 1958 len 32768
36126 024/300 Write block 4832 len 20480 data: 6 7 7 7 6
36127 025/300 Write block 4514 len 28672 data: 7 6 7 6 7 9 8
36128 026/300 Read block 2179 len 28672
36129 027/300 Read block 1066 len 8192
36130 028/300 Read block 2084 len 24576
36131 029/300 Read block 1780 len 12288
36132 030/300 Read block 3545 len 8192
36133 031/300 Flush
36134 032/300 Flush
36135 033/300 Flush
36136 034/300 Write block 2522 len 36864 data: 5 5 6 5 4 4 5 4 6
36137 035/300 Flush
36138 036/300 Read block 1533 len 16384
36139 037/300 Write block 3169 len 20480 data: 4 3 3 2 3
36140 038/300 Read block 2254 len 24576
36141 039/300 Write block 3975 len 36864 data: 4 5 4 4 4 5 5 5 5
36142 040/300 Read block 3054 len 4096
36143 041/300 Read block 4919 len 24576
36144 042/300 Read block 4089 len 16384
36145 043/300 Flush
36146 044/300 Read block 1437 len 24576
36147 045/300 Write block 1858 len 4096 data: 7
36148 046/300 Read block 4847 len 40960
36149 047/300 Write block 1721 len 24576 data: 9 8 9 7 8 8
36150 048/300 Read block 780 len 24576
36151 049/300 Read block 1470 len 32768
36152 050/300 Write block 1812 len 16384 data: 6 4 3 4
36153 051/300 Read block 1311 len 32768
36154 052/300 Read block 2717 len 20480
36155 053/300 Read block 991 len 8192
36156 054/300 Flush
36157 055/300 Read block 2496 len 24576
36158 056/300 Read block 555 len 12288
36159 057/300 Write block 2785 len 36864 data: 7 6 6 8 7 8 8 8 5
36160 058/300 Write block 2545 len 12288 data: 6 8 9
36161 059/300 Read block 2730 len 28672
36162 060/300 Write block 1635 len 24576 data: 7 7 7 9 9 10
36163 061/300 Read block 1621 len 28672
36164 062/300 Read block 4992 len 28672
36165 063/300 Read block 4592 len 32768
36166 064/300 Read block 363 len 40960
36167 065/300 Read block 4093 len 16384
36168 066/300 Read block 432 len 16384
36169 067/300 Write block 2454 len 32768 data: 6 6 7 5 4 4 5 5
36170 068/300 Read block 807 len 24576
36171 069/300 Write block 772 len 20480 data: 4 4 5 6 6
36172 070/300 Write block 4335 len 20480 data: 5 5 5 5 7
36173 071/300 Write block 1198 len 40960 data: 5 5 5 3 3 4 5 4 4 5
36174 072/300 Read block 4455 len 16384
36175 073/300 Read block 1724 len 36864
36176 074/300 Read block 357 len 12288
36177 075/300 Write block 4138 len 36864 data: 7 7 7 7 10 9 8 9 7
36178 076/300 Write block 3339 len 8192 data: 9 11
36179 077/300 Flush
36180 078/300 Write block 4569 len 36864 data: 6 4 5 4 5 5 8 8 6
36181 079/300 Flush
36182 080/300 Write block 1892 len 32768 data: 4 5 6 6 7 7 6 5
36183 081/300 Flush
36184 082/300 Read block 2936 len 24576
36185 083/300 Write block 2385 len 32768 data: 4 4 4 3 5 5 5 5
36186 084/300 Write block 2292 len 40960 data: 4 4 6 7 8 6 7 7 8 6
36187 085/300 Flush
36188 086/300 Read block 1653 len 16384
36189 087/300 Read block 3926 len 36864
36190 088/300 Read block 3695 len 8192
36191 089/300 Read block 3943 len 8192
36192 090/300 Read block 2906 len 28672
36193 091/300 Flush
36194 092/300 Write block 546 len 40960 data: 7 5 5 6 7 8 7 6 7 7
36195 093/300 Read block 1954 len 28672
36196 094/300 Write block 1570 len 28672 data: 5 5 5 5 5 6 4
36197 095/300 Write block 4041 len 8192 data: 8 8
36198 096/300 Read block 2240 len 28672
36199 097/300 Flush
36200 098/300 Write block 2590 len 12288 data: 2 2 3
36201 099/300 Write block 729 len 32768 data: 11 9 10 9 10 9 8 8
36202 100/300 Flush
36203 101/300 Flush
36204 102/300 Read block 105 len 20480
36205 103/300 Flush
36206 104/300 Read block 2740 len 32768
36207 105/300 Read block 1118 len 28672
36208 106/300 Read block 2363 len 32768
36209 107/300 Read block 4616 len 12288
36210 108/300 Read block 269 len 28672
36211 109/300 Write block 1436 len 40960 data: 10 8 6 7 6 4 6 7 5 5
36212 110/300 Write block 3378 len 32768 data: 5 6 7 8 9 9 11 10
36213 111/300 Read block 3345 len 8192
36214 112/300 Write block 1300 len 24576 data: 4 6 7 6 6 5
36215 113/300 Write block 2035 len 40960 data: 9 8 9 10 10 11 10 9 8 6
36216 114/300 Read block 878 len 8192
36217 115/300 Read block 327 len 4096
36218 116/300 Flush
36219 117/300 Read block 1233 len 40960
36220 118/300 Write block 4353 len 40960 data: 10 11 10 10 10 7 4 4 4 4
36221 119/300 Read block 222 len 28672
36222 120/300 Write block 874 len 12288 data: 6 7 8
36223 121/300 Read block 1465 len 32768
36224 122/300 Flush
36225 123/300 Write block 1311 len 36864 data: 5 4 6 6 5 5 4 4 7
36226 124/300 Flush
36227 125/300 Write block 4911 len 24576 data: 8 7 7 6 7 8
36228 126/300 Write block 893 len 20480 data: 4 5 5 4 4
36229 127/300 Write block 4533 len 40960 data: 9 7 9 9 9 8 8 7 8 6
36230 128/300 Read block 3595 len 12288
36231 129/300 Read block 3004 len 4096
36232 130/300 Read block 3331 len 36864
36233 131/300 Write block 4618 len 4096 data: 4
36234 132/300 Write block 1962 len 40960 data: 11 8 5 5 5 4 4 5 4 4
36235 133/300 Flush
36236 134/300 Write block 958 len 8192 data: 2 4
36237 135/300 Read block 444 len 24576
36238 136/300 Read block 1910 len 32768
36239 137/300 Read block 2047 len 12288
36240 138/300 Write block 3828 len 12288 data: 5 3 3
36241 139/300 Read block 3099 len 32768
36242 140/300 Write block 4803 len 24576 data: 9 9 8 8 8 7
36243 141/300 Read block 3313 len 16384
36244 142/300 Flush
36245 143/300 Read block 1425 len 36864
36246 144/300 Read block 246 len 20480
36247 145/300 Read block 391 len 32768
36248 146/300 Flush
36249 147/300 Read block 2202 len 32768
36250 148/300 Read block 3497 len 36864
36251 149/300 Read block 3789 len 40960
36252 150/300 Read block 798 len 24576
36253 151/300 Read block 1234 len 40960
36254 152/300 Read block 260 len 4096
36255 153/300 Read block 774 len 12288
36256 154/300 Read block 3042 len 4096
36257 155/300 Read block 1891 len 24576
36258 156/300 Read block 3653 len 8192
36259 157/300 Write block 1795 len 8192 data: 4 4
36260 158/300 Write block 3902 len 24576 data: 6 5 3 5 5 7
36261 159/300 Read block 1681 len 24576
36262 160/300 Write block 1235 len 28672 data: 7 6 5 6 5 4 4
36263 161/300 Flush
36264 162/300 Read block 2275 len 36864
36265 163/300 Write block 1621 len 4096 data: 6
36266 164/300 Read block 547 len 32768
36267 165/300 Flush
36268 166/300 Write block 744 len 20480 data: 6 8 8 8 12
36269 167/300 Read block 1214 len 8192
36270 168/300 Write block 3211 len 24576 data: 12 13 12 11 10 7
36271 169/300 Write block 2047 len 36864 data: 4 6 6 7 5 7 7 8 6
36272 170/300 Flush
36273 171/300 Read block 2318 len 24576
36274 172/300 Read block 689 len 16384
36275 173/300 Write block 1091 len 16384 data: 7 7 5 5
36276 174/300 Write block 1180 len 12288 data: 4 5 4
36277 175/300 Read block 709 len 12288
36278 176/300 Read block 89 len 24576
36279 177/300 Write block 143 len 32768 data: 7 5 5 5 5 5 6 5
36280 178/300 Flush
36281 179/300 Write block 3895 len 36864 data: 8 9 6 5 5 8 9 7 6
36282 180/300 Write block 4352 len 16384 data: 6 11 12 11
36283 181/300 Read block 1188 len 32768
36284 182/300 Write block 21 len 32768 data: 3 4 8 8 7 6 6 6
36285 183/300 Write block 3312 len 8192 data: 3 2
36286 184/300 Read block 38 len 40960
36287 185/300 Write block 1031 len 4096 data: 4
36288 186/300 Write block 3068 len 8192 data: 7 8
36289 187/300 Read block 1660 len 16384
36290 188/300 Read block 1657 len 28672
36291 189/300 Read block 4270 len 12288
36292 190/300 Write block 2631 len 40960 data: 6 6 6 5 6 4 4 4 5 5
36293 191/300 Flush
36294 192/300 Read block 1664 len 40960
36295 193/300 Write block 2999 len 40960 data: 6 6 6 6 5 5 5 5 5 5
36296 194/300 Write block 1017 len 4096 data: 4
36297 195/300 Write block 1747 len 12288 data: 6 6 5
36298 196/300 Write block 2298 len 8192 data: 8 8
36299 197/300 Write block 2360 len 28672 data: 5 4 5 4 3 3 3
36300 198/300 Read block 4201 len 24576
36301 199/300 Read block 4708 len 20480
36302 200/300 Write block 3102 len 8192 data: 4 4
36303 201/300 Flush
36304 202/300 Write block 4784 len 40960 data: 6 6 8 7 7 7 8 8 8 8
36305 203/300 Write block 3824 len 28672 data: 8 8 6 5 6 4 4
36306 204/300 Read block 4306 len 40960
36307 205/300 Read block 4282 len 8192
36308 206/300 Read block 3709 len 40960
36309 207/300 Write block 2077 len 32768 data: 6 6 7 7 6 5 6 6
36310 208/300 Write block 2783 len 24576 data: 6 5 8 7 7 9
36311 209/300 Read block 502 len 40960
36312 210/300 Write block 2697 len 36864 data: 7 8 7 7 6 5 5 5 4
36313 211/300 Read block 795 len 4096
36314 212/300 Read block 1168 len 24576
36315 213/300 Read block 1905 len 8192
36316 214/300 Read block 4627 len 20480
36317 215/300 Read block 3412 len 32768
36318 216/300 Flush
36319 217/300 Read block 2959 len 40960
36320 218/300 Read block 1413 len 4096
36321 219/300 Read block 2487 len 16384
36322 220/300 Read block 1039 len 8192
36323 221/300 Write block 4867 len 20480 data: 8 7 8 7 8
36324 222/300 Read block 1120 len 20480
36325 223/300 Read block 3635 len 36864
36326 224/300 Write block 4464 len 32768 data: 7 7 5 4 3 2 3 3
36327 225/300 Write block 1420 len 28672 data: 4 4 4 5 7 7 6
36328 226/300 Write block 2002 len 8192 data: 9 9
36329 227/300 Write block 4022 len 4096 data: 9
36330 228/300 Write block 4578 len 12288 data: 7 6 6
36331 229/300 Flush
36332 230/300 Flush
36333 231/300 Read block 1748 len 40960
36334 232/300 Read block 4256 len 16384
36335 233/300 Write block 515 len 28672 data: 4 4 6 6 6 7 7
36336 234/300 Read block 3611 len 20480
36337 235/300 Read block 986 len 12288
36338 236/300 Read block 2483 len 40960
36339 237/300 Read block 3344 len 36864
36340 238/300 Write block 2831 len 24576 data: 5 9 8 10 9 8
36341 239/300 Read block 3182 len 40960
36342 240/300 Read block 2720 len 36864
36343 241/300 Read block 4344 len 28672
36344 242/300 Read block 805 len 20480
36345 243/300 Write block 1147 len 28672 data: 4 4 5 7 7 8 6
36346 244/300 Read block 446 len 8192
36347 245/300 Read block 737 len 24576
36348 246/300 Write block 3167 len 28672 data: 4 4 5 4 4 3 4
36349 247/300 Read block 1012 len 28672
36350 248/300 Read block 597 len 40960
36351 249/300 Read block 4101 len 4096
36352 250/300 Flush
36353 251/300 Write block 3424 len 40960 data: 11 11 11 9 8 9 8 5 6 10
36354 252/300 Read block 1520 len 40960
36355 253/300 Write block 548 len 20480 data: 6 7 8 9 8
36356 254/300 Write block 4520 len 36864 data: 9 7 8 7 7 7 5 8 9
36357 255/300 Flush
36358 256/300 Read block 3084 len 24576
36359 257/300 Write block 629 len 40960 data: 2 2 3 3 2 2 2 2 2 2
36360 258/300 Write block 3678 len 36864 data: 9 10 9 10 9 7 7 4 5
36361 259/300 Write block 585 len 12288 data: 4 4 4
36362 260/300 Read block 1838 len 4096
36363 261/300 Flush
36364 262/300 Read block 1798 len 32768
36365 263/300 Write block 1280 len 28672 data: 9 7 7 8 8 6 6
36366 264/300 Write block 3251 len 4096 data: 7
36367 265/300 Read block 3424 len 32768
36368 266/300 Write block 2926 len 36864 data: 4 4 4 5 6 4 4 4 4
36369 267/300 Write block 880 len 8192 data: 6 6
36370 268/300 Write block 204 len 24576 data: 4 3 4 5 6 6
36371 269/300 Read block 3002 len 4096
36372 270/300 Read block 4293 len 40960
36373 271/300 Read block 4561 len 8192
36374 272/300 Read block 4177 len 16384
36375 273/300 Write block 3831 len 36864 data: 3 4 4 5 6 5 7 6 9
36376 274/300 Read block 567 len 8192
36377 275/300 Write block 4694 len 36864 data: 7 7 8 7 9 8 13 12 12
36378 276/300 Read block 4522 len 16384
36379 277/300 Write block 3714 len 28672 data: 7 7 8 6 7 11 10
36380 278/300 Read block 2865 len 28672
36381 279/300 Write block 2887 len 20480 data: 6 6 6 6 6
36382 280/300 Flush
36383 281/300 Read block 3867 len 40960
36384 282/300 Write block 334 len 16384 data: 4 5 5 5
36385 283/300 Write block 3777 len 8192 data: 4 4
36386 284/300 Read block 2169 len 36864
36387 285/300 Write block 3143 len 20480 data: 3 5 5 5 5
36388 286/300 Write block 3199 len 40960 data: 5 5 4 5 6 8 8 7 9 10
36389 287/300 Read block 461 len 8192
36390 288/300 Write block 3869 len 12288 data: 7 8 9
36391 289/300 Write block 4394 len 16384 data: 5 5 5 5
36392 290/300 Read block 3867 len 32768
36393 291/300 Read block 185 len 32768
36394 292/300 Read block 440 len 8192
36395 293/300 Write block 3328 len 8192 data: 4 5
36396 294/300 Write block 437 len 36864 data: 3 4 6 9 9 8 9 10 9
36397 295/300 Flush
36398 296/300 Write block 73 len 16384 data: 2 3 4 5
36399 297/300 Read block 4816 len 36864
36400 298/300 Read block 3490 len 20480
36401 299/300 Read block 244 len 4096
36402 300/300 Read block 2093 len 36864
36403 301/300 Write block 3341 len 4096 data: 9
36404 Main task has work to do, go find it
36405 got dsc Start(1)
36406 start 1
36407 Mar 17 22:42:40.752 INFO request completed, latency_us: 249, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4bd49398-4153-4dc9-bb3f-310127ccf086, remote_addr: 127.0.0.1:35180, local_addr: 127.0.0.1:9998
36408 Replay: started 1, returned:Ok(())
36409 ----------------------------------------------------------------
36410 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36411 States: Active Offline Active
36412 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36413 10057 Acked Write 1 Done Sent Done false
36414 10058 Acked Flush 0 Done Sent Done false
36415 10059 Acked Flush 0 Done Sent Done false
36416 10060 Acked Write 2 Done Sent Done false
36417 10061 Acked Write 1 Done Sent Done false
36418 10062 Acked Read 10 Done Sent Done false
36419 10063 Acked Write 1 Done Sent Done false
36420 10064 Acked Read 1 Done Sent Done false
36421 10065 Acked Write 2 Done Sent Done false
36422 10066 Acked Write 10 Done Sent Done false
36423 10067 Acked Read 3 Done Sent Done false
36424 10068 Acked Read 2 Done Sent Done false
36425 10069 Acked Read 5 Done Sent Done false
36426 10070 Acked Read 7 Done Sent Done false
36427 10071 Acked Read 8 Done Sent Done false
36428 10072 Acked Read 6 Done Sent Done false
36429 10073 Acked Read 8 Done Sent Done false
36430 10074 Acked Write 4 Done Sent Done false
36431 10075 Acked Read 8 Done Sent Done false
36432 10076 Acked Read 2 Done Sent Done false
36433 10077 Acked Read 8 Done Sent Done false
36434 10078 Acked Read 6 Done Sent Done false
36435 10079 Acked Read 8 Done Sent Done false
36436 10080 Acked Write 5 Done Sent Done false
36437 10081 Acked Write 7 Done Sent Done false
36438 10082 Acked Read 7 Done Sent Done false
36439 10083 Acked Read 2 Done Sent Done false
36440 10084 Acked Read 6 Done Sent Done false
36441 10085 Acked Read 3 Done Sent Done false
36442 10086 Acked Read 2 Done Sent Done false
36443 10087 Acked Flush 0 Done Sent Done false
36444 10088 Acked Flush 0 Done Sent Done false
36445 10089 Acked Flush 0 Done Sent Done false
36446 10090 Acked Write 9 Done Sent Done false
36447 10091 Acked Flush 0 Done Sent Done false
36448 10092 Acked Read 4 Done Sent Done false
36449 10093 Acked Write 5 Done Sent Done false
36450 10094 Acked Read 6 Done Sent Done false
36451 10095 Acked Write 9 Done Sent Done false
36452 10096 Acked Read 1 Done Sent Done false
36453 10097 Acked Read 6 Done Sent Done false
36454 10098 Acked Read 4 Done Sent Done false
36455 10099 Acked Flush 0 Done Sent Done false
36456 10100 Acked Read 6 Done Sent Done false
36457 10101 Acked Write 1 Done Sent Done false
36458 10102 Acked Read 10 Done Sent Done false
36459 10103 Acked Write 6 Done Sent Done false
36460 10104 Acked Read 6 Done Sent Done false
36461 10105 Acked Read 8 Done Sent Done false
36462 10106 Acked Write 4 Done Sent Done false
36463 10107 Acked Read 8 Done Sent Done false
36464 10108 Acked Read 5 Done Sent Done false
36465 10109 Acked Read 2 Done Sent Done false
36466 10110 Acked Flush 0 Done Sent Done false
36467 10111 Acked Read 6 Done Sent Done false
36468 10112 Acked Read 3 Done Sent Done false
36469 10113 Acked Write 9 Done Sent Done false
36470 10114 Acked Write 3 Done Sent Done false
36471 10115 Acked Read 7 Done Sent Done false
36472 10116 Acked Write 6 Done Sent Done false
36473 10117 Acked Read 7 Done Sent Done false
36474 10118 Acked Read 7 Done Sent Done false
36475 10119 Acked Read 8 Done Sent Done false
36476 10120 Acked Read 10 Done Sent Done false
36477 10121 Acked Read 4 Done Sent Done false
36478 10122 Acked Read 4 Done Sent Done false
36479 10123 Acked Write 8 Done Sent Done false
36480 10124 Acked Read 6 Done Sent Done false
36481 10125 Acked Write 5 Done Sent Done false
36482 10126 Acked Write 5 Done Sent Done false
36483 10127 Acked Write 10 Done Sent Done false
36484 10128 Acked Read 4 Done Sent Done false
36485 10129 Acked Read 9 Done Sent Done false
36486 10130 Acked Read 3 Done Sent Done false
36487 10131 Acked Write 9 Done Sent Done false
36488 10132 Acked Write 2 Done Sent Done false
36489 10133 Acked Flush 0 Done Sent Done false
36490 10134 Acked Write 9 Done Sent Done false
36491 10135 Acked Flush 0 Done Sent Done false
36492 10136 Acked Write 8 Done Sent Done false
36493 10137 Acked Flush 0 Done Sent Done false
36494 10138 Acked Read 6 Done Sent Done false
36495 10139 Acked Write 8 Done Sent Done false
36496 10140 Acked Write 10 Done Sent Done false
36497 10141 Acked Flush 0 Done Sent Done false
36498 10142 Acked Read 4 Done Sent Done false
36499 10143 Acked Read 9 Done Sent Done false
36500 10144 Acked Read 2 Done Sent Done false
36501 10145 Acked Read 2 Done Sent Done false
36502 10146 Acked Read 7 Done Sent Done false
36503 10147 Acked Flush 0 Done Sent Done false
36504 10148 Acked Write 10 Done Sent Done false
36505 10149 Acked Read 7 Done Sent Done false
36506 10150 Acked Write 7 Done Sent Done false
36507 10151 Acked Write 2 Done Sent Done false
36508 10152 Acked Read 7 Done Sent Done false
36509 10153 Acked Flush 0 Done Sent Done false
36510 10154 Acked Write 3 Done Sent Done false
36511 10155 Acked Write 8 Done Sent Done false
36512 10156 Acked Flush 0 Done Sent Done false
36513 10157 Acked Flush 0 Done Sent Done false
36514 10158 Acked Read 5 Done Sent Done false
36515 10159 Acked Flush 0 Done Sent Done false
36516 10160 Acked Read 8 Done Sent Done false
36517 10161 Acked Read 7 Done Sent Done false
36518 10162 Acked Read 8 Done Sent Done false
36519 10163 Acked Read 3 Done Sent Done false
36520 10164 Acked Read 7 Done Sent Done false
36521 10165 Acked Write 10 Done Sent Done false
36522 10166 Acked Write 8 Done Sent Done false
36523 10167 Acked Read 2 Done Sent Done false
36524 10168 Acked Write 6 Done Sent Done false
36525 10169 Acked Write 10 Done Sent Done false
36526 10170 Acked Read 2 Done Sent Done false
36527 10171 Acked Read 1 Done Sent Done false
36528 10172 Acked Flush 0 Done Sent Done false
36529 10173 Acked Read 10 Done Sent Done false
36530 10174 Acked Write 10 Done Sent Done false
36531 10175 Acked Read 7 Done Sent Done false
36532 10176 Acked Write 3 Done Sent Done false
36533 10177 Acked Read 8 Done Sent Done false
36534 10178 Acked Flush 0 Done Sent Done false
36535 10179 Acked Write 9 Done Sent Done false
36536 10180 Acked Flush 0 Done Sent Done false
36537 10181 Acked Write 6 Done Sent Done false
36538 10182 Acked Write 5 Done Sent Done false
36539 10183 Acked Write 10 Done Sent Done false
36540 10184 Acked Read 3 Done Sent Done false
36541 10185 Acked Read 1 Done Sent Done false
36542 10186 Acked Read 9 Done Sent Done false
36543 10187 Acked Write 1 Done Sent Done false
36544 10188 Acked Write 10 Done Sent Done false
36545 10189 Acked Flush 0 Done Sent Done false
36546 10190 Acked Write 2 Done Sent Done false
36547 10191 Acked Read 6 Done Sent Done false
36548 10192 Acked Read 8 Done Sent Done false
36549 10193 Acked Read 3 Done Sent Done false
36550 10194 Acked Write 3 Done Sent Done false
36551 10195 Acked Read 8 Done Sent Done false
36552 10196 Acked Write 6 Done Sent Done false
36553 10197 Acked Read 4 Done Sent Done false
36554 10198 Acked Flush 0 Done Sent Done false
36555 10199 Acked Read 9 Done Sent Done false
36556 10200 Acked Read 5 Done Sent Done false
36557 10201 Acked Read 8 Done Sent Done false
36558 10202 Acked Flush 0 Done Sent Done false
36559 10203 Acked Read 8 Done Sent Done false
36560 10204 Acked Read 9 Done Sent Done false
36561 10205 Acked Read 10 Done Sent Done false
36562 10206 Acked Read 6 Done Sent Done false
36563 10207 Acked Read 10 Done Sent Done false
36564 10208 Acked Read 1 Done Sent Done false
36565 10209 Acked Read 3 Done Sent Done false
36566 10210 Acked Read 1 Done Sent Done false
36567 10211 Acked Read 6 Done Sent Done false
36568 10212 Acked Read 2 Done Sent Done false
36569 10213 Acked Write 2 Done Sent Done false
36570 10214 Acked Write 6 Done Sent Done false
36571 10215 Acked Read 6 Done Sent Done false
36572 10216 Acked Write 7 Done Sent Done false
36573 10217 Acked Flush 0 Done Sent Done false
36574 10218 Acked Read 9 Done Sent Done false
36575 10219 Acked Write 1 Done Sent Done false
36576 10220 Acked Read 8 Done Sent Done false
36577 10221 Acked Flush 0 Done Sent Done false
36578 10222 Acked Write 5 Done Sent Done false
36579 10223 Acked Read 2 Done Sent Done false
36580 10224 Acked Write 6 Done Sent Done false
36581 10225 Acked Write 9 Done Sent Done false
36582 10226 Acked Flush 0 Done Sent Done false
36583 10227 Acked Read 6 Done Sent Done false
36584 10228 Acked Read 4 Done Sent Done false
36585 10229 Acked Write 4 Done Sent Done false
36586 10230 Acked Write 3 Done Sent Done false
36587 10231 Acked Read 3 Done Sent Done false
36588 10232 Acked Read 6 Done Sent Done false
36589 10233 Acked Write 8 Done Sent Done false
36590 10234 Acked Flush 0 Done Sent Done false
36591 10235 Acked Write 9 Done Sent Done false
36592 10236 Acked Write 4 Done Sent Done false
36593 10237 Acked Read 8 Done Sent Done false
36594 10238 Acked Write 8 Done Sent Done false
36595 10239 Acked Write 2 Done Sent Done false
36596 10240 Acked Read 10 Done Sent Done false
36597 10241 Acked Write 1 Done Sent Done false
36598 10242 Acked Write 2 Done Sent Done false
36599 10243 Acked Read 4 Done Sent Done false
36600 10244 Acked Read 7 Done Sent Done false
36601 10245 Acked Read 3 Done Sent Done false
36602 10246 Acked Write 10 Done Sent Done false
36603 10247 Acked Flush 0 Done Sent Done false
36604 10248 Acked Read 10 Done Sent Done false
36605 10249 Acked Write 10 Done Sent Done false
36606 10250 Acked Write 1 Done Sent Done false
36607 10251 Acked Write 3 Done Sent Done false
36608 10252 Acked Write 2 Done Sent Done false
36609 10253 Acked Write 7 Done Sent Done false
36610 10254 Acked Read 6 Done Sent Done false
36611 10255 Acked Read 5 Done Sent Done false
36612 10256 Acked Write 2 Done Sent Done false
36613 10257 Acked Flush 0 Done Sent Done false
36614 10258 Acked Write 10 Done Sent Done false
36615 10259 Acked Write 7 Done Sent Done false
36616 10260 Acked Read 10 Done Sent Done false
36617 10261 Acked Read 2 Done Sent Done false
36618 10262 Acked Read 10 Done Sent Done false
36619 10263 Acked Write 8 Done Sent Done false
36620 10264 Acked Write 6 Done Sent Done false
36621 10265 Acked Read 10 Done Sent Done false
36622 10266 Acked Write 9 Done Sent Done false
36623 10267 Acked Read 1 Done Sent Done false
36624 10268 Acked Read 6 Done Sent Done false
36625 10269 Acked Read 2 Done Sent Done false
36626 10270 Acked Read 5 Done Sent Done false
36627 10271 Acked Read 8 Done Sent Done false
36628 10272 Acked Flush 0 Done Sent Done false
36629 10273 Acked Read 10 Done Sent Done false
36630 10274 Acked Read 1 Done Sent Done false
36631 10275 Acked Read 4 Done Sent Done false
36632 10276 Acked Read 2 Done Sent Done false
36633 10277 Acked Write 5 Done Sent Done false
36634 10278 Acked Read 5 Done Sent Done false
36635 10279 Acked Read 9 Done Sent Done false
36636 10280 Acked Write 8 Done Sent Done false
36637 10281 Acked Write 7 Done Sent Done false
36638 10282 Acked Write 2 Done Sent Done false
36639 10283 Acked Write 1 Done Sent Done false
36640 10284 Acked Write 3 Done Sent Done false
36641 10285 Acked Flush 0 Done Sent Done false
36642 10286 Acked Flush 0 Done Sent Done false
36643 10287 Acked Read 10 Done Sent Done false
36644 10288 Acked Read 4 Done Sent Done false
36645 10289 Acked Write 7 Done Sent Done false
36646 10290 Acked Read 5 Done Sent Done false
36647 10291 Acked Read 3 Done Sent Done false
36648 10292 Acked Read 10 Done Sent Done false
36649 10293 Acked Read 9 Done Sent Done false
36650 10294 Acked Write 6 Done Sent Done false
36651 10295 Acked Read 10 Done Sent Done false
36652 10296 Acked Read 9 Done Sent Done false
36653 10297 Acked Read 7 Done Sent Done false
36654 10298 Acked Read 5 Done Sent Done false
36655 10299 Acked Write 7 Done Sent Done false
36656 10300 Acked Read 2 Done Sent Done false
36657 10301 Acked Read 6 Done Sent Done false
36658 10302 Acked Write 7 Done Sent Done false
36659 10303 Acked Read 7 Done Sent Done false
36660 10304 Acked Read 10 Done Sent Done false
36661 10305 Acked Read 1 Done Sent Done false
36662 10306 Acked Flush 0 Done Sent Done false
36663 10307 Acked Write 10 Done Sent Done false
36664 10308 Acked Read 10 Done Sent Done false
36665 10309 Acked Write 5 Done Sent Done false
36666 10310 Acked Write 9 Done Sent Done false
36667 10311 Acked Flush 0 Done Sent Done false
36668 10312 Acked Read 6 Done Sent Done false
36669 10313 Acked Write 10 Done Sent Done false
36670 10314 Acked Write 9 Done Sent Done false
36671 10315 Acked Write 3 Done Sent Done false
36672 10316 Acked Read 1 Done Sent Done false
36673 10317 Acked Flush 0 Done Sent Done false
36674 10318 Acked Read 8 Done Sent Done false
36675 10319 Acked Write 7 Done Sent Done false
36676 10320 Acked Write 1 Done Sent Done false
36677 10321 Acked Read 8 Done Sent Done false
36678 10322 Acked Write 9 Done Sent Done false
36679 10323 Acked Write 2 Done Sent Done false
36680 10324 Acked Write 6 Done Sent Done false
36681 10325 Acked Read 1 Done Sent Done false
36682 10326 Acked Read 10 Done Sent Done false
36683 10327 Acked Read 2 Done Sent Done false
36684 10328 Acked Read 4 Done Sent Done false
36685 10329 Acked Write 9 Done Sent Done false
36686 10330 Acked Read 2 Done Sent Done false
36687 10331 Acked Write 9 Done Sent Done false
36688 10332 Acked Read 4 Done Sent Done false
36689 10333 Acked Write 7 Done Sent Done false
36690 10334 Acked Read 7 Done Sent Done false
36691 10335 Acked Write 5 Done Sent Done false
36692 10336 Acked Flush 0 Done Sent Done false
36693 10337 Acked Read 10 Done Sent Done false
36694 10338 Acked Write 4 Done Sent Done false
36695 10339 Acked Write 2 Done Sent Done false
36696 10340 Acked Read 9 Done Sent Done false
36697 10341 Acked Write 5 Done Sent Done false
36698 10342 Acked Write 10 Done Sent Done false
36699 10343 Acked Read 2 Done Sent Done false
36700 10344 Acked Write 3 Done Sent Done false
36701 10345 Acked Write 4 Done Sent Done false
36702 10346 Acked Read 8 Done Sent Done false
36703 10347 Acked Read 8 Done Sent Done false
36704 10348 Acked Read 2 Done Sent Done false
36705 10349 Acked Write 2 Done Sent Done false
36706 10350 Acked Write 9 Done Sent Done false
36707 10351 Acked Flush 0 Done Sent Done false
36708 10352 Acked Write 4 Done Sent Done false
36709 10353 Acked Read 9 Done Sent Done false
36710 10354 Acked Read 5 Done Sent Done false
36711 10355 Acked Read 1 Done Sent Done false
36712 10356 Acked Read 9 Done Sent Done false
36713 10357 Acked Write 1 Done Sent Done false
36714 STATES DS:0 DS:1 DS:2 TOTAL
36715 Sent 0 301 0 301
36716 Done 301 0 301 602
36717 Skipped 0 0 0 0
36718 Error 0 0 0 0
36719 Last Flush: Some(JobId(10351)) Some(JobId(10056)) Some(JobId(10351))
36720 Downstairs last five completed: 10056 10055 10054 10053 10052
36721 Upstairs last five completed: 10357 10356 10355 10354 10353
36722 CLIENT: Up:0 ds:301 act:2
367232025-03-17T22:42:41.254ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
367242025-03-17T22:42:42.068ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
36725 [8820] got start action
36726 [8820] I am going to restart
36727 Starting downstairs at port 8820
36728 Make output file at "/tmp/dsc/downstairs-8820.txt"
36729 [8820][1] reports Starting
36730 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1239)
36731 [8820][1] reports Running
367322025-03-17T22:42:43.070ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
367332025-03-17T22:42:44.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
36734 ----------------------------------------------------------------
36735 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36736 States: Active Offline Active
36737 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36738 10057 Acked Write 1 Done Sent Done false
36739 10058 Acked Flush 0 Done Sent Done false
36740 10059 Acked Flush 0 Done Sent Done false
36741 10060 Acked Write 2 Done Sent Done false
36742 10061 Acked Write 1 Done Sent Done false
36743 10062 Acked Read 10 Done Sent Done false
36744 10063 Acked Write 1 Done Sent Done false
36745 10064 Acked Read 1 Done Sent Done false
36746 10065 Acked Write 2 Done Sent Done false
36747 10066 Acked Write 10 Done Sent Done false
36748 10067 Acked Read 3 Done Sent Done false
36749 10068 Acked Read 2 Done Sent Done false
36750 10069 Acked Read 5 Done Sent Done false
36751 10070 Acked Read 7 Done Sent Done false
36752 10071 Acked Read 8 Done Sent Done false
36753 10072 Acked Read 6 Done Sent Done false
36754 10073 Acked Read 8 Done Sent Done false
36755 10074 Acked Write 4 Done Sent Done false
36756 10075 Acked Read 8 Done Sent Done false
36757 10076 Acked Read 2 Done Sent Done false
36758 10077 Acked Read 8 Done Sent Done false
36759 10078 Acked Read 6 Done Sent Done false
36760 10079 Acked Read 8 Done Sent Done false
36761 10080 Acked Write 5 Done Sent Done false
36762 10081 Acked Write 7 Done Sent Done false
36763 10082 Acked Read 7 Done Sent Done false
36764 10083 Acked Read 2 Done Sent Done false
36765 10084 Acked Read 6 Done Sent Done false
36766 10085 Acked Read 3 Done Sent Done false
36767 10086 Acked Read 2 Done Sent Done false
36768 10087 Acked Flush 0 Done Sent Done false
36769 10088 Acked Flush 0 Done Sent Done false
36770 10089 Acked Flush 0 Done Sent Done false
36771 10090 Acked Write 9 Done Sent Done false
36772 10091 Acked Flush 0 Done Sent Done false
36773 10092 Acked Read 4 Done Sent Done false
36774 10093 Acked Write 5 Done Sent Done false
36775 10094 Acked Read 6 Done Sent Done false
36776 10095 Acked Write 9 Done Sent Done false
36777 10096 Acked Read 1 Done Sent Done false
36778 10097 Acked Read 6 Done Sent Done false
36779 10098 Acked Read 4 Done Sent Done false
36780 10099 Acked Flush 0 Done Sent Done false
36781 10100 Acked Read 6 Done Sent Done false
36782 10101 Acked Write 1 Done Sent Done false
36783 10102 Acked Read 10 Done Sent Done false
36784 10103 Acked Write 6 Done Sent Done false
36785 10104 Acked Read 6 Done Sent Done false
36786 10105 Acked Read 8 Done Sent Done false
36787 10106 Acked Write 4 Done Sent Done false
36788 10107 Acked Read 8 Done Sent Done false
36789 10108 Acked Read 5 Done Sent Done false
36790 10109 Acked Read 2 Done Sent Done false
36791 10110 Acked Flush 0 Done Sent Done false
36792 10111 Acked Read 6 Done Sent Done false
36793 10112 Acked Read 3 Done Sent Done false
36794 10113 Acked Write 9 Done Sent Done false
36795 10114 Acked Write 3 Done Sent Done false
36796 10115 Acked Read 7 Done Sent Done false
36797 10116 Acked Write 6 Done Sent Done false
36798 10117 Acked Read 7 Done Sent Done false
36799 10118 Acked Read 7 Done Sent Done false
36800 10119 Acked Read 8 Done Sent Done false
36801 10120 Acked Read 10 Done Sent Done false
36802 10121 Acked Read 4 Done Sent Done false
36803 10122 Acked Read 4 Done Sent Done false
36804 10123 Acked Write 8 Done Sent Done false
36805 10124 Acked Read 6 Done Sent Done false
36806 10125 Acked Write 5 Done Sent Done false
36807 10126 Acked Write 5 Done Sent Done false
36808 10127 Acked Write 10 Done Sent Done false
36809 10128 Acked Read 4 Done Sent Done false
36810 10129 Acked Read 9 Done Sent Done false
36811 10130 Acked Read 3 Done Sent Done false
36812 10131 Acked Write 9 Done Sent Done false
36813 10132 Acked Write 2 Done Sent Done false
36814 10133 Acked Flush 0 Done Sent Done false
36815 10134 Acked Write 9 Done Sent Done false
36816 10135 Acked Flush 0 Done Sent Done false
36817 10136 Acked Write 8 Done Sent Done false
36818 10137 Acked Flush 0 Done Sent Done false
36819 10138 Acked Read 6 Done Sent Done false
36820 10139 Acked Write 8 Done Sent Done false
36821 10140 Acked Write 10 Done Sent Done false
36822 10141 Acked Flush 0 Done Sent Done false
36823 10142 Acked Read 4 Done Sent Done false
36824 10143 Acked Read 9 Done Sent Done false
36825 10144 Acked Read 2 Done Sent Done false
36826 10145 Acked Read 2 Done Sent Done false
36827 10146 Acked Read 7 Done Sent Done false
36828 10147 Acked Flush 0 Done Sent Done false
36829 10148 Acked Write 10 Done Sent Done false
36830 10149 Acked Read 7 Done Sent Done false
36831 10150 Acked Write 7 Done Sent Done false
36832 10151 Acked Write 2 Done Sent Done false
36833 10152 Acked Read 7 Done Sent Done false
36834 10153 Acked Flush 0 Done Sent Done false
36835 10154 Acked Write 3 Done Sent Done false
36836 10155 Acked Write 8 Done Sent Done false
36837 10156 Acked Flush 0 Done Sent Done false
36838 10157 Acked Flush 0 Done Sent Done false
36839 10158 Acked Read 5 Done Sent Done false
36840 10159 Acked Flush 0 Done Sent Done false
36841 10160 Acked Read 8 Done Sent Done false
36842 10161 Acked Read 7 Done Sent Done false
36843 10162 Acked Read 8 Done Sent Done false
36844 10163 Acked Read 3 Done Sent Done false
36845 10164 Acked Read 7 Done Sent Done false
36846 10165 Acked Write 10 Done Sent Done false
36847 10166 Acked Write 8 Done Sent Done false
36848 10167 Acked Read 2 Done Sent Done false
36849 10168 Acked Write 6 Done Sent Done false
36850 10169 Acked Write 10 Done Sent Done false
36851 10170 Acked Read 2 Done Sent Done false
36852 10171 Acked Read 1 Done Sent Done false
36853 10172 Acked Flush 0 Done Sent Done false
36854 10173 Acked Read 10 Done Sent Done false
36855 10174 Acked Write 10 Done Sent Done false
36856 10175 Acked Read 7 Done Sent Done false
36857 10176 Acked Write 3 Done Sent Done false
36858 10177 Acked Read 8 Done Sent Done false
36859 10178 Acked Flush 0 Done Sent Done false
36860 10179 Acked Write 9 Done Sent Done false
36861 10180 Acked Flush 0 Done Sent Done false
36862 10181 Acked Write 6 Done Sent Done false
36863 10182 Acked Write 5 Done Sent Done false
36864 10183 Acked Write 10 Done Sent Done false
36865 10184 Acked Read 3 Done Sent Done false
36866 10185 Acked Read 1 Done Sent Done false
36867 10186 Acked Read 9 Done Sent Done false
36868 10187 Acked Write 1 Done Sent Done false
36869 10188 Acked Write 10 Done Sent Done false
36870 10189 Acked Flush 0 Done Sent Done false
36871 10190 Acked Write 2 Done Sent Done false
36872 10191 Acked Read 6 Done Sent Done false
36873 10192 Acked Read 8 Done Sent Done false
36874 10193 Acked Read 3 Done Sent Done false
36875 10194 Acked Write 3 Done Sent Done false
36876 10195 Acked Read 8 Done Sent Done false
36877 10196 Acked Write 6 Done Sent Done false
36878 10197 Acked Read 4 Done Sent Done false
36879 10198 Acked Flush 0 Done Sent Done false
36880 10199 Acked Read 9 Done Sent Done false
36881 10200 Acked Read 5 Done Sent Done false
36882 10201 Acked Read 8 Done Sent Done false
36883 10202 Acked Flush 0 Done Sent Done false
36884 10203 Acked Read 8 Done Sent Done false
36885 10204 Acked Read 9 Done Sent Done false
36886 10205 Acked Read 10 Done Sent Done false
36887 10206 Acked Read 6 Done Sent Done false
36888 10207 Acked Read 10 Done Sent Done false
36889 10208 Acked Read 1 Done Sent Done false
36890 10209 Acked Read 3 Done Sent Done false
36891 10210 Acked Read 1 Done Sent Done false
36892 10211 Acked Read 6 Done Sent Done false
36893 10212 Acked Read 2 Done Sent Done false
36894 10213 Acked Write 2 Done Sent Done false
36895 10214 Acked Write 6 Done Sent Done false
36896 10215 Acked Read 6 Done Sent Done false
36897 10216 Acked Write 7 Done Sent Done false
36898 10217 Acked Flush 0 Done Sent Done false
36899 10218 Acked Read 9 Done Sent Done false
36900 10219 Acked Write 1 Done Sent Done false
36901 10220 Acked Read 8 Done Sent Done false
36902 10221 Acked Flush 0 Done Sent Done false
36903 10222 Acked Write 5 Done Sent Done false
36904 10223 Acked Read 2 Done Sent Done false
36905 10224 Acked Write 6 Done Sent Done false
36906 10225 Acked Write 9 Done Sent Done false
36907 10226 Acked Flush 0 Done Sent Done false
36908 10227 Acked Read 6 Done Sent Done false
36909 10228 Acked Read 4 Done Sent Done false
36910 10229 Acked Write 4 Done Sent Done false
36911 10230 Acked Write 3 Done Sent Done false
36912 10231 Acked Read 3 Done Sent Done false
36913 10232 Acked Read 6 Done Sent Done false
36914 10233 Acked Write 8 Done Sent Done false
36915 10234 Acked Flush 0 Done Sent Done false
36916 10235 Acked Write 9 Done Sent Done false
36917 10236 Acked Write 4 Done Sent Done false
36918 10237 Acked Read 8 Done Sent Done false
36919 10238 Acked Write 8 Done Sent Done false
36920 10239 Acked Write 2 Done Sent Done false
36921 10240 Acked Read 10 Done Sent Done false
36922 10241 Acked Write 1 Done Sent Done false
36923 10242 Acked Write 2 Done Sent Done false
36924 10243 Acked Read 4 Done Sent Done false
36925 10244 Acked Read 7 Done Sent Done false
36926 10245 Acked Read 3 Done Sent Done false
36927 10246 Acked Write 10 Done Sent Done false
36928 10247 Acked Flush 0 Done Sent Done false
36929 10248 Acked Read 10 Done Sent Done false
36930 10249 Acked Write 10 Done Sent Done false
36931 10250 Acked Write 1 Done Sent Done false
36932 10251 Acked Write 3 Done Sent Done false
36933 10252 Acked Write 2 Done Sent Done false
36934 10253 Acked Write 7 Done Sent Done false
36935 10254 Acked Read 6 Done Sent Done false
36936 10255 Acked Read 5 Done Sent Done false
36937 10256 Acked Write 2 Done Sent Done false
36938 10257 Acked Flush 0 Done Sent Done false
36939 10258 Acked Write 10 Done Sent Done false
36940 10259 Acked Write 7 Done Sent Done false
36941 10260 Acked Read 10 Done Sent Done false
36942 10261 Acked Read 2 Done Sent Done false
36943 10262 Acked Read 10 Done Sent Done false
36944 10263 Acked Write 8 Done Sent Done false
36945 10264 Acked Write 6 Done Sent Done false
36946 10265 Acked Read 10 Done Sent Done false
36947 10266 Acked Write 9 Done Sent Done false
36948 10267 Acked Read 1 Done Sent Done false
36949 10268 Acked Read 6 Done Sent Done false
36950 10269 Acked Read 2 Done Sent Done false
36951 10270 Acked Read 5 Done Sent Done false
36952 10271 Acked Read 8 Done Sent Done false
36953 10272 Acked Flush 0 Done Sent Done false
36954 10273 Acked Read 10 Done Sent Done false
36955 10274 Acked Read 1 Done Sent Done false
36956 10275 Acked Read 4 Done Sent Done false
36957 10276 Acked Read 2 Done Sent Done false
36958 10277 Acked Write 5 Done Sent Done false
36959 10278 Acked Read 5 Done Sent Done false
36960 10279 Acked Read 9 Done Sent Done false
36961 10280 Acked Write 8 Done Sent Done false
36962 10281 Acked Write 7 Done Sent Done false
36963 10282 Acked Write 2 Done Sent Done false
36964 10283 Acked Write 1 Done Sent Done false
36965 10284 Acked Write 3 Done Sent Done false
36966 10285 Acked Flush 0 Done Sent Done false
36967 10286 Acked Flush 0 Done Sent Done false
36968 10287 Acked Read 10 Done Sent Done false
36969 10288 Acked Read 4 Done Sent Done false
36970 10289 Acked Write 7 Done Sent Done false
36971 10290 Acked Read 5 Done Sent Done false
36972 10291 Acked Read 3 Done Sent Done false
36973 10292 Acked Read 10 Done Sent Done false
36974 10293 Acked Read 9 Done Sent Done false
36975 10294 Acked Write 6 Done Sent Done false
36976 10295 Acked Read 10 Done Sent Done false
36977 10296 Acked Read 9 Done Sent Done false
36978 10297 Acked Read 7 Done Sent Done false
36979 10298 Acked Read 5 Done Sent Done false
36980 10299 Acked Write 7 Done Sent Done false
36981 10300 Acked Read 2 Done Sent Done false
36982 10301 Acked Read 6 Done Sent Done false
36983 10302 Acked Write 7 Done Sent Done false
36984 10303 Acked Read 7 Done Sent Done false
36985 10304 Acked Read 10 Done Sent Done false
36986 10305 Acked Read 1 Done Sent Done false
36987 10306 Acked Flush 0 Done Sent Done false
36988 10307 Acked Write 10 Done Sent Done false
36989 10308 Acked Read 10 Done Sent Done false
36990 10309 Acked Write 5 Done Sent Done false
36991 10310 Acked Write 9 Done Sent Done false
36992 10311 Acked Flush 0 Done Sent Done false
36993 10312 Acked Read 6 Done Sent Done false
36994 10313 Acked Write 10 Done Sent Done false
36995 10314 Acked Write 9 Done Sent Done false
36996 10315 Acked Write 3 Done Sent Done false
36997 10316 Acked Read 1 Done Sent Done false
36998 10317 Acked Flush 0 Done Sent Done false
36999 10318 Acked Read 8 Done Sent Done false
37000 10319 Acked Write 7 Done Sent Done false
37001 10320 Acked Write 1 Done Sent Done false
37002 10321 Acked Read 8 Done Sent Done false
37003 10322 Acked Write 9 Done Sent Done false
37004 10323 Acked Write 2 Done Sent Done false
37005 10324 Acked Write 6 Done Sent Done false
37006 10325 Acked Read 1 Done Sent Done false
37007 10326 Acked Read 10 Done Sent Done false
37008 10327 Acked Read 2 Done Sent Done false
37009 10328 Acked Read 4 Done Sent Done false
37010 10329 Acked Write 9 Done Sent Done false
37011 10330 Acked Read 2 Done Sent Done false
37012 10331 Acked Write 9 Done Sent Done false
37013 10332 Acked Read 4 Done Sent Done false
37014 10333 Acked Write 7 Done Sent Done false
37015 10334 Acked Read 7 Done Sent Done false
37016 10335 Acked Write 5 Done Sent Done false
37017 10336 Acked Flush 0 Done Sent Done false
37018 10337 Acked Read 10 Done Sent Done false
37019 10338 Acked Write 4 Done Sent Done false
37020 10339 Acked Write 2 Done Sent Done false
37021 10340 Acked Read 9 Done Sent Done false
37022 10341 Acked Write 5 Done Sent Done false
37023 10342 Acked Write 10 Done Sent Done false
37024 10343 Acked Read 2 Done Sent Done false
37025 10344 Acked Write 3 Done Sent Done false
37026 10345 Acked Write 4 Done Sent Done false
37027 10346 Acked Read 8 Done Sent Done false
37028 10347 Acked Read 8 Done Sent Done false
37029 10348 Acked Read 2 Done Sent Done false
37030 10349 Acked Write 2 Done Sent Done false
37031 10350 Acked Write 9 Done Sent Done false
37032 10351 Acked Flush 0 Done Sent Done false
37033 10352 Acked Write 4 Done Sent Done false
37034 10353 Acked Read 9 Done Sent Done false
37035 10354 Acked Read 5 Done Sent Done false
37036 10355 Acked Read 1 Done Sent Done false
37037 10356 Acked Read 9 Done Sent Done false
37038 10357 Acked Write 1 Done Sent Done false
37039 STATES DS:0 DS:1 DS:2 TOTAL
37040 Sent 0 301 0 301
37041 Done 301 0 301 602
37042 Skipped 0 0 0 0
37043 Error 0 0 0 0
37044 Last Flush: Some(JobId(10351)) Some(JobId(10056)) Some(JobId(10351))
37045 Downstairs last five completed: 10056 10055 10054 10053 10052
37046 Upstairs last five completed: 10357 10356 10355 10354 10353
37047 CLIENT: Up:0 ds:301 act:2
370482025-03-17T22:42:45.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370492025-03-17T22:42:46.074ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370502025-03-17T22:42:46.494ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370512025-03-17T22:42:46.494ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33861) = io task client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370522025-03-17T22:42:46.495ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370532025-03-17T22:42:46.495ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370542025-03-17T22:42:46.495ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370552025-03-17T22:42:46.495ZINFOcrucible: send last flush ID to this DS: Some(JobId(10056)) = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370562025-03-17T22:42:46.495ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370572025-03-17T22:42:46.495ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10056)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
370582025-03-17T22:42:46.496ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10056)) = downstairs session_id = 34ae9076-d4ea-416d-9168-6f1a5fc19802
37059 ----------------------------------------------------------------
37060 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37061 Downstairs last five completed: 10358 10357 10356 10355 10354
37062 Upstairs last five completed: 10358 10357 10356 10355 10354
37063 CLIENT: Up:0 ds:0 act:3
37064 Replay: All jobs finished
37065 Test replay has completed
37066 Wrote out file "/tmp/test_replay_verify.log"
37067 CLIENT: Tests done. All submitted work has been ACK'd
37068 ----------------------------------------------------------------
37069 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37070 Downstairs last five completed: 10358 10357 10356 10355 10354
37071 Upstairs last five completed: 10358 10357 10356 10355 10354
37072 CLIENT: Up:0 ds:0 act:3
37073 CLIENT: All jobs finished, all DS active.
37074 Replays:30 time: 8:16
37075 Do final verify
37076 Mar 17 22:42:48.832 INFO accepted connection, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
37077 Mar 17 22:42:48.833 INFO request completed, latency_us: 213, response_code: 200, uri: /regioninfo, method: GET, req_id: ee57dc0f-872f-462c-8972-481f61272166, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
370782025-03-17T22:42:48.833ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
37079 Mar 17 22:42:48.834 INFO request completed, latency_us: 88, response_code: 200, uri: /regioncount, method: GET, req_id: e5fb4655-1a3e-4197-9e4b-81fe1be58da3, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
370802025-03-17T22:42:48.834ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
37081 Mar 17 22:42:48.834 INFO request completed, latency_us: 111, response_code: 200, uri: /port/cid/0, method: GET, req_id: 31043112-7b73-47f4-b5d6-8704286e9db4, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
37082 Mar 17 22:42:48.835 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/1, method: GET, req_id: 46149fa0-6809-44a8-af89-811899b5fee2, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
37083 Mar 17 22:42:48.835 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/2, method: GET, req_id: 8b84f2c9-0227-4a1d-a443-63e175dcf670, remote_addr: 127.0.0.1:54316, local_addr: 127.0.0.1:9998
370842025-03-17T22:42:48.835ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
370852025-03-17T22:42:48.836ZINFOcrucible: Upstairs starts
370862025-03-17T22:42:48.836ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "45feeae3dd1bdcd88dce11a80acb5415f57b38bc", git_commit_timestamp: "2025-03-17T22:11:58.000000000Z", git_branch: "renovate/opentelemetry-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
370872025-03-17T22:42:48.836ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
370882025-03-17T22:42:48.836ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
370892025-03-17T22:42:48.836ZINFOcrucible: Crucible 016423b1-3806-4096-afd7-769cf59959ec has session id: 0fd81ac4-235e-4561-aed2-74d56cd61da8 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370902025-03-17T22:42:48.836ZINFOcrucible: Upstairs opts: Upstairs UUID: 016423b1-3806-4096-afd7-769cf59959ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370912025-03-17T22:42:48.836ZINFOcrucible: Crucible stats registered with UUID: 016423b1-3806-4096-afd7-769cf59959ec session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370922025-03-17T22:42:48.836ZWARNcrucible: could not find Downstairs address for Nexus session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370932025-03-17T22:42:48.837ZINFOcrucible: Downstairs in volume = 3 task = crutest
370942025-03-17T22:42:48.837ZINFOcrucible: The guest has requested activation with gen:3
370952025-03-17T22:42:48.837ZINFOcrucible: 016423b1-3806-4096-afd7-769cf59959ec active request set session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370962025-03-17T22:42:48.837ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370972025-03-17T22:42:48.837ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370982025-03-17T22:42:48.837ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
370992025-03-17T22:42:48.837ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371002025-03-17T22:42:48.837ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371012025-03-17T22:42:48.838ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371022025-03-17T22:42:48.838ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371032025-03-17T22:42:48.838ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371042025-03-17T22:42:48.838ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371052025-03-17T22:42:48.838ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46425) = io task client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371062025-03-17T22:42:48.838ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40619) = io task client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371072025-03-17T22:42:48.838ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42863) = io task client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371082025-03-17T22:42:48.838ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371092025-03-17T22:42:48.838ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371102025-03-17T22:42:48.838ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371112025-03-17T22:42:48.839ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371122025-03-17T22:42:48.839ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371132025-03-17T22:42:48.839ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371142025-03-17T22:42:48.840ZINFOcrucible: Waiting for more clients to be ready session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371152025-03-17T22:42:48.840ZINFOcrucible: Waiting for more clients to be ready session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371162025-03-17T22:42:48.840ZINFOcrucible: [0]R flush_numbers[0..12]: [1001, 990, 999, 1000, 1000, 998, 998, 989, 999, 985, 992, 996] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371172025-03-17T22:42:48.840ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371182025-03-17T22:42:48.840ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371192025-03-17T22:42:48.840ZINFOcrucible: [1]R flush_numbers[0..12]: [1001, 990, 999, 1000, 1000, 998, 998, 989, 999, 985, 992, 996] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371202025-03-17T22:42:48.840ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371212025-03-17T22:42:48.840ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371222025-03-17T22:42:48.840ZINFOcrucible: [2]R flush_numbers[0..12]: [1001, 990, 999, 1000, 1000, 998, 998, 989, 999, 985, 992, 996] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371232025-03-17T22:42:48.840ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371242025-03-17T22:42:48.840ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371252025-03-17T22:42:48.840ZINFOcrucible: Max found gen is 3 = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371262025-03-17T22:42:48.840ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371272025-03-17T22:42:48.840ZINFOcrucible: Next flush: 1002 = downstairs session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
37128 {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
37129 ,"time":"2025-03-17T22:42:48.840678489Z","hostname":"w-01JPK3TRDDGVTD8M7PA5T22JF0","pid":1244,"":"downstairs","":"downstairs","session_id":"0fd81ac4-235e-4561-aed2-74d56cd61da8"}
371302025-03-17T22:42:48.840ZINFOcrucible: No downstairs reconciliation required session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371312025-03-17T22:42:48.840ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371322025-03-17T22:42:48.840ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371332025-03-17T22:42:48.840ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371342025-03-17T22:42:48.840ZINFOcrucible: All required reconciliation work is completed session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371352025-03-17T22:42:48.840ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371362025-03-17T22:42:48.840ZINFOcrucible: 016423b1-3806-4096-afd7-769cf59959ec is now active with session: 0fd81ac4-235e-4561-aed2-74d56cd61da8 session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371372025-03-17T22:42:48.840ZINFOcrucible: Set Active after no reconciliation session_id = 0fd81ac4-235e-4561-aed2-74d56cd61da8
371382025-03-17T22:42:48.840ZINFOcrucible: The guest has finished waiting for activation with:3
371392025-03-17T22:42:48.840ZINFOcrucible: Activated sub_volume 016423b1-3806-4096-afd7-769cf59959ec
37140 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
37141 Loading write count information from file "/tmp/test_replay_verify.log"
37142 Read and Verify all blocks (0..5000 range:false)
37143 Wrote out file "/tmp/test_replay_verify.log"
37144 Verify test completed
37145 Wrote out file "/tmp/test_replay_verify.log"
37146 CLIENT: Tests done. All submitted work has been ACK'd
37147 ----------------------------------------------------------------
37148 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
37149 States: Active Active Active
37150 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37151 1000 Acked Read 256 Done Done Done false
37152 1001 Acked Read 256 Done Done Done false
37153 1002 Acked Read 256 Done Done Done false
37154 1003 Acked Read 256 Done Done Done false
37155 1004 Acked Read 256 Done Done Done false
37156 1005 Acked Read 256 Done Done Done false
37157 1006 Acked Read 256 Done Done Done false
37158 1007 Acked Read 256 Done Done Done false
37159 1008 Acked Read 256 Done Done Done false
37160 1009 Acked Read 256 Done Done Done false
37161 1010 Acked Read 256 Done Done Done false
37162 1011 Acked Read 256 Done Done Done false
37163 1012 Acked Read 256 Done Done Done false
37164 1013 Acked Read 256 Done Done Done false
37165 1014 Acked Read 256 Done Done Done false
37166 1015 Acked Read 256 Done Done Done false
37167 1016 Acked Read 256 Done Done Done false
37168 1017 Acked Read 256 Done Done Done false
37169 1018 Acked Read 256 Done Done Done false
37170 1019 Acked Read 136 Done Done Done false
37171 STATES DS:0 DS:1 DS:2 TOTAL
37172 Sent 0 0 0 0
37173 Done 20 20 20 60
37174 Skipped 0 0 0 0
37175 Error 0 0 0 0
37176 Last Flush: None None None
37177 Downstairs last five completed:
37178 Upstairs last five completed: 1019 1018 1017 1016 1015
37179 CLIENT: Up:0 ds:20 act:3
37180 ----------------------------------------------------------------
37181 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
37182 Downstairs last five completed: 1020 1019 1018 1017 1016
37183 Upstairs last five completed: 1020 1019 1018 1017 1016
37184 CLIENT: Up:0 ds:0 act:3
37185 CLIENT: All crucible jobs finished, exiting program
371862025-03-17T22:42:53.226ZWARNcrucible: Guest handle has been dropped
37187 Mar 17 22:42:53.301 INFO accepted connection, remote_addr: 127.0.0.1:64942, local_addr: 127.0.0.1:9998
37188 Main task has work to do, go find it
37189 got dsc Shutdown
37190 Shutdown
37191 Shut it down
37192 Mar 17 22:42:53.301 INFO request completed, latency_us: 174, response_code: 204[8810] Disable keep_running
37193 , uri:[8810] Got stop action so:false kr:false
37194 /shutdown, method: POST, req_id: f6ce4d61-6a9e-41e9-ac26-850536144b8d, remote_addr: 127.0.0.1:64942, local_addr: 127.0[8820] Disable keep_running
37195 .[8820] Got stop action so:false kr:false
37196 0.[8830] Disable keep_running
37197 1[8830] Got stop action so:false kr:false
37198 :9998
37199 [8830] Exited with: ExitStatus(unix_wait_status(9))
37200 [8830][2] reports Exit
37201 [8820] Exited with: ExitStatus(unix_wait_status(9))
37202 [8810] Exited with: ExitStatus(unix_wait_status(9))
37203 [8820][1] reports Exit
37204 [8810][0] reports Exit
37205 March 17, 2025 at 10:42:58 PM UTC Test ends with 0