Max level shown:
1 starting February 25, 2025 at 08:46:34 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.25910717
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.022918047
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.021579752
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Starting downstairs at port 8830
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 Make output file at "/tmp/dsc/downstairs-8810.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1196)
26 [8820][1] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1197)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1198)
30 [8810][0] initial start wait reports Running
31 All downstairs are running
32 Feb 25 20:46:34.790 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Feb 25 20:46:39.888 INFO accepted connection, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
36 Feb 25 20:46:39.891 INFO request completed, latency_us: 672, response_code: 200, uri: /regioninfo, method: GET, req_id: 48616750-be4a-4ac4-91c6-092753c911de, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
37 Feb 25 20:46:39.894 INFO request completed, latency_us: 115, response_code: 200, uri: /regioncount, method: GET, req_id: 26438fc5-3bb7-432d-a775-061d2f165c6c, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
382025-02-25T20:46:39.894ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-02-25T20:46:39.894ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Feb 25 20:46:39.895 INFO request completed, latency_us: 174, response_code: 200, uri: /port/cid/0, method: GET, req_id: 63dcb280-665e-49d6-924d-b889de76fedc, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
41 Feb 25 20:46:39.895 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/1, method: GET, req_id: 325e9dd5-09d3-44cf-ac93-1feaaf3d9437, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
42 Feb 25 20:46:39.896 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/2, method: GET, req_id: e323f804-fceb-4271-ae28-53aa99b721ce, remote_addr: 127.0.0.1:36991, local_addr: 127.0.0.1:9998
432025-02-25T20:46:39.896ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-02-25T20:46:39.897ZINFOcrucible: Upstairs starts
452025-02-25T20:46:39.898ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "cdb6c7ae221cade941351dba5e763db1f87ef907", git_commit_timestamp: "2025-02-25T18:15:07.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-02-25T20:46:39.898ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-02-25T20:46:39.898ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-02-25T20:46:39.899ZINFOcrucible: Crucible c2ce92ac-9153-41d3-ae37-84cd23f492dd has session id: 8c92a376-4a4a-47a1-a7e0-6fae41598c94 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
492025-02-25T20:46:39.899ZINFOcrucible: Upstairs opts: Upstairs UUID: c2ce92ac-9153-41d3-ae37-84cd23f492dd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
502025-02-25T20:46:39.899ZINFOcrucible: Crucible stats registered with UUID: c2ce92ac-9153-41d3-ae37-84cd23f492dd session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
512025-02-25T20:46:39.899ZWARNcrucible: could not find Downstairs address for Nexus session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
522025-02-25T20:46:39.901ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-02-25T20:46:39.901ZINFOcrucible: The guest has requested activation with gen:1
542025-02-25T20:46:39.901ZINFOcrucible: c2ce92ac-9153-41d3-ae37-84cd23f492dd active request set session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
552025-02-25T20:46:39.901ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
562025-02-25T20:46:39.901ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
572025-02-25T20:46:39.901ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
582025-02-25T20:46:39.901ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
592025-02-25T20:46:39.902ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
602025-02-25T20:46:39.902ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
612025-02-25T20:46:39.902ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
622025-02-25T20:46:39.902ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
632025-02-25T20:46:39.902ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
642025-02-25T20:46:39.902ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46928) = io task client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
652025-02-25T20:46:39.902ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35335) = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
662025-02-25T20:46:39.902ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60986) = io task client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
672025-02-25T20:46:39.904ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
682025-02-25T20:46:39.904ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
692025-02-25T20:46:39.904ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
702025-02-25T20:46:39.905ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
712025-02-25T20:46:39.905ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
722025-02-25T20:46:39.905ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
732025-02-25T20:46:39.906ZINFOcrucible: Waiting for more clients to be ready session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
742025-02-25T20:46:39.906ZINFOcrucible: Waiting for more clients to be ready session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
752025-02-25T20:46:39.906ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
762025-02-25T20:46:39.906ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
772025-02-25T20:46:39.906ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
782025-02-25T20:46:39.906ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
792025-02-25T20:46:39.906ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
802025-02-25T20:46:39.906ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
812025-02-25T20:46:39.906ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
822025-02-25T20:46:39.906ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
832025-02-25T20:46:39.906ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
842025-02-25T20:46:39.906ZINFOcrucible: Max found gen is 1 = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
852025-02-25T20:46:39.906ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
862025-02-25T20:46:39.906ZINFOcrucible: Next flush: 1 = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
872025-02-25T20:46:39.906ZINFOcrucible: All extents match = downstairs session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
882025-02-25T20:46:39.906ZINFOcrucible: No downstairs reconciliation required session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
892025-02-25T20:46:39.906ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
90 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
91 ,"Wait for a query_work_queue command to finish before sending IO
92 time":"2025-02-25T20:46:39.906950918Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1202,"client":"1","":"downstairs","session_id":"8c92a376-4a4a-47a1-a7e0-6fae41598c94"}
932025-02-25T20:46:39.906ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
942025-02-25T20:46:39.906ZINFOcrucible: All required reconciliation work is completed session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
952025-02-25T20:46:39.906ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
962025-02-25T20:46:39.907ZINFOcrucible: c2ce92ac-9153-41d3-ae37-84cd23f492dd is now active with session: 8c92a376-4a4a-47a1-a7e0-6fae41598c94 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
972025-02-25T20:46:39.907ZINFOcrucible: Set Active after no reconciliation session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
982025-02-25T20:46:39.907ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 256 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 136 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-02-25T20:46:44.490ZWARNcrucible: Guest handle has been dropped
1442025-02-25T20:46:44.490ZWARNcrucible: client request queue closed unexpectedly; is the program exiting? = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
1452025-02-25T20:46:44.490ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
1462025-02-25T20:46:44.490ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
1472025-02-25T20:46:44.490ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8c92a376-4a4a-47a1-a7e0-6fae41598c94
148 Replay loop starts now February 25, 2025 at 08:46:44 PM UTC
149 Feb 25 20:46:44.561 INFO accepted connection, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
150 Feb 25 20:46:44.561 INFO request completed, latency_us: 159, response_code: 200, uri: /regioninfo, method: GET, req_id: 04ea08f0-2de0-4d33-950f-1ca96d3cb522, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
151 Feb 25 20:46:44.562 INFO request completed, latency_us: 103, response_code: {"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:46:44.562557929Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"task":"crutest"}
152 200, uri: /regioncount, method: GET, req_id: 37f3a74e-369b-413a-98c2-cdf0bc522f74, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
1532025-02-25T20:46:44.563ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
154 Feb 25 20:46:44.563 INFO request completed, latency_us: 112, response_code: 200, uri: /port/cid/0, method: GET, req_id: 456b60bc-38a2-4b55-a11f-2e88c022655d, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
155 Feb 25 20:46:44.564 INFO request completed, latency_us: 108, response_code: 200, uri: /port/cid/1, method: GET, req_id: ddfef100-9561-4bff-990a-111520440dcd, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
156 Feb 25 20:46:44.564 INFO request completed, latency_us: 106, response_code: 200, uri: /port/cid/2, method: GET, req_id: fdd68a3d-da62-4f64-8bda-c661e62e5670, remote_addr: 127.0.0.1:45061, local_addr: 127.0.0.1:9998
1572025-02-25T20:46:44.564ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1582025-02-25T20:46:44.565ZINFOcrucible: Upstairs starts
1592025-02-25T20:46:44.565ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "cdb6c7ae221cade941351dba5e763db1f87ef907", git_commit_timestamp: "2025-02-25T18:15:07.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1602025-02-25T20:46:44.565ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1612025-02-25T20:46:44.565ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1622025-02-25T20:46:44.565ZINFOcrucible: Crucible da1267cd-6ccd-4d91-9563-00eaae1e975f has session id: 0093020f-8db1-4b17-8f9d-499d3d7b91d5 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1632025-02-25T20:46:44.565ZINFOcrucible: Upstairs opts: Upstairs UUID: da1267cd-6ccd-4d91-9563-00eaae1e975f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1642025-02-25T20:46:44.565ZINFOcrucible: Crucible stats registered with UUID: da1267cd-6ccd-4d91-9563-00eaae1e975f session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1652025-02-25T20:46:44.565ZWARNcrucible: could not find Downstairs address for Nexus session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1662025-02-25T20:46:44.566ZINFOcrucible: Downstairs in volume = 3 task = crutest
1672025-02-25T20:46:44.566ZINFOcrucible: The guest has requested activation with gen:2
1682025-02-25T20:46:44.566ZINFOcrucible: da1267cd-6ccd-4d91-9563-00eaae1e975f active request set session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1692025-02-25T20:46:44.566ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1702025-02-25T20:46:44.566ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1712025-02-25T20:46:44.566ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1722025-02-25T20:46:44.566ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1732025-02-25T20:46:44.566ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1742025-02-25T20:46:44.566ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1752025-02-25T20:46:44.566ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1762025-02-25T20:46:44.566ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1772025-02-25T20:46:44.566ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1782025-02-25T20:46:44.566ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64977) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1792025-02-25T20:46:44.566ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41336) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1802025-02-25T20:46:44.567ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49252) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1812025-02-25T20:46:44.567ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1822025-02-25T20:46:44.567ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1832025-02-25T20:46:44.567ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1842025-02-25T20:46:44.568ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1852025-02-25T20:46:44.568ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1862025-02-25T20:46:44.568ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1872025-02-25T20:46:44.569ZINFOcrucible: Waiting for more clients to be ready session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1882025-02-25T20:46:44.569ZINFOcrucible: Waiting for more clients to be ready session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1892025-02-25T20:46:44.569ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1902025-02-25T20:46:44.569ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1912025-02-25T20:46:44.569ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1922025-02-25T20:46:44.569ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1932025-02-25T20:46:44.569ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1942025-02-25T20:46:44.569ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1952025-02-25T20:46:44.569ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1962025-02-25T20:46:44.569ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1972025-02-25T20:46:44.569ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1982025-02-25T20:46:44.569ZINFOcrucible: Max found gen is 2 = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
199 {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:46:44.569365589Z"Activate successful
200 ,"Wait for a query_work_queue command to finish before sending IO
201 hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"downstairs","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
2022025-02-25T20:46:44.569ZINFOcrucible: Next flush: 2 = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2032025-02-25T20:46:44.569ZINFOcrucible: All extents match = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2042025-02-25T20:46:44.569ZINFOcrucible: No downstairs reconciliation required session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2052025-02-25T20:46:44.569ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2062025-02-25T20:46:44.569ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2072025-02-25T20:46:44.569ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2082025-02-25T20:46:44.569ZINFOcrucible: All required reconciliation work is completed session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2092025-02-25T20:46:44.569ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2102025-02-25T20:46:44.569ZINFOcrucible: da1267cd-6ccd-4d91-9563-00eaae1e975f is now active with session: 0093020f-8db1-4b17-8f9d-499d3d7b91d5 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2112025-02-25T20:46:44.569ZINFOcrucible: Set Active after no reconciliation session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2122025-02-25T20:46:44.569ZINFOcrucible: The guest has finished waiting for activation with:2
213 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
214 Loading write count information from file "/tmp/test_replay_verify.log"
215 Feb 25 20:46:44.601 INFO accepted connection, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
216 Main task has work to do, go find it
217 got dsc Stop(2)
218 stop 2
219 Feb 25 20:46:44.602 INFO request completed, latency_us: 184, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 310b2722-51be-4667-8856-11e2aecbe7d1, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
220 [8830] Got stop action so:false kr:false
2212025-02-25T20:46:44.603ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2222025-02-25T20:46:44.603ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2232025-02-25T20:46:44.603ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
224 Feb 25 20:46:44.603 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:46:44.603297947Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
225 GET, req_id: 0e2505ee-3d08-4f1b-b4ab-43dcf8685f6d, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
226 [8830] Exited with: ExitStatus(unix_wait_status(9))
227 [8830][2] reports Exit
228 Feb 25 20:46:48.606 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9be5bb97-db95-4493-b754-29a6745b5361, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
229 001/300 Write block 3834 len 8192 data: 2 2
230 002/300 Flush
231 003/300 Write block 2688 len 12288 data: 2 2 2
232 004/300 Write block 704 len 12288 data: 2 2 2
233 005/300 Read block 3498 len 8192
234 006/300 Write block 1783 len 20480 data: 2 2 2 2 2
235 007/300 Write block 1874 len 28672 data: 2 2 2 2 2 2 2
236 008/300 Write block 3928 len 8192 data: 2 2
237 009/300 Write block 2469 len 12288 data: 2 2 2
238 010/300 Write block 1431 len 32768 data: 2 2 2 2 2 2 2 2
239 011/300 Read block 3677 len 16384
240 012/300 Flush
241 013/300 Write block 3022 len 36864 data: 2 2 2 2 2 2 2 2 2
242 014/300 Read block 1971 len 24576
243 015/300 Write block 3901 len 32768 data: 2 2 2 2 2 2 2 2
244 016/300 Write block 4777 len 4096 data: 2
245 017/300 Write block 2688 len 4096 data: 3
246 018/300 Read block 1772 len 16384
247 019/300 Read block 1940 len 36864
248 020/300 Read block 876 len 40960
249 021/300 Flush
250 022/300 Read block 2187 len 20480
251 023/300 Read block 4912 len 16384
252 024/300 Write block 2507 len 8192 data: 2 2
253 025/300 Write block 1560 len 4096 data: 2
254 026/300 Write block 674 len 24576 data: 2 2 2 2 2 2
255 027/300 Write block 2322 len 4096 data: 2
256 028/300 Read block 4321 len 24576
257 029/300 Read block 841 len 12288
258 030/300 Read block 1996 len 4096
259 031/300 Read block 4853 len 4096
260 032/300 Read block 1265 len 28672
261 033/300 Write block 1930 len 16384 data: 2 2 2 2
262 034/300 Write block 3009 len 4096 data: 2
263 035/300 Write block 3323 len 16384 data: 2 2 2 2
264 036/300 Flush
265 037/300 Write block 4163 len 24576 data: 2 2 2 2 2 2
266 038/300 Write block 3836 len 8192 data: 2 2
267 039/300 Write block 3371 len 12288 data: 2 2 2
268 040/300 Read block 3211 len 12288
269 041/300 Read block 926 len 24576
270 042/300 Write block 3567 len 32768 data: 2 2 2 2 2 2 2 2
271 043/300 Write block 4600 len 28672 data: 2 2 2 2 2 2 2
272 044/300 Write block 3198 len 20480 data: 2 2 2 2 2
273 045/300 Read block 3920 len 40960
274 046/300 Read block 29 len 36864
275 047/300 Write block 214 len 36864 data: 2 2 2 2 2 2 2 2 2
276 048/300 Read block 1660 len 24576
277 049/300 Read block 4942 len 32768
278 050/300 Write block 3962 len 8192 data: 2 2
279 051/300 Write block 1235 len 16384 data: 2 2 2 2
280 052/300 Read block 535 len 4096
281 053/300 Read block 1227 len 12288
282 054/300 Read block 2287 len 28672
283 055/300 Flush
284 056/300 Flush
285 057/300 Write block 4909 len 28672 data: 2 2 2 2 2 2 2
286 058/300 Read block 4056 len 20480
287 059/300 Read block 2631 len 20480
288 060/300 Write block 2412 len 16384 data: 2 2 2 2
289 061/300 Read block 3736 len 8192
290 062/300 Read block 3164 len 36864
291 063/300 Read block 23 len 36864
292 064/300 Read block 1374 len 28672
293 065/300 Read block 1748 len 8192
294 066/300 Read block 351 len 24576
295 067/300 Read block 789 len 40960
296 068/300 Write block 507 len 4096 data: 2
297 069/300 Write block 951 len 16384 data: 2 2 2 2
298 070/300 Read block 229 len 4096
299 071/300 Write block 3005 len 8192 data: 2 2
300 072/300 Write block 742 len 4096 data: 2
301 073/300 Write block 2623 len 24576 data: 2 2 2 2 2 2
302 074/300 Write block 1970 len 28672 data: 2 2 2 2 2 2 2
303 075/300 Write block 1528 len 36864 data: 2 2 2 2 2 2 2 2 2
304 076/300 Read block 4586 len 8192
305 077/300 Read block 4817 len 12288
306 078/300 Write block 3370 len 4096 data: 2
307 079/300 Write block 209 len 32768 data: 2 2 2 2 2 3 3 3
308 080/300 Read block 3260 len 20480
309 081/300 Read block 2859 len 36864
310 082/300 Flush
311 083/300 Read block 1740 len 28672
312 084/300 Read block 269 len 32768
313 085/300 Read block 3154 len 16384
314 086/300 Read block 169 len 32768
315 087/300 Read block 1993 len 40960
316 088/300 Write block 2389 len 4096 data: 2
317 089/300 Write block 1848 len 16384 data: 2 2 2 2
318 090/300 Write block 3105 len 4096 data: 2
319 091/300 Write block 4690 len 36864 data: 2 2 2 2 2 2 2 2 2
320 092/300 Write block 4037 len 20480 data: 2 2 2 2 2
321 093/300 Read block 2769 len 36864
322 094/300 Write block 4099 len 24576 data: 2 2 2 2 2 2
323 095/300 Read block 2031 len 4096
324 096/300 Read block 191 len 28672
325 097/300 Read block 65 len 28672
326 098/300 Write block 1734 len 8192 data: 2 2
327 099/300 Write block 2401 len 8192 data: 2 2
328 100/300 Write block 3707 len 40960 data: 2 2 2 2 2 2 2 2 2 2
329 101/300 Read block 1537 len 36864
330 102/300 Write block 3946 len 4096 data: 2
331 103/300 Read block 4844 len 28672
332 104/300 Read block 3509 len 32768
333 105/300 Read block 2713 len 20480
334 106/300 Read block 2453 len 28672
335 107/300 Read block 4465 len 16384
336 108/300 Write block 1393 len 12288 data: 2 2 2
337 109/300 Read block 4345 len 24576
338 110/300 Read block 583 len 20480
339 111/300 Read block 1881 len 12288
340 112/300 Read block 442 len 4096
341 113/300 Read block 862 len 36864
342 114/300 Read block 2620 len 4096
343 115/300 Read block 3152 len 8192
344 116/300 Write block 3434 len 12288 data: 2 2 2
345 117/300 Flush
346 118/300 Write block 3640 len 28672 data: 2 2 2 2 2 2 2
347 119/300 Write block 124 len 8192 data: 2 2
348 120/300 Write block 3272 len 28672 data: 2 2 2 2 2 2 2
349 121/300 Read block 4023 len 16384
350 122/300 Write block 3233 len 40960 data: 2 2 2 2 2 2 2 2 2 2
351 123/300 Write block 2487 len 36864 data: 2 2 2 2 2 2 2 2 2
352 124/300 Write block 1042 len 36864 data: 2 2 2 2 2 2 2 2 2
353 125/300 Write block 3561 len 32768 data: 2 2 2 2 2 2 3 3
354 126/300 Read block 2967 len 28672
355 127/300 Flush
356 128/300 Read block 1187 len 24576
357 129/300 Write block 3619 len 24576 data: 2 2 2 2 2 2
358 130/300 Read block 4659 len 4096
359 131/300 Read block 1372 len 20480
360 132/300 Read block 2045 len 40960
361 133/300 Read block 3168 len 24576
362 134/300 Read block 4578 len 32768
363 135/300 Read block 3101 len 40960
364 136/300 Read block 313 len 8192
365 137/300 Read block 3718 len 12288
366 138/300 Read block 3348 len 40960
367 139/300 Write block 711 len 24576 data: 2 2 2 2 2 2
368 140/300 Write block 1190 len 20480 data: 2 2 2 2 2
369 141/300 Flush
370 142/300 Write block 2625 len 24576 data: 3 3 3 3 2 2
371 143/300 Write block 2450 len 24576 data: 2 2 2 2 2 2
372 144/300 Read block 3515 len 16384
373 145/300 Flush
374 146/300 Read block 1419 len 28672
375 147/300 Write block 2200 len 20480 data: 2 2 2 2 2
376 148/300 Read block 683 len 32768
377 149/300 Write block 2243 len 16384 data: 2 2 2 2
378 150/300 Flush
379 151/300 Write block 205 len 8192 data: 2 2
380 152/300 Write block 268 len 28672 data: 2 2 2 2 2 2 2
381 153/300 Write block 4374 len 16384 data: 2 2 2 2
382 154/300 Write block 942 len 4096 data: 2
383 155/300 Write block 942 len 24576 data: 3 2 2 2 2 2
384 156/300 Read block 1265 len 12288
385 157/300 Write block 3637 len 40960 data: 2 2 2 3 3 3 3 3 3 3
386 158/300 Flush
387 159/300 Write block 3484 len 40960 data: 2 2 2 2 2 2 2 2 2 2
388 160/300 Flush
389 161/300 Flush
390 162/300 Read block 2107 len 36864
391 163/300 Flush
392 164/300 Read block 4635 len 32768
393 165/300 Read block 2238 len 4096
394 166/300 Read block 772 len 4096
395 167/300 Read block 2734 len 24576
396 168/300 Write block 4421 len 8192 data: 2 2
397 169/300 Read block 4899 len 8192
398 170/300 Read block 1361 len 16384
399 171/300 Read block 3677 len 12288
400 172/300 Flush
401 173/300 Write block 4022 len 32768 data: 2 2 2 2 2 2 2 2
402 174/300 Read block 1031 len 16384
403 175/300 Write block 3630 len 40960 data: 2 2 2 2 2 2 2 3 3 3
404 176/300 Write block 4602 len 20480 data: 3 3 3 3 3
405 177/300 Read block 4869 len 20480
406 178/300 Flush
407 179/300 Flush
408 180/300 Read block 771 len 4096
409 181/300 Write block 3099 len 36864 data: 2 2 2 2 2 2 3 2 2
410 182/300 Read block 3068 len 8192
411 183/300 Read block 4400 len 8192
412 184/300 Read block 4222 len 28672
413 185/300 Flush
414 186/300 Write block 2882 len 8192 data: 2 2
415 187/300 Read block 3792 len 40960
416 188/300 Write block 2341 len 12288 data: 2 2 2
417 189/300 Read block 4949 len 8192
418 190/300 Write block 306 len 40960 data: 2 2 2 2 2 2 2 2 2 2
419 191/300 Flush
420 192/300 Write block 284 len 4096 data: 2
421 193/300 Read block 3647 len 8192
422 194/300 Write block 1361 len 20480 data: 2 2 2 2 2
423 195/300 Read block 4048 len 36864
424 196/300 Write block 1676 len 8192 data: 2 2
425 197/300 Write block 4924 len 12288 data: 2 2 2
426 198/300 Write block 2680 len 28672 data: 2 2 2 2 2 2 2
427 199/300 Read block 1753 len 20480
428 200/300 Write block 2244 len 36864 data: 3 3 3 2 2 2 2 2 2
429 201/300 Write block 4056 len 36864 data: 2 2 2 2 2 2 2 2 2
430 202/300 Read block 2613 len 40960
431 203/300 Read block 3137 len 32768
432 204/300 Read block 3841 len 8192
433 205/300 Write block 3072 len 16384 data: 2 2 2 2
434 206/300 Read block 1505 len 8192
435 207/300 Read block 2259 len 8192
436 208/300 Read block 1831 len 36864
437 209/300 Read block 1614 len 20480
438 210/300 Read block 104 len 28672
439 211/300 Read block 68 len 32768
440 212/300 Read block 3247 len 4096
441 213/300 Write block 3825 len 28672 data: 2 2 2 2 2 2 2
442 214/300 Flush
443 215/300 Read block 4574 len 36864
444 216/300 Write block 4781 len 40960 data: 2 2 2 2 2 2 2 2 2 2
445 217/300 Write block 2767 len 40960 data: 2 2 2 2 2 2 2 2 2 2
446 218/300 Read block 678 len 28672
447 219/300 Write block 835 len 24576 data: 2 2 2 2 2 2
448 220/300 Write block 4891 len 32768 data: 2 2 2 2 2 2 2 2
449 221/300 Write block 2111 len 8192 data: 2 2
450 222/300 Read block 1264 len 4096
451 223/300 Write block 3152 len 36864 data: 2 2 2 2 2 2 2 2 2
452 224/300 Read block 1561 len 12288
453 225/300 Write block 2035 len 12288 data: 2 2 2
454 226/300 Write block 1030 len 40960 data: 2 2 2 2 2 2 2 2 2 2
455 227/300 Write block 863 len 20480 data: 2 2 2 2 2
456 228/300 Read block 78 len 32768
457 229/300 Flush
458 230/300 Write block 4418 len 40960 data: 2 2 2 3 3 2 2 2 2 2
459 231/300 Flush
460 232/300 Read block 1327 len 24576
461 233/300 Read block 651 len 24576
462 234/300 Write block 2348 len 40960 data: 2 2 2 2 2 2 2 2 2 2
463 235/300 Read block 3836 len 16384
464 236/300 Flush
465 237/300 Write block 2453 len 12288 data: 3 3 3
466 238/300 Write block 393 len 20480 data: 2 2 2 2 2
467 239/300 Write block 4099 len 32768 data: 3 3 3 3 3 3 2 2
468 240/300 Flush
469 241/300 Write block 2208 len 24576 data: 2 2 2 2 2 2
470 242/300 Write block 975 len 16384 data: 2 2 2 2
471 243/300 Read block 4552 len 12288
472 244/300 Write block 4023 len 4096 data: 3
473 245/300 Read block 1539 len 36864
474 246/300 Read block 4796 len 8192
475 247/300 Read block 4685 len 8192
476 248/300 Read block 3029 len 32768
477 249/300 Flush
478 250/300 Write block 268 len 40960 data: 3 3 3 3 3 3 3 2 2 2
479 251/300 Write block 2819 len 32768 data: 2 2 2 2 2 2 2 2
480 252/300 Read block 1785 len 28672
481 253/300 Write block 2800 len 24576 data: 2 2 2 2 2 2
482 254/300 Write block 2107 len 12288 data: 2 2 2
483 255/300 Read block 2685 len 40960
484 256/300 Read block 2626 len 32768
485 257/300 Write block 2499 len 24576 data: 2 2 2 2 2 2
486 258/300 Read block 2813 len 16384
487 259/300 Read block 800 len 20480
488 260/300 Read block 4574 len 12288
489 261/300 Read block 2003 len 32768
490 262/300 Read block 64 len 8192
491 263/300 Write block 1556 len 28672 data: 2 2 2 2 3 2 2
492 264/300 Read block 2245 len 20480
493 265/300 Read block 3812 len 28672
494 266/300 Write block 1806 len 8192 data: 2 2
495 267/300 Write block 623 len 40960 data: 2 2 2 2 2 2 2 2 2 2
496 268/300 Write block 1025 len 4096 data: 2
497 269/300 Read block 3892 len 8192
498 270/300 Read block 820 len 28672
499 271/300 Write block 3170 len 24576 data: 2 2 2 2 2 2
500 272/300 Read block 1990 len 20480
501 273/300 Read block 63 len 32768
502 274/300 Write block 244 len 36864 data: 2 2 2 2 2 2 2 2 2
503 275/300 Read block 3750 len 36864
504 276/300 Write block 721 len 36864 data: 2 2 2 2 2 2 2 2 2
505 277/300 Read block 849 len 8192
506 278/300 Read block 2902 len 4096
507 279/300 Read block 1670 len 28672
508 280/300 Read block 4305 len 4096
509 281/300 Write block 4846 len 24576 data: 2 2 2 2 2 2
510 282/300 Read block 2450 len 36864
511 283/300 Write block 561 len 4096 data: 2
512 284/300 Write block 81 len 20480 data: 2 2 2 2 2
513 285/300 Read block 2118 len 28672
514 286/300 Write block 3882 len 20480 data: 2 2 2 2 2
515 287/300 Flush
516 288/300 Write block 4339 len 28672 data: 2 2 2 2 2 2 2
517 289/300 Flush
518 290/300 Write block 2009 len 8192 data: 2 2
519 291/300 Read block 828 len 36864
520 292/300 Write block 4821 len 20480 data: 2 2 2 2 2
521 293/300 Read block 2336 len 12288
522 294/300 Read block 2181 len 24576
523 295/300 Write block 1505 len 8192 data: 2 2
524 296/300 Write block 4946 len 32768 data: 2 2 2 2 2 2 2 2
525 297/300 Write block 417 len 12288 data: 2 2 2
526 298/300 Write block 3442 len 12288 data: 2 2 2
527 299/300 Write block 4348 len 12288 data: 2 2 2
528 300/300 Read block 4958 len 40960
529 301/300 Flush
530 Main task has work to do, go find it
531 got dsc Start(2)
532 start 2
533 Feb 25 20:46:48.843 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: 278f0994-1417-417d-b521-db104f4f76ac, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
534 Replay: started 2, returned:Ok(())
535 ----------------------------------------------------------------
536 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
537 States: Active Active Offline
538 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
539 1000 Acked Write 2 Done Done Sent false
540 1001 Acked Flush 0 Done Done Sent false
541 1002 Acked Write 3 Done Done Sent false
542 1003 Acked Write 3 Done Done Sent false
543 1004 Acked Read 2 Done Done Sent false
544 1005 Acked Write 5 Done Done Sent false
545 1006 Acked Write 7 Done Done Sent false
546 1007 Acked Write 2 Done Done Sent false
547 1008 Acked Write 3 Done Done Sent false
548 1009 Acked Write 8 Done Done Sent false
549 1010 Acked Read 4 Done Done Sent false
550 1011 Acked Flush 0 Done Done Sent false
551 1012 Acked Write 9 Done Done Sent false
552 1013 Acked Read 6 Done Done Sent false
553 1014 Acked Write 8 Done Done Sent false
554 1015 Acked Write 1 Done Done Sent false
555 1016 Acked Write 1 Done Done Sent false
556 1017 Acked Read 4 Done Done Sent false
557 1018 Acked Read 9 Done Done Sent false
558 1019 Acked Read 10 Done Done Sent false
559 1020 Acked Flush 0 Done Done Sent false
560 1021 Acked Read 5 Done Done Sent false
561 1022 Acked Read 4 Done Done Sent false
562 1023 Acked Write 2 Done Done Sent false
563 1024 Acked Write 1 Done Done Sent false
564 1025 Acked Write 6 Done Done Sent false
565 1026 Acked Write 1 Done Done Sent false
566 1027 Acked Read 6 Done Done Sent false
567 1028 Acked Read 3 Done Done Sent false
568 1029 Acked Read 1 Done Done Sent false
569 1030 Acked Read 1 Done Done Sent false
570 1031 Acked Read 7 Done Done Sent false
571 1032 Acked Write 4 Done Done Sent false
572 1033 Acked Write 1 Done Done Sent false
573 1034 Acked Write 4 Done Done Sent false
574 1035 Acked Flush 0 Done Done Sent false
575 1036 Acked Write 6 Done Done Sent false
576 1037 Acked Write 2 Done Done Sent false
577 1038 Acked Write 3 Done Done Sent false
578 1039 Acked Read 3 Done Done Sent false
579 1040 Acked Read 6 Done Done Sent false
580 1041 Acked Write 8 Done Done Sent false
581 1042 Acked Write 7 Done Done Sent false
582 1043 Acked Write 5 Done Done Sent false
583 1044 Acked Read 10 Done Done Sent false
584 1045 Acked Read 9 Done Done Sent false
585 1046 Acked Write 9 Done Done Sent false
586 1047 Acked Read 6 Done Done Sent false
587 1048 Acked Read 8 Done Done Sent false
588 1049 Acked Write 2 Done Done Sent false
589 1050 Acked Write 4 Done Done Sent false
590 1051 Acked Read 1 Done Done Sent false
591 1052 Acked Read 3 Done Done Sent false
592 1053 Acked Read 7 Done Done Sent false
593 1054 Acked Flush 0 Done Done Sent false
594 1055 Acked Flush 0 Done Done Sent false
595 1056 Acked Write 7 Done Done Sent false
596 1057 Acked Read 5 Done Done Sent false
597 1058 Acked Read 5 Done Done Sent false
598 1059 Acked Write 4 Done Done Sent false
599 1060 Acked Read 2 Done Done Sent false
600 1061 Acked Read 9 Done Done Sent false
601 1062 Acked Read 9 Done Done Sent false
602 1063 Acked Read 7 Done Done Sent false
603 1064 Acked Read 2 Done Done Sent false
604 1065 Acked Read 6 Done Done Sent false
605 1066 Acked Read 10 Done Done Sent false
606 1067 Acked Write 1 Done Done Sent false
607 1068 Acked Write 4 Done Done Sent false
608 1069 Acked Read 1 Done Done Sent false
609 1070 Acked Write 2 Done Done Sent false
610 1071 Acked Write 1 Done Done Sent false
611 1072 Acked Write 6 Done Done Sent false
612 1073 Acked Write 7 Done Done Sent false
613 1074 Acked Write 9 Done Done Sent false
614 1075 Acked Read 2 Done Done Sent false
615 1076 Acked Read 3 Done Done Sent false
616 1077 Acked Write 1 Done Done Sent false
617 1078 Acked Write 8 Done Done Sent false
618 1079 Acked Read 5 Done Done Sent false
619 1080 Acked Read 9 Done Done Sent false
620 1081 Acked Flush 0 Done Done Sent false
621 1082 Acked Read 7 Done Done Sent false
622 1083 Acked Read 8 Done Done Sent false
623 1084 Acked Read 4 Done Done Sent false
624 1085 Acked Read 8 Done Done Sent false
625 1086 Acked Read 10 Done Done Sent false
626 1087 Acked Write 1 Done Done Sent false
627 1088 Acked Write 4 Done Done Sent false
628 1089 Acked Write 1 Done Done Sent false
629 1090 Acked Write 9 Done Done Sent false
630 1091 Acked Write 5 Done Done Sent false
631 1092 Acked Read 9 Done Done Sent false
632 1093 Acked Write 6 Done Done Sent false
633 1094 Acked Read 1 Done Done Sent false
634 1095 Acked Read 7 Done Done Sent false
635 1096 Acked Read 7 Done Done Sent false
636 1097 Acked Write 2 Done Done Sent false
637 1098 Acked Write 2 Done Done Sent false
638 1099 Acked Write 10 Done Done Sent false
639 1100 Acked Read 9 Done Done Sent false
640 1101 Acked Write 1 Done Done Sent false
641 1102 Acked Read 7 Done Done Sent false
642 1103 Acked Read 8 Done Done Sent false
643 1104 Acked Read 5 Done Done Sent false
644 1105 Acked Read 7 Done Done Sent false
645 1106 Acked Read 4 Done Done Sent false
646 1107 Acked Write 3 Done Done Sent false
647 1108 Acked Read 6 Done Done Sent false
648 1109 Acked Read 5 Done Done Sent false
649 1110 Acked Read 3 Done Done Sent false
650 1111 Acked Read 1 Done Done Sent false
651 1112 Acked Read 9 Done Done Sent false
652 1113 Acked Read 1 Done Done Sent false
653 1114 Acked Read 2 Done Done Sent false
654 1115 Acked Write 3 Done Done Sent false
655 1116 Acked Flush 0 Done Done Sent false
656 1117 Acked Write 7 Done Done Sent false
657 1118 Acked Write 2 Done Done Sent false
658 1119 Acked Write 7 Done Done Sent false
659 1120 Acked Read 4 Done Done Sent false
660 1121 Acked Write 10 Done Done Sent false
661 1122 Acked Write 9 Done Done Sent false
662 1123 Acked Write 9 Done Done Sent false
663 1124 Acked Write 8 Done Done Sent false
664 1125 Acked Read 7 Done Done Sent false
665 1126 Acked Flush 0 Done Done Sent false
666 1127 Acked Read 6 Done Done Sent false
667 1128 Acked Write 6 Done Done Sent false
668 1129 Acked Read 1 Done Done Sent false
669 1130 Acked Read 5 Done Done Sent false
670 1131 Acked Read 10 Done Done Sent false
671 1132 Acked Read 6 Done Done Sent false
672 1133 Acked Read 8 Done Done Sent false
673 1134 Acked Read 10 Done Done Sent false
674 1135 Acked Read 2 Done Done Sent false
675 1136 Acked Read 3 Done Done Sent false
676 1137 Acked Read 10 Done Done Sent false
677 1138 Acked Write 6 Done Done Sent false
678 1139 Acked Write 5 Done Done Sent false
679 1140 Acked Flush 0 Done Done Sent false
680 1141 Acked Write 6 Done Done Sent false
681 1142 Acked Write 6 Done Done Sent false
682 1143 Acked Read 4 Done Done Sent false
683 1144 Acked Flush 0 Done Done Sent false
684 1145 Acked Read 7 Done Done Sent false
685 1146 Acked Write 5 Done Done Sent false
686 1147 Acked Read 8 Done Done Sent false
687 1148 Acked Write 4 Done Done Sent false
688 1149 Acked Flush 0 Done Done Sent false
689 1150 Acked Write 2 Done Done Sent false
690 1151 Acked Write 7 Done Done Sent false
691 1152 Acked Write 4 Done Done Sent false
692 1153 Acked Write 1 Done Done Sent false
693 1154 Acked Write 6 Done Done Sent false
694 1155 Acked Read 3 Done Done Sent false
695 1156 Acked Write 10 Done Done Sent false
696 1157 Acked Flush 0 Done Done Sent false
697 1158 Acked Write 10 Done Done Sent false
698 1159 Acked Flush 0 Done Done Sent false
699 1160 Acked Flush 0 Done Done Sent false
700 1161 Acked Read 9 Done Done Sent false
701 1162 Acked Flush 0 Done Done Sent false
702 1163 Acked Read 8 Done Done Sent false
703 1164 Acked Read 1 Done Done Sent false
704 1165 Acked Read 1 Done Done Sent false
705 1166 Acked Read 6 Done Done Sent false
706 1167 Acked Write 2 Done Done Sent false
707 1168 Acked Read 2 Done Done Sent false
708 1169 Acked Read 4 Done Done Sent false
709 1170 Acked Read 3 Done Done Sent false
710 1171 Acked Flush 0 Done Done Sent false
711 1172 Acked Write 8 Done Done Sent false
712 1173 Acked Read 4 Done Done Sent false
713 1174 Acked Write 10 Done Done Sent false
714 1175 Acked Write 5 Done Done Sent false
715 1176 Acked Read 5 Done Done Sent false
716 1177 Acked Flush 0 Done Done Sent false
717 1178 Acked Flush 0 Done Done Sent false
718 1179 Acked Read 1 Done Done Sent false
719 1180 Acked Write 9 Done Done Sent false
720 1181 Acked Read 2 Done Done Sent false
721 1182 Acked Read 2 Done Done Sent false
722 1183 Acked Read 7 Done Done Sent false
723 1184 Acked Flush 0 Done Done Sent false
724 1185 Acked Write 2 Done Done Sent false
725 1186 Acked Read 10 Done Done Sent false
726 1187 Acked Write 3 Done Done Sent false
727 1188 Acked Read 2 Done Done Sent false
728 1189 Acked Write 10 Done Done Sent false
729 1190 Acked Flush 0 Done Done Sent false
730 1191 Acked Write 1 Done Done Sent false
731 1192 Acked Read 2 Done Done Sent false
732 1193 Acked Write 5 Done Done Sent false
733 1194 Acked Read 9 Done Done Sent false
734 1195 Acked Write 2 Done Done Sent false
735 1196 Acked Write 3 Done Done Sent false
736 1197 Acked Write 7 Done Done Sent false
737 1198 Acked Read 5 Done Done Sent false
738 1199 Acked Write 9 Done Done Sent false
739 1200 Acked Write 9 Done Done Sent false
740 1201 Acked Read 10 Done Done Sent false
741 1202 Acked Read 8 Done Done Sent false
742 1203 Acked Read 2 Done Done Sent false
743 1204 Acked Write 4 Done Done Sent false
744 1205 Acked Read 2 Done Done Sent false
745 1206 Acked Read 2 Done Done Sent false
746 1207 Acked Read 9 Done Done Sent false
747 1208 Acked Read 5 Done Done Sent false
748 1209 Acked Read 7 Done Done Sent false
749 1210 Acked Read 8 Done Done Sent false
750 1211 Acked Read 1 Done Done Sent false
751 1212 Acked Write 7 Done Done Sent false
752 1213 Acked Flush 0 Done Done Sent false
753 1214 Acked Read 9 Done Done Sent false
754 1215 Acked Write 10 Done Done Sent false
755 1216 Acked Write 10 Done Done Sent false
756 1217 Acked Read 7 Done Done Sent false
757 1218 Acked Write 6 Done Done Sent false
758 1219 Acked Write 8 Done Done Sent false
759 1220 Acked Write 2 Done Done Sent false
760 1221 Acked Read 1 Done Done Sent false
761 1222 Acked Write 9 Done Done Sent false
762 1223 Acked Read 3 Done Done Sent false
763 1224 Acked Write 3 Done Done Sent false
764 1225 Acked Write 10 Done Done Sent false
765 1226 Acked Write 5 Done Done Sent false
766 1227 Acked Read 8 Done Done Sent false
767 1228 Acked Flush 0 Done Done Sent false
768 1229 Acked Write 10 Done Done Sent false
769 1230 Acked Flush 0 Done Done Sent false
770 1231 Acked Read 6 Done Done Sent false
771 1232 Acked Read 6 Done Done Sent false
772 1233 Acked Write 10 Done Done Sent false
773 1234 Acked Read 4 Done Done Sent false
774 1235 Acked Flush 0 Done Done Sent false
775 1236 Acked Write 3 Done Done Sent false
776 1237 Acked Write 5 Done Done Sent false
777 1238 Acked Write 8 Done Done Sent false
778 1239 Acked Flush 0 Done Done Sent false
779 1240 Acked Write 6 Done Done Sent false
780 1241 Acked Write 4 Done Done Sent false
781 1242 Acked Read 3 Done Done Sent false
782 1243 Acked Write 1 Done Done Sent false
783 1244 Acked Read 9 Done Done Sent false
784 1245 Acked Read 2 Done Done Sent false
785 1246 Acked Read 2 Done Done Sent false
786 1247 Acked Read 8 Done Done Sent false
787 1248 Acked Flush 0 Done Done Sent false
788 1249 Acked Write 10 Done Done Sent false
789 1250 Acked Write 8 Done Done Sent false
790 1251 Acked Read 7 Done Done Sent false
791 1252 Acked Write 6 Done Done Sent false
792 1253 Acked Write 3 Done Done Sent false
793 1254 Acked Read 10 Done Done Sent false
794 1255 Acked Read 8 Done Done Sent false
795 1256 Acked Write 6 Done Done Sent false
796 1257 Acked Read 4 Done Done Sent false
797 1258 Acked Read 5 Done Done Sent false
798 1259 Acked Read 3 Done Done Sent false
799 1260 Acked Read 8 Done Done Sent false
800 1261 Acked Read 2 Done Done Sent false
801 1262 Acked Write 7 Done Done Sent false
802 1263 Acked Read 5 Done Done Sent false
803 1264 Acked Read 7 Done Done Sent false
804 1265 Acked Write 2 Done Done Sent false
805 1266 Acked Write 10 Done Done Sent false
806 1267 Acked Write 1 Done Done Sent false
807 1268 Acked Read 2 Done Done Sent false
808 1269 Acked Read 7 Done Done Sent false
809 1270 Acked Write 6 Done Done Sent false
810 1271 Acked Read 5 Done Done Sent false
811 1272 Acked Read 8 Done Done Sent false
812 1273 Acked Write 9 Done Done Sent false
813 1274 Acked Read 9 Done Done Sent false
814 1275 Acked Write 9 Done Done Sent false
815 1276 Acked Read 2 Done Done Sent false
816 1277 Acked Read 1 Done Done Sent false
817 1278 Acked Read 7 Done Done Sent false
818 1279 Acked Read 1 Done Done Sent false
819 1280 Acked Write 6 Done Done Sent false
820 1281 Acked Read 9 Done Done Sent false
821 1282 Acked Write 1 Done Done Sent false
822 1283 Acked Write 5 Done Done Sent false
823 1284 Acked Read 7 Done Done Sent false
824 1285 Acked Write 5 Done Done Sent false
825 1286 Acked Flush 0 Done Done Sent false
826 1287 Acked Write 7 Done Done Sent false
827 1288 Acked Flush 0 Done Done Sent false
828 1289 Acked Write 2 Done Done Sent false
829 1290 Acked Read 9 Done Done Sent false
830 1291 Acked Write 5 Done Done Sent false
831 1292 Acked Read 3 Done Done Sent false
832 1293 Acked Read 6 Done Done Sent false
833 1294 Acked Write 2 Done Done Sent false
834 1295 Acked Write 8 Done Done Sent false
835 1296 Acked Write 3 Done Done Sent false
836 1297 Acked Write 3 Done Done Sent false
837 1298 Acked Write 3 Done Done Sent false
838 1299 Acked Read 10 Done Done Sent false
839 1300 Acked Flush 0 Done Done Sent false
840 STATES DS:0 DS:1 DS:2 TOTAL
841 Sent 0 0 301 301
842 Done 301 301 0 602
843 Skipped 0 0 0 0
844 Error 0 0 0 0
845 Last Flush: Some(JobId(1300)) Some(JobId(1300)) None
846 Downstairs last five completed:
847 Upstairs last five completed: 1300 1299 1298 1297 1296
848 CLIENT: Up:0 ds:301 act:2
8492025-02-25T20:46:49.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8502025-02-25T20:46:50.074ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
851 [8830] got start action
852 [8830] I am going to restart
853 Starting downstairs at port 8830
854 Make output file at "/tmp/dsc/downstairs-8830.txt"
855 [8830][2] reports Starting
856 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1207)
857 [8830][2] reports Running
8582025-02-25T20:46:51.076ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8592025-02-25T20:46:52.078ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
860 ----------------------------------------------------------------
861 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
862 States: Active Active Offline
863 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
864 1000 Acked Write 2 Done Done Sent false
865 1001 Acked Flush 0 Done Done Sent false
866 1002 Acked Write 3 Done Done Sent false
867 1003 Acked Write 3 Done Done Sent false
868 1004 Acked Read 2 Done Done Sent false
869 1005 Acked Write 5 Done Done Sent false
870 1006 Acked Write 7 Done Done Sent false
871 1007 Acked Write 2 Done Done Sent false
872 1008 Acked Write 3 Done Done Sent false
873 1009 Acked Write 8 Done Done Sent false
874 1010 Acked Read 4 Done Done Sent false
875 1011 Acked Flush 0 Done Done Sent false
876 1012 Acked Write 9 Done Done Sent false
877 1013 Acked Read 6 Done Done Sent false
878 1014 Acked Write 8 Done Done Sent false
879 1015 Acked Write 1 Done Done Sent false
880 1016 Acked Write 1 Done Done Sent false
881 1017 Acked Read 4 Done Done Sent false
882 1018 Acked Read 9 Done Done Sent false
883 1019 Acked Read 10 Done Done Sent false
884 1020 Acked Flush 0 Done Done Sent false
885 1021 Acked Read 5 Done Done Sent false
886 1022 Acked Read 4 Done Done Sent false
887 1023 Acked Write 2 Done Done Sent false
888 1024 Acked Write 1 Done Done Sent false
889 1025 Acked Write 6 Done Done Sent false
890 1026 Acked Write 1 Done Done Sent false
891 1027 Acked Read 6 Done Done Sent false
892 1028 Acked Read 3 Done Done Sent false
893 1029 Acked Read 1 Done Done Sent false
894 1030 Acked Read 1 Done Done Sent false
895 1031 Acked Read 7 Done Done Sent false
896 1032 Acked Write 4 Done Done Sent false
897 1033 Acked Write 1 Done Done Sent false
898 1034 Acked Write 4 Done Done Sent false
899 1035 Acked Flush 0 Done Done Sent false
900 1036 Acked Write 6 Done Done Sent false
901 1037 Acked Write 2 Done Done Sent false
902 1038 Acked Write 3 Done Done Sent false
903 1039 Acked Read 3 Done Done Sent false
904 1040 Acked Read 6 Done Done Sent false
905 1041 Acked Write 8 Done Done Sent false
906 1042 Acked Write 7 Done Done Sent false
907 1043 Acked Write 5 Done Done Sent false
908 1044 Acked Read 10 Done Done Sent false
909 1045 Acked Read 9 Done Done Sent false
910 1046 Acked Write 9 Done Done Sent false
911 1047 Acked Read 6 Done Done Sent false
912 1048 Acked Read 8 Done Done Sent false
913 1049 Acked Write 2 Done Done Sent false
914 1050 Acked Write 4 Done Done Sent false
915 1051 Acked Read 1 Done Done Sent false
916 1052 Acked Read 3 Done Done Sent false
917 1053 Acked Read 7 Done Done Sent false
918 1054 Acked Flush 0 Done Done Sent false
919 1055 Acked Flush 0 Done Done Sent false
920 1056 Acked Write 7 Done Done Sent false
921 1057 Acked Read 5 Done Done Sent false
922 1058 Acked Read 5 Done Done Sent false
923 1059 Acked Write 4 Done Done Sent false
924 1060 Acked Read 2 Done Done Sent false
925 1061 Acked Read 9 Done Done Sent false
926 1062 Acked Read 9 Done Done Sent false
927 1063 Acked Read 7 Done Done Sent false
928 1064 Acked Read 2 Done Done Sent false
929 1065 Acked Read 6 Done Done Sent false
930 1066 Acked Read 10 Done Done Sent false
931 1067 Acked Write 1 Done Done Sent false
932 1068 Acked Write 4 Done Done Sent false
933 1069 Acked Read 1 Done Done Sent false
934 1070 Acked Write 2 Done Done Sent false
935 1071 Acked Write 1 Done Done Sent false
936 1072 Acked Write 6 Done Done Sent false
937 1073 Acked Write 7 Done Done Sent false
938 1074 Acked Write 9 Done Done Sent false
939 1075 Acked Read 2 Done Done Sent false
940 1076 Acked Read 3 Done Done Sent false
941 1077 Acked Write 1 Done Done Sent false
942 1078 Acked Write 8 Done Done Sent false
943 1079 Acked Read 5 Done Done Sent false
944 1080 Acked Read 9 Done Done Sent false
945 1081 Acked Flush 0 Done Done Sent false
946 1082 Acked Read 7 Done Done Sent false
947 1083 Acked Read 8 Done Done Sent false
948 1084 Acked Read 4 Done Done Sent false
949 1085 Acked Read 8 Done Done Sent false
950 1086 Acked Read 10 Done Done Sent false
951 1087 Acked Write 1 Done Done Sent false
952 1088 Acked Write 4 Done Done Sent false
953 1089 Acked Write 1 Done Done Sent false
954 1090 Acked Write 9 Done Done Sent false
955 1091 Acked Write 5 Done Done Sent false
956 1092 Acked Read 9 Done Done Sent false
957 1093 Acked Write 6 Done Done Sent false
958 1094 Acked Read 1 Done Done Sent false
959 1095 Acked Read 7 Done Done Sent false
960 1096 Acked Read 7 Done Done Sent false
961 1097 Acked Write 2 Done Done Sent false
962 1098 Acked Write 2 Done Done Sent false
963 1099 Acked Write 10 Done Done Sent false
964 1100 Acked Read 9 Done Done Sent false
965 1101 Acked Write 1 Done Done Sent false
966 1102 Acked Read 7 Done Done Sent false
967 1103 Acked Read 8 Done Done Sent false
968 1104 Acked Read 5 Done Done Sent false
969 1105 Acked Read 7 Done Done Sent false
970 1106 Acked Read 4 Done Done Sent false
971 1107 Acked Write 3 Done Done Sent false
972 1108 Acked Read 6 Done Done Sent false
973 1109 Acked Read 5 Done Done Sent false
974 1110 Acked Read 3 Done Done Sent false
975 1111 Acked Read 1 Done Done Sent false
976 1112 Acked Read 9 Done Done Sent false
977 1113 Acked Read 1 Done Done Sent false
978 1114 Acked Read 2 Done Done Sent false
979 1115 Acked Write 3 Done Done Sent false
980 1116 Acked Flush 0 Done Done Sent false
981 1117 Acked Write 7 Done Done Sent false
982 1118 Acked Write 2 Done Done Sent false
983 1119 Acked Write 7 Done Done Sent false
984 1120 Acked Read 4 Done Done Sent false
985 1121 Acked Write 10 Done Done Sent false
986 1122 Acked Write 9 Done Done Sent false
987 1123 Acked Write 9 Done Done Sent false
988 1124 Acked Write 8 Done Done Sent false
989 1125 Acked Read 7 Done Done Sent false
990 1126 Acked Flush 0 Done Done Sent false
991 1127 Acked Read 6 Done Done Sent false
992 1128 Acked Write 6 Done Done Sent false
993 1129 Acked Read 1 Done Done Sent false
994 1130 Acked Read 5 Done Done Sent false
995 1131 Acked Read 10 Done Done Sent false
996 1132 Acked Read 6 Done Done Sent false
997 1133 Acked Read 8 Done Done Sent false
998 1134 Acked Read 10 Done Done Sent false
999 1135 Acked Read 2 Done Done Sent false
1000 1136 Acked Read 3 Done Done Sent false
1001 1137 Acked Read 10 Done Done Sent false
1002 1138 Acked Write 6 Done Done Sent false
1003 1139 Acked Write 5 Done Done Sent false
1004 1140 Acked Flush 0 Done Done Sent false
1005 1141 Acked Write 6 Done Done Sent false
1006 1142 Acked Write 6 Done Done Sent false
1007 1143 Acked Read 4 Done Done Sent false
1008 1144 Acked Flush 0 Done Done Sent false
1009 1145 Acked Read 7 Done Done Sent false
1010 1146 Acked Write 5 Done Done Sent false
1011 1147 Acked Read 8 Done Done Sent false
1012 1148 Acked Write 4 Done Done Sent false
1013 1149 Acked Flush 0 Done Done Sent false
1014 1150 Acked Write 2 Done Done Sent false
1015 1151 Acked Write 7 Done Done Sent false
1016 1152 Acked Write 4 Done Done Sent false
1017 1153 Acked Write 1 Done Done Sent false
1018 1154 Acked Write 6 Done Done Sent false
1019 1155 Acked Read 3 Done Done Sent false
1020 1156 Acked Write 10 Done Done Sent false
1021 1157 Acked Flush 0 Done Done Sent false
1022 1158 Acked Write 10 Done Done Sent false
1023 1159 Acked Flush 0 Done Done Sent false
1024 1160 Acked Flush 0 Done Done Sent false
1025 1161 Acked Read 9 Done Done Sent false
1026 1162 Acked Flush 0 Done Done Sent false
1027 1163 Acked Read 8 Done Done Sent false
1028 1164 Acked Read 1 Done Done Sent false
1029 1165 Acked Read 1 Done Done Sent false
1030 1166 Acked Read 6 Done Done Sent false
1031 1167 Acked Write 2 Done Done Sent false
1032 1168 Acked Read 2 Done Done Sent false
1033 1169 Acked Read 4 Done Done Sent false
1034 1170 Acked Read 3 Done Done Sent false
1035 1171 Acked Flush 0 Done Done Sent false
1036 1172 Acked Write 8 Done Done Sent false
1037 1173 Acked Read 4 Done Done Sent false
1038 1174 Acked Write 10 Done Done Sent false
1039 1175 Acked Write 5 Done Done Sent false
1040 1176 Acked Read 5 Done Done Sent false
1041 1177 Acked Flush 0 Done Done Sent false
1042 1178 Acked Flush 0 Done Done Sent false
1043 1179 Acked Read 1 Done Done Sent false
1044 1180 Acked Write 9 Done Done Sent false
1045 1181 Acked Read 2 Done Done Sent false
1046 1182 Acked Read 2 Done Done Sent false
1047 1183 Acked Read 7 Done Done Sent false
1048 1184 Acked Flush 0 Done Done Sent false
1049 1185 Acked Write 2 Done Done Sent false
1050 1186 Acked Read 10 Done Done Sent false
1051 1187 Acked Write 3 Done Done Sent false
1052 1188 Acked Read 2 Done Done Sent false
1053 1189 Acked Write 10 Done Done Sent false
1054 1190 Acked Flush 0 Done Done Sent false
1055 1191 Acked Write 1 Done Done Sent false
1056 1192 Acked Read 2 Done Done Sent false
1057 1193 Acked Write 5 Done Done Sent false
1058 1194 Acked Read 9 Done Done Sent false
1059 1195 Acked Write 2 Done Done Sent false
1060 1196 Acked Write 3 Done Done Sent false
1061 1197 Acked Write 7 Done Done Sent false
1062 1198 Acked Read 5 Done Done Sent false
1063 1199 Acked Write 9 Done Done Sent false
1064 1200 Acked Write 9 Done Done Sent false
1065 1201 Acked Read 10 Done Done Sent false
1066 1202 Acked Read 8 Done Done Sent false
1067 1203 Acked Read 2 Done Done Sent false
1068 1204 Acked Write 4 Done Done Sent false
1069 1205 Acked Read 2 Done Done Sent false
1070 1206 Acked Read 2 Done Done Sent false
1071 1207 Acked Read 9 Done Done Sent false
1072 1208 Acked Read 5 Done Done Sent false
1073 1209 Acked Read 7 Done Done Sent false
1074 1210 Acked Read 8 Done Done Sent false
1075 1211 Acked Read 1 Done Done Sent false
1076 1212 Acked Write 7 Done Done Sent false
1077 1213 Acked Flush 0 Done Done Sent false
1078 1214 Acked Read 9 Done Done Sent false
1079 1215 Acked Write 10 Done Done Sent false
1080 1216 Acked Write 10 Done Done Sent false
1081 1217 Acked Read 7 Done Done Sent false
1082 1218 Acked Write 6 Done Done Sent false
1083 1219 Acked Write 8 Done Done Sent false
1084 1220 Acked Write 2 Done Done Sent false
1085 1221 Acked Read 1 Done Done Sent false
1086 1222 Acked Write 9 Done Done Sent false
1087 1223 Acked Read 3 Done Done Sent false
1088 1224 Acked Write 3 Done Done Sent false
1089 1225 Acked Write 10 Done Done Sent false
1090 1226 Acked Write 5 Done Done Sent false
1091 1227 Acked Read 8 Done Done Sent false
1092 1228 Acked Flush 0 Done Done Sent false
1093 1229 Acked Write 10 Done Done Sent false
1094 1230 Acked Flush 0 Done Done Sent false
1095 1231 Acked Read 6 Done Done Sent false
1096 1232 Acked Read 6 Done Done Sent false
1097 1233 Acked Write 10 Done Done Sent false
1098 1234 Acked Read 4 Done Done Sent false
1099 1235 Acked Flush 0 Done Done Sent false
1100 1236 Acked Write 3 Done Done Sent false
1101 1237 Acked Write 5 Done Done Sent false
1102 1238 Acked Write 8 Done Done Sent false
1103 1239 Acked Flush 0 Done Done Sent false
1104 1240 Acked Write 6 Done Done Sent false
1105 1241 Acked Write 4 Done Done Sent false
1106 1242 Acked Read 3 Done Done Sent false
1107 1243 Acked Write 1 Done Done Sent false
1108 1244 Acked Read 9 Done Done Sent false
1109 1245 Acked Read 2 Done Done Sent false
1110 1246 Acked Read 2 Done Done Sent false
1111 1247 Acked Read 8 Done Done Sent false
1112 1248 Acked Flush 0 Done Done Sent false
1113 1249 Acked Write 10 Done Done Sent false
1114 1250 Acked Write 8 Done Done Sent false
1115 1251 Acked Read 7 Done Done Sent false
1116 1252 Acked Write 6 Done Done Sent false
1117 1253 Acked Write 3 Done Done Sent false
1118 1254 Acked Read 10 Done Done Sent false
1119 1255 Acked Read 8 Done Done Sent false
1120 1256 Acked Write 6 Done Done Sent false
1121 1257 Acked Read 4 Done Done Sent false
1122 1258 Acked Read 5 Done Done Sent false
1123 1259 Acked Read 3 Done Done Sent false
1124 1260 Acked Read 8 Done Done Sent false
1125 1261 Acked Read 2 Done Done Sent false
1126 1262 Acked Write 7 Done Done Sent false
1127 1263 Acked Read 5 Done Done Sent false
1128 1264 Acked Read 7 Done Done Sent false
1129 1265 Acked Write 2 Done Done Sent false
1130 1266 Acked Write 10 Done Done Sent false
1131 1267 Acked Write 1 Done Done Sent false
1132 1268 Acked Read 2 Done Done Sent false
1133 1269 Acked Read 7 Done Done Sent false
1134 1270 Acked Write 6 Done Done Sent false
1135 1271 Acked Read 5 Done Done Sent false
1136 1272 Acked Read 8 Done Done Sent false
1137 1273 Acked Write 9 Done Done Sent false
1138 1274 Acked Read 9 Done Done Sent false
1139 1275 Acked Write 9 Done Done Sent false
1140 1276 Acked Read 2 Done Done Sent false
1141 1277 Acked Read 1 Done Done Sent false
1142 1278 Acked Read 7 Done Done Sent false
1143 1279 Acked Read 1 Done Done Sent false
1144 1280 Acked Write 6 Done Done Sent false
1145 1281 Acked Read 9 Done Done Sent false
1146 1282 Acked Write 1 Done Done Sent false
1147 1283 Acked Write 5 Done Done Sent false
1148 1284 Acked Read 7 Done Done Sent false
1149 1285 Acked Write 5 Done Done Sent false
1150 1286 Acked Flush 0 Done Done Sent false
1151 1287 Acked Write 7 Done Done Sent false
1152 1288 Acked Flush 0 Done Done Sent false
1153 1289 Acked Write 2 Done Done Sent false
1154 1290 Acked Read 9 Done Done Sent false
1155 1291 Acked Write 5 Done Done Sent false
1156 1292 Acked Read 3 Done Done Sent false
1157 1293 Acked Read 6 Done Done Sent false
1158 1294 Acked Write 2 Done Done Sent false
1159 1295 Acked Write 8 Done Done Sent false
1160 1296 Acked Write 3 Done Done Sent false
1161 1297 Acked Write 3 Done Done Sent false
1162 1298 Acked Write 3 Done Done Sent false
1163 1299 Acked Read 10 Done Done Sent false
1164 1300 Acked Flush 0 Done Done Sent false
1165 STATES DS:0 DS:1 DS:2 TOTAL
1166 Sent 0 0 301 301
1167 Done 301 301 0 602
1168 Skipped 0 0 0 0
1169 Error 0 0 0 0
1170 Last Flush: Some(JobId(1300)) Some(JobId(1300)) None
1171 Downstairs last five completed:
1172 Upstairs last five completed: 1300 1299 1298 1297 1296
1173 CLIENT: Up:0 ds:301 act:2
11742025-02-25T20:46:53.348ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11752025-02-25T20:46:54.082ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11762025-02-25T20:46:54.604ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11772025-02-25T20:46:54.604ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55154) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11782025-02-25T20:46:54.605ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11792025-02-25T20:46:54.606ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11802025-02-25T20:46:54.606ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11812025-02-25T20:46:54.606ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11822025-02-25T20:46:54.606ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11832025-02-25T20:46:54.606ZINFOcrucible: [2] client re-new 301 jobs since flush None = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11842025-02-25T20:46:54.606ZINFOcrucible: [2] Replayed 301 jobs since flush: None = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1185 ----------------------------------------------------------------
1186 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1187 Downstairs last five completed: 1300 1299 1298 1297 1296
1188 Upstairs last five completed: 1300 1299 1298 1297 1296
1189 CLIENT: Up:0 ds:0 act:3
1190 Replay: All jobs finished
1191 Main task has work to do, go find it
1192 got dsc Stop(1)
1193 stop 1
1194 Feb 25 20:46:56.848 INFO request completed, latency_us: 373, response_code: 204[8820] Got stop action so:false kr:false
1195 , uri: /stop/cid/1, method: POST, req_id: cb9fb31f-2365-4f2b-9e8b-6a06919d229f, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
11962025-02-25T20:46:56.849ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11972025-02-25T20:46:56.849ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11982025-02-25T20:46:56.849ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1199 Feb 25 20:46:56.849 INFO request completed, latency_us: 159, response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:46:56.849701885Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
1200 , uri: /state/cid/1, method: GET, req_id: f428bbfd-9d6f-4a5e-9f03-941afac336a4, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
1201 [8820] Exited with: ExitStatus(unix_wait_status(9))
1202 [8820][1] reports Exit
1203 Feb 25 20:47:00.852 INFO request completed, latency_us: 281, response_code: 200, uri: /state/cid/1, method: GET, req_id: cf489df0-b099-49b1-a65e-0e770f68eb54, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
1204 001/300 Read block 1773 len 40960
1205 002/300 Flush
1206 003/300 Flush
1207 004/300 Read block 4637 len 28672
1208 005/300 Flush
1209 006/300 Write block 2582 len 28672 data: 2 2 2 2 2 2 2
1210 007/300 Write block 1266 len 8192 data: 2 2
1211 008/300 Read block 252 len 4096
1212 009/300 Read block 1833 len 12288
1213 010/300 Read block 4342 len 12288
1214 011/300 Write block 4733 len 36864 data: 2 2 2 2 2 2 2 2 2
1215 012/300 Read block 2163 len 36864
1216 013/300 Read block 2453 len 4096
1217 014/300 Read block 1325 len 36864
1218 015/300 Read block 692 len 24576
1219 016/300 Write block 3200 len 36864 data: 3 3 3 2 2 2 2 2 2
1220 017/300 Write block 3877 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1221 018/300 Write block 3977 len 4096 data: 2
1222 019/300 Read block 3568 len 20480
1223 020/300 Write block 2036 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1224 021/300 Write block 262 len 16384 data: 2 2 2 2
1225 022/300 Write block 1946 len 36864 data: 2 2 2 2 2 2 2 2 2
1226 023/300 Write block 2853 len 16384 data: 2 2 2 2
1227 024/300 Write block 1612 len 24576 data: 2 2 2 2 2 2
1228 025/300 Read block 1360 len 24576
1229 026/300 Flush
1230 027/300 Flush
1231 028/300 Read block 2538 len 24576
1232 029/300 Read block 1150 len 32768
1233 030/300 Write block 576 len 20480 data: 2 2 2 2 2
1234 031/300 Write block 2813 len 4096 data: 2
1235 032/300 Read block 2222 len 36864
1236 033/300 Read block 4520 len 20480
1237 034/300 Read block 3879 len 32768
1238 035/300 Read block 2389 len 36864
1239 036/300 Read block 3161 len 24576
1240 037/300 Read block 41 len 40960
1241 038/300 Read block 1915 len 12288
1242 039/300 Write block 266 len 4096 data: 2
1243 040/300 Read block 2257 len 20480
1244 041/300 Read block 3546 len 24576
1245 042/300 Read block 887 len 28672
1246 043/300 Write block 933 len 8192 data: 2 2
1247 044/300 Read block 800 len 16384
1248 045/300 Write block 884 len 32768 data: 2 2 2 2 2 2 2 2
1249 046/300 Read block 1770 len 4096
1250 047/300 Read block 216 len 36864
1251 048/300 Read block 1827 len 4096
1252 049/300 Read block 65 len 20480
1253 050/300 Write block 725 len 28672 data: 3 3 3 3 3 2 2
1254 051/300 Write block 2122 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1255 052/300 Flush
1256 053/300 Read block 1374 len 36864
1257 054/300 Read block 3426 len 32768
1258 055/300 Read block 3948 len 8192
1259 056/300 Read block 1990 len 24576
1260 057/300 Read block 1312 len 12288
1261 058/300 Flush
1262 059/300 Read block 4383 len 8192
1263 060/300 Read block 36 len 12288
1264 061/300 Read block 1128 len 28672
1265 062/300 Write block 1910 len 36864 data: 2 2 2 2 2 2 2 2 2
1266 063/300 Read block 309 len 28672
1267 064/300 Write block 536 len 4096 data: 2
1268 065/300 Flush
1269 066/300 Read block 3416 len 40960
1270 067/300 Flush
1271 068/300 Read block 4336 len 32768
1272 069/300 Write block 328 len 20480 data: 2 2 2 2 2
1273 070/300 Write block 4242 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1274 071/300 Read block 199 len 8192
1275 072/300 Write block 1750 len 4096 data: 2
1276 073/300 Read block 3452 len 20480
1277 074/300 Read block 1803 len 20480
1278 075/300 Write block 1484 len 4096 data: 2
1279 076/300 Read block 1880 len 28672
1280 077/300 Read block 3576 len 24576
1281 078/300 Read block 4168 len 32768
1282 079/300 Read block 2808 len 40960
1283 080/300 Read block 3145 len 4096
1284 081/300 Read block 3688 len 8192
1285 082/300 Read block 2284 len 4096
1286 083/300 Read block 4234 len 20480
1287 084/300 Write block 1996 len 20480 data: 2 2 2 2 2
1288 085/300 Write block 702 len 24576 data: 2 2 3 3 3 2
1289 086/300 Flush
1290 087/300 Read block 1473 len 4096
1291 088/300 Write block 1476 len 24576 data: 2 2 2 2 2 2
1292 089/300 Read block 3827 len 40960
1293 090/300 Read block 4268 len 4096
1294 091/300 Write block 1580 len 4096 data: 2
1295 092/300 Write block 722 len 4096 data: 3
1296 093/300 Read block 1980 len 16384
1297 094/300 Read block 3895 len 12288
1298 095/300 Write block 2272 len 4096 data: 2
1299 096/300 Write block 4499 len 4096 data: 2
1300 097/300 Read block 653 len 32768
1301 098/300 Read block 1677 len 28672
1302 099/300 Write block 182 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1303 100/300 Write block 4368 len 12288 data: 2 2 2
1304 101/300 Read block 369 len 24576
1305 102/300 Write block 1003 len 12288 data: 2 2 2
1306 103/300 Read block 2847 len 28672
1307 104/300 Read block 4186 len 24576
1308 105/300 Write block 4716 len 4096 data: 2
1309 106/300 Read block 4414 len 16384
1310 107/300 Write block 4851 len 24576 data: 3 2 2 2 2 2
1311 108/300 Write block 1748 len 28672 data: 2 2 3 2 2 2 2
1312 109/300 Write block 25 len 12288 data: 2 2 2
1313 110/300 Write block 3860 len 16384 data: 2 2 2 2
1314 111/300 Read block 3385 len 16384
1315 112/300 Write block 3642 len 36864 data: 4 4 4 4 4 2 2 2 2
1316 113/300 Read block 4327 len 40960
1317 114/300 Write block 3871 len 32768 data: 2 2 2 2 2 2 3 3
1318 115/300 Write block 4745 len 12288 data: 2 2 2
1319 116/300 Write block 2987 len 8192 data: 2 2
1320 117/300 Flush
1321 118/300 Write block 3950 len 36864 data: 2 2 2 2 2 2 2 2 2
1322 119/300 Write block 1582 len 12288 data: 2 2 2
1323 120/300 Read block 973 len 12288
1324 121/300 Write block 2152 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1325 122/300 Read block 4967 len 40960
1326 123/300 Flush
1327 124/300 Read block 4972 len 40960
1328 125/300 Read block 1691 len 12288
1329 126/300 Read block 3748 len 4096
1330 127/300 Read block 4452 len 40960
1331 128/300 Write block 4454 len 8192 data: 2 2
1332 129/300 Write block 1727 len 4096 data: 2
1333 130/300 Read block 2627 len 36864
1334 131/300 Read block 1404 len 28672
1335 132/300 Write block 1036 len 20480 data: 3 3 3 3 2
1336 133/300 Read block 3677 len 24576
1337 134/300 Write block 3520 len 28672 data: 2 2 2 2 2 2 2
1338 135/300 Write block 4426 len 24576 data: 3 3 2 2 2 2
1339 136/300 Read block 2074 len 32768
1340 137/300 Flush
1341 138/300 Write block 3646 len 20480 data: 5 3 3 3 3
1342 139/300 Write block 3679 len 12288 data: 2 2 2
1343 140/300 Flush
1344 141/300 Write block 2081 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1345 142/300 Read block 3007 len 36864
1346 143/300 Read block 947 len 8192
1347 144/300 Write block 4459 len 4096 data: 2
1348 145/300 Read block 3532 len 36864
1349 146/300 Read block 3374 len 32768
1350 147/300 Write block 1730 len 12288 data: 2 2 2
1351 148/300 Write block 267 len 12288 data: 2 4 4
1352 149/300 Write block 4066 len 8192 data: 2 2
1353 150/300 Read block 4752 len 28672
1354 151/300 Read block 355 len 32768
1355 152/300 Read block 3885 len 28672
1356 153/300 Write block 2031 len 4096 data: 2
1357 154/300 Write block 1985 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1358 155/300 Write block 3786 len 8192 data: 2 2
1359 156/300 Write block 3346 len 4096 data: 2
1360 157/300 Write block 684 len 20480 data: 2 2 2 2 2
1361 158/300 Read block 2468 len 12288
1362 159/300 Read block 4589 len 36864
1363 160/300 Write block 1779 len 4096 data: 2
1364 161/300 Write block 2708 len 16384 data: 2 2 2 2
1365 162/300 Write block 4590 len 24576 data: 2 2 2 2 2 2
1366 163/300 Read block 1003 len 36864
1367 164/300 Write block 4378 len 4096 data: 2
1368 165/300 Read block 1558 len 40960
1369 166/300 Write block 2534 len 4096 data: 2
1370 167/300 Read block 275 len 4096
1371 168/300 Write block 3654 len 20480 data: 2 2 2 2 2
1372 169/300 Read block 2668 len 20480
1373 170/300 Read block 4053 len 28672
1374 171/300 Read block 3578 len 32768
1375 172/300 Flush
1376 173/300 Read block 3525 len 8192
1377 174/300 Write block 1600 len 4096 data: 2
1378 175/300 Flush
1379 176/300 Read block 3001 len 32768
1380 177/300 Read block 2885 len 16384
1381 178/300 Read block 846 len 4096
1382 179/300 Write block 92 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1383 180/300 Read block 527 len 40960
1384 181/300 Write block 2890 len 4096 data: 2
1385 182/300 Read block 2571 len 12288
1386 183/300 Read block 3489 len 4096
1387 184/300 Write block 4782 len 8192 data: 3 3
1388 185/300 Read block 4663 len 8192
1389 186/300 Read block 2144 len 16384
1390 187/300 Read block 2585 len 28672
1391 188/300 Read block 3384 len 20480
1392 189/300 Flush
1393 190/300 Read block 3317 len 28672
1394 191/300 Write block 3174 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1395 192/300 Write block 3759 len 36864 data: 2 2 2 2 2 2 2 2 2
1396 193/300 Read block 476 len 28672
1397 194/300 Write block 2923 len 32768 data: 2 2 2 2 2 2 2 2
1398 195/300 Read block 883 len 8192
1399 196/300 Read block 3313 len 12288
1400 197/300 Write block 2426 len 24576 data: 2 2 2 2 2 2
1401 198/300 Read block 1652 len 8192
1402 199/300 Write block 3660 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1403 200/300 Read block 2471 len 20480
1404 201/300 Write block 2434 len 16384 data: 2 2 2 2
1405 202/300 Read block 3734 len 4096
1406 203/300 Flush
1407 204/300 Write block 1126 len 16384 data: 2 2 2 2
1408 205/300 Read block 441 len 20480
1409 206/300 Write block 281 len 36864 data: 2 2 2 3 2 2 2 2 2
1410 207/300 Write block 2254 len 12288 data: 2 2 2
1411 208/300 Flush
1412 209/300 Write block 2537 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1413 210/300 Read block 3252 len 20480
1414 211/300 Write block 2471 len 12288 data: 3 2 2
1415 212/300 Write block 1462 len 24576 data: 2 2 2 2 2 2
1416 213/300 Read block 413 len 32768
1417 214/300 Read block 4835 len 28672
1418 215/300 Write block 2889 len 8192 data: 2 3
1419 216/300 Flush
1420 217/300 Read block 3580 len 36864
1421 218/300 Read block 2607 len 12288
1422 219/300 Write block 2988 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1423 220/300 Write block 2757 len 12288 data: 2 2 2
1424 221/300 Write block 1317 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1425 222/300 Write block 4314 len 28672 data: 2 2 2 2 2 2 2
1426 223/300 Read block 807 len 12288
1427 224/300 Read block 779 len 20480
1428 225/300 Read block 3880 len 32768
1429 226/300 Write block 2710 len 4096 data: 3
1430 227/300 Read block 3518 len 16384
1431 228/300 Read block 2595 len 36864
1432 229/300 Write block 4246 len 8192 data: 3 3
1433 230/300 Read block 2152 len 20480
1434 231/300 Read block 3793 len 40960
1435 232/300 Read block 2208 len 36864
1436 233/300 Read block 2271 len 16384
1437 234/300 Write block 4100 len 20480 data: 4 4 4 4 4
1438 235/300 Write block 1624 len 36864 data: 2 2 2 2 2 2 2 2 2
1439 236/300 Flush
1440 237/300 Flush
1441 238/300 Write block 2203 len 24576 data: 3 3 2 2 2 3
1442 239/300 Read block 2940 len 8192
1443 240/300 Read block 394 len 32768
1444 241/300 Read block 3658 len 12288
1445 242/300 Read block 1188 len 20480
1446 243/300 Read block 4744 len 16384
1447 244/300 Read block 4605 len 32768
1448 245/300 Flush
1449 246/300 Read block 4891 len 12288
1450 247/300 Write block 3026 len 28672 data: 3 3 3 3 3 2 2
1451 248/300 Read block 4687 len 32768
1452 249/300 Read block 4470 len 32768
1453 250/300 Flush
1454 251/300 Write block 1204 len 36864 data: 2 2 2 2 2 2 2 2 2
1455 252/300 Read block 1009 len 24576
1456 253/300 Read block 2289 len 32768
1457 254/300 Read block 4526 len 16384
1458 255/300 Read block 2616 len 32768
1459 256/300 Write block 250 len 32768 data: 3 3 3 2 2 2 2 2
1460 257/300 Read block 260 len 20480
1461 258/300 Read block 2854 len 32768
1462 259/300 Write block 3003 len 32768 data: 2 2 3 3 2 2 3 2
1463 260/300 Flush
1464 261/300 Read block 3970 len 16384
1465 262/300 Read block 4608 len 36864
1466 263/300 Flush
1467 264/300 Flush
1468 265/300 Read block 3031 len 24576
1469 266/300 Write block 1860 len 20480 data: 2 2 2 2 2
1470 267/300 Write block 1163 len 28672 data: 2 2 2 2 2 2 2
1471 268/300 Write block 161 len 12288 data: 2 2 2
1472 269/300 Write block 3180 len 12288 data: 3 3 3
1473 270/300 Flush
1474 271/300 Read block 3566 len 12288
1475 272/300 Read block 2543 len 28672
1476 273/300 Read block 2243 len 24576
1477 274/300 Flush
1478 275/300 Write block 1218 len 4096 data: 2
1479 276/300 Read block 4204 len 4096
1480 277/300 Write block 1531 len 20480 data: 3 3 3 3 3
1481 278/300 Write block 986 len 24576 data: 2 2 2 2 2 2
1482 279/300 Read block 246 len 12288
1483 280/300 Write block 3006 len 28672 data: 4 3 3 4 3 2 2
1484 281/300 Write block 548 len 36864 data: 2 2 2 2 2 2 2 2 2
1485 282/300 Read block 280 len 4096
1486 283/300 Read block 1475 len 24576
1487 284/300 Read block 3441 len 28672
1488 285/300 Write block 2514 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1489 286/300 Read block 1714 len 24576
1490 287/300 Read block 3806 len 24576
1491 288/300 Read block 2057 len 36864
1492 289/300 Write block 87 len 4096 data: 2
1493 290/300 Write block 527 len 32768 data: 2 2 2 2 2 2 2 2
1494 291/300 Write block 4428 len 16384 data: 3 3 3 3
1495 292/300 Read block 3455 len 28672
1496 293/300 Read block 1683 len 8192
1497 294/300 Read block 780 len 8192
1498 295/300 Read block 366 len 36864
1499 296/300 Write block 2889 len 8192 data: 3 4
1500 297/300 Write block 2824 len 24576 data: 3 3 3 2 2 2
1501 298/300 Read block 4664 len 12288
1502 299/300 Write block 160 len 28672 data: 2 3 3 3 2 2 2
1503 300/300 Read block 362 len 40960
1504 301/300 Read block 2479 len 4096
1505 Main task has work to do, go find it
1506 got dsc Start(1)
1507 start 1
1508 Feb 25 20:47:01.100 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/1, method: POST, req_id: 1d037edb-84e1-495f-9ae9-854c895a42fd, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
1509 Replay: started 1, returned:Ok(())
1510 ----------------------------------------------------------------
1511 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1512 States: Active Offline Active
1513 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1514 1301 Acked Read 10 Done Sent Done false
1515 1302 Acked Flush 0 Done Sent Done false
1516 1303 Acked Flush 0 Done Sent Done false
1517 1304 Acked Read 7 Done Sent Done false
1518 1305 Acked Flush 0 Done Sent Done false
1519 1306 Acked Write 7 Done Sent Done false
1520 1307 Acked Write 2 Done Sent Done false
1521 1308 Acked Read 1 Done Sent Done false
1522 1309 Acked Read 3 Done Sent Done false
1523 1310 Acked Read 3 Done Sent Done false
1524 1311 Acked Write 9 Done Sent Done false
1525 1312 Acked Read 9 Done Sent Done false
1526 1313 Acked Read 1 Done Sent Done false
1527 1314 Acked Read 9 Done Sent Done false
1528 1315 Acked Read 6 Done Sent Done false
1529 1316 Acked Write 9 Done Sent Done false
1530 1317 Acked Write 10 Done Sent Done false
1531 1318 Acked Write 1 Done Sent Done false
1532 1319 Acked Read 5 Done Sent Done false
1533 1320 Acked Write 10 Done Sent Done false
1534 1321 Acked Write 4 Done Sent Done false
1535 1322 Acked Write 9 Done Sent Done false
1536 1323 Acked Write 4 Done Sent Done false
1537 1324 Acked Write 6 Done Sent Done false
1538 1325 Acked Read 6 Done Sent Done false
1539 1326 Acked Flush 0 Done Sent Done false
1540 1327 Acked Flush 0 Done Sent Done false
1541 1328 Acked Read 6 Done Sent Done false
1542 1329 Acked Read 8 Done Sent Done false
1543 1330 Acked Write 5 Done Sent Done false
1544 1331 Acked Write 1 Done Sent Done false
1545 1332 Acked Read 9 Done Sent Done false
1546 1333 Acked Read 5 Done Sent Done false
1547 1334 Acked Read 8 Done Sent Done false
1548 1335 Acked Read 9 Done Sent Done false
1549 1336 Acked Read 6 Done Sent Done false
1550 1337 Acked Read 10 Done Sent Done false
1551 1338 Acked Read 3 Done Sent Done false
1552 1339 Acked Write 1 Done Sent Done false
1553 1340 Acked Read 5 Done Sent Done false
1554 1341 Acked Read 6 Done Sent Done false
1555 1342 Acked Read 7 Done Sent Done false
1556 1343 Acked Write 2 Done Sent Done false
1557 1344 Acked Read 4 Done Sent Done false
1558 1345 Acked Write 8 Done Sent Done false
1559 1346 Acked Read 1 Done Sent Done false
1560 1347 Acked Read 9 Done Sent Done false
1561 1348 Acked Read 1 Done Sent Done false
1562 1349 Acked Read 5 Done Sent Done false
1563 1350 Acked Write 7 Done Sent Done false
1564 1351 Acked Write 10 Done Sent Done false
1565 1352 Acked Flush 0 Done Sent Done false
1566 1353 Acked Read 9 Done Sent Done false
1567 1354 Acked Read 8 Done Sent Done false
1568 1355 Acked Read 2 Done Sent Done false
1569 1356 Acked Read 6 Done Sent Done false
1570 1357 Acked Read 3 Done Sent Done false
1571 1358 Acked Flush 0 Done Sent Done false
1572 1359 Acked Read 2 Done Sent Done false
1573 1360 Acked Read 3 Done Sent Done false
1574 1361 Acked Read 7 Done Sent Done false
1575 1362 Acked Write 9 Done Sent Done false
1576 1363 Acked Read 7 Done Sent Done false
1577 1364 Acked Write 1 Done Sent Done false
1578 1365 Acked Flush 0 Done Sent Done false
1579 1366 Acked Read 10 Done Sent Done false
1580 1367 Acked Flush 0 Done Sent Done false
1581 1368 Acked Read 8 Done Sent Done false
1582 1369 Acked Write 5 Done Sent Done false
1583 1370 Acked Write 10 Done Sent Done false
1584 1371 Acked Read 2 Done Sent Done false
1585 1372 Acked Write 1 Done Sent Done false
1586 1373 Acked Read 5 Done Sent Done false
1587 1374 Acked Read 5 Done Sent Done false
1588 1375 Acked Write 1 Done Sent Done false
1589 1376 Acked Read 7 Done Sent Done false
1590 1377 Acked Read 6 Done Sent Done false
1591 1378 Acked Read 8 Done Sent Done false
1592 1379 Acked Read 10 Done Sent Done false
1593 1380 Acked Read 1 Done Sent Done false
1594 1381 Acked Read 2 Done Sent Done false
1595 1382 Acked Read 1 Done Sent Done false
1596 1383 Acked Read 5 Done Sent Done false
1597 1384 Acked Write 5 Done Sent Done false
1598 1385 Acked Write 6 Done Sent Done false
1599 1386 Acked Flush 0 Done Sent Done false
1600 1387 Acked Read 1 Done Sent Done false
1601 1388 Acked Write 6 Done Sent Done false
1602 1389 Acked Read 10 Done Sent Done false
1603 1390 Acked Read 1 Done Sent Done false
1604 1391 Acked Write 1 Done Sent Done false
1605 1392 Acked Write 1 Done Sent Done false
1606 1393 Acked Read 4 Done Sent Done false
1607 1394 Acked Read 3 Done Sent Done false
1608 1395 Acked Write 1 Done Sent Done false
1609 1396 Acked Write 1 Done Sent Done false
1610 1397 Acked Read 8 Done Sent Done false
1611 1398 Acked Read 7 Done Sent Done false
1612 1399 Acked Write 10 Done Sent Done false
1613 1400 Acked Write 3 Done Sent Done false
1614 1401 Acked Read 6 Done Sent Done false
1615 1402 Acked Write 3 Done Sent Done false
1616 1403 Acked Read 7 Done Sent Done false
1617 1404 Acked Read 6 Done Sent Done false
1618 1405 Acked Write 1 Done Sent Done false
1619 1406 Acked Read 4 Done Sent Done false
1620 1407 Acked Write 6 Done Sent Done false
1621 1408 Acked Write 7 Done Sent Done false
1622 1409 Acked Write 3 Done Sent Done false
1623 1410 Acked Write 4 Done Sent Done false
1624 1411 Acked Read 4 Done Sent Done false
1625 1412 Acked Write 9 Done Sent Done false
1626 1413 Acked Read 10 Done Sent Done false
1627 1414 Acked Write 8 Done Sent Done false
1628 1415 Acked Write 3 Done Sent Done false
1629 1416 Acked Write 2 Done Sent Done false
1630 1417 Acked Flush 0 Done Sent Done false
1631 1418 Acked Write 9 Done Sent Done false
1632 1419 Acked Write 3 Done Sent Done false
1633 1420 Acked Read 3 Done Sent Done false
1634 1421 Acked Write 10 Done Sent Done false
1635 1422 Acked Read 10 Done Sent Done false
1636 1423 Acked Flush 0 Done Sent Done false
1637 1424 Acked Read 10 Done Sent Done false
1638 1425 Acked Read 3 Done Sent Done false
1639 1426 Acked Read 1 Done Sent Done false
1640 1427 Acked Read 10 Done Sent Done false
1641 1428 Acked Write 2 Done Sent Done false
1642 1429 Acked Write 1 Done Sent Done false
1643 1430 Acked Read 9 Done Sent Done false
1644 1431 Acked Read 7 Done Sent Done false
1645 1432 Acked Write 5 Done Sent Done false
1646 1433 Acked Read 6 Done Sent Done false
1647 1434 Acked Write 7 Done Sent Done false
1648 1435 Acked Write 6 Done Sent Done false
1649 1436 Acked Read 8 Done Sent Done false
1650 1437 Acked Flush 0 Done Sent Done false
1651 1438 Acked Write 5 Done Sent Done false
1652 1439 Acked Write 3 Done Sent Done false
1653 1440 Acked Flush 0 Done Sent Done false
1654 1441 Acked Write 10 Done Sent Done false
1655 1442 Acked Read 9 Done Sent Done false
1656 1443 Acked Read 2 Done Sent Done false
1657 1444 Acked Write 1 Done Sent Done false
1658 1445 Acked Read 9 Done Sent Done false
1659 1446 Acked Read 8 Done Sent Done false
1660 1447 Acked Write 3 Done Sent Done false
1661 1448 Acked Write 3 Done Sent Done false
1662 1449 Acked Write 2 Done Sent Done false
1663 1450 Acked Read 7 Done Sent Done false
1664 1451 Acked Read 8 Done Sent Done false
1665 1452 Acked Read 7 Done Sent Done false
1666 1453 Acked Write 1 Done Sent Done false
1667 1454 Acked Write 10 Done Sent Done false
1668 1455 Acked Write 2 Done Sent Done false
1669 1456 Acked Write 1 Done Sent Done false
1670 1457 Acked Write 5 Done Sent Done false
1671 1458 Acked Read 3 Done Sent Done false
1672 1459 Acked Read 9 Done Sent Done false
1673 1460 Acked Write 1 Done Sent Done false
1674 1461 Acked Write 4 Done Sent Done false
1675 1462 Acked Write 6 Done Sent Done false
1676 1463 Acked Read 9 Done Sent Done false
1677 1464 Acked Write 1 Done Sent Done false
1678 1465 Acked Read 10 Done Sent Done false
1679 1466 Acked Write 1 Done Sent Done false
1680 1467 Acked Read 1 Done Sent Done false
1681 1468 Acked Write 5 Done Sent Done false
1682 1469 Acked Read 5 Done Sent Done false
1683 1470 Acked Read 7 Done Sent Done false
1684 1471 Acked Read 8 Done Sent Done false
1685 1472 Acked Flush 0 Done Sent Done false
1686 1473 Acked Read 2 Done Sent Done false
1687 1474 Acked Write 1 Done Sent Done false
1688 1475 Acked Flush 0 Done Sent Done false
1689 1476 Acked Read 8 Done Sent Done false
1690 1477 Acked Read 4 Done Sent Done false
1691 1478 Acked Read 1 Done Sent Done false
1692 1479 Acked Write 10 Done Sent Done false
1693 1480 Acked Read 10 Done Sent Done false
1694 1481 Acked Write 1 Done Sent Done false
1695 1482 Acked Read 3 Done Sent Done false
1696 1483 Acked Read 1 Done Sent Done false
1697 1484 Acked Write 2 Done Sent Done false
1698 1485 Acked Read 2 Done Sent Done false
1699 1486 Acked Read 4 Done Sent Done false
1700 1487 Acked Read 7 Done Sent Done false
1701 1488 Acked Read 5 Done Sent Done false
1702 1489 Acked Flush 0 Done Sent Done false
1703 1490 Acked Read 7 Done Sent Done false
1704 1491 Acked Write 10 Done Sent Done false
1705 1492 Acked Write 9 Done Sent Done false
1706 1493 Acked Read 7 Done Sent Done false
1707 1494 Acked Write 8 Done Sent Done false
1708 1495 Acked Read 2 Done Sent Done false
1709 1496 Acked Read 3 Done Sent Done false
1710 1497 Acked Write 6 Done Sent Done false
1711 1498 Acked Read 2 Done Sent Done false
1712 1499 Acked Write 10 Done Sent Done false
1713 1500 Acked Read 5 Done Sent Done false
1714 1501 Acked Write 4 Done Sent Done false
1715 1502 Acked Read 1 Done Sent Done false
1716 1503 Acked Flush 0 Done Sent Done false
1717 1504 Acked Write 4 Done Sent Done false
1718 1505 Acked Read 5 Done Sent Done false
1719 1506 Acked Write 9 Done Sent Done false
1720 1507 Acked Write 3 Done Sent Done false
1721 1508 Acked Flush 0 Done Sent Done false
1722 1509 Acked Write 10 Done Sent Done false
1723 1510 Acked Read 5 Done Sent Done false
1724 1511 Acked Write 3 Done Sent Done false
1725 1512 Acked Write 6 Done Sent Done false
1726 1513 Acked Read 8 Done Sent Done false
1727 1514 Acked Read 7 Done Sent Done false
1728 1515 Acked Write 2 Done Sent Done false
1729 1516 Acked Flush 0 Done Sent Done false
1730 1517 Acked Read 9 Done Sent Done false
1731 1518 Acked Read 3 Done Sent Done false
1732 1519 Acked Write 10 Done Sent Done false
1733 1520 Acked Write 3 Done Sent Done false
1734 1521 Acked Write 10 Done Sent Done false
1735 1522 Acked Write 7 Done Sent Done false
1736 1523 Acked Read 3 Done Sent Done false
1737 1524 Acked Read 5 Done Sent Done false
1738 1525 Acked Read 8 Done Sent Done false
1739 1526 Acked Write 1 Done Sent Done false
1740 1527 Acked Read 4 Done Sent Done false
1741 1528 Acked Read 9 Done Sent Done false
1742 1529 Acked Write 2 Done Sent Done false
1743 1530 Acked Read 5 Done Sent Done false
1744 1531 Acked Read 10 Done Sent Done false
1745 1532 Acked Read 9 Done Sent Done false
1746 1533 Acked Read 4 Done Sent Done false
1747 1534 Acked Write 5 Done Sent Done false
1748 1535 Acked Write 9 Done Sent Done false
1749 1536 Acked Flush 0 Done Sent Done false
1750 1537 Acked Flush 0 Done Sent Done false
1751 1538 Acked Write 6 Done Sent Done false
1752 1539 Acked Read 2 Done Sent Done false
1753 1540 Acked Read 8 Done Sent Done false
1754 1541 Acked Read 3 Done Sent Done false
1755 1542 Acked Read 5 Done Sent Done false
1756 1543 Acked Read 4 Done Sent Done false
1757 1544 Acked Read 8 Done Sent Done false
1758 1545 Acked Flush 0 Done Sent Done false
1759 1546 Acked Read 3 Done Sent Done false
1760 1547 Acked Write 7 Done Sent Done false
1761 1548 Acked Read 8 Done Sent Done false
1762 1549 Acked Read 8 Done Sent Done false
1763 1550 Acked Flush 0 Done Sent Done false
1764 1551 Acked Write 9 Done Sent Done false
1765 1552 Acked Read 6 Done Sent Done false
1766 1553 Acked Read 8 Done Sent Done false
1767 1554 Acked Read 4 Done Sent Done false
1768 1555 Acked Read 8 Done Sent Done false
1769 1556 Acked Write 8 Done Sent Done false
1770 1557 Acked Read 5 Done Sent Done false
1771 1558 Acked Read 8 Done Sent Done false
1772 1559 Acked Write 8 Done Sent Done false
1773 1560 Acked Flush 0 Done Sent Done false
1774 1561 Acked Read 4 Done Sent Done false
1775 1562 Acked Read 9 Done Sent Done false
1776 1563 Acked Flush 0 Done Sent Done false
1777 1564 Acked Flush 0 Done Sent Done false
1778 1565 Acked Read 6 Done Sent Done false
1779 1566 Acked Write 5 Done Sent Done false
1780 1567 Acked Write 7 Done Sent Done false
1781 1568 Acked Write 3 Done Sent Done false
1782 1569 Acked Write 3 Done Sent Done false
1783 1570 Acked Flush 0 Done Sent Done false
1784 1571 Acked Read 3 Done Sent Done false
1785 1572 Acked Read 7 Done Sent Done false
1786 1573 Acked Read 6 Done Sent Done false
1787 1574 Acked Flush 0 Done Sent Done false
1788 1575 Acked Write 1 Done Sent Done false
1789 1576 Acked Read 1 Done Sent Done false
1790 1577 Acked Write 5 Done Sent Done false
1791 1578 Acked Write 6 Done Sent Done false
1792 1579 Acked Read 3 Done Sent Done false
1793 1580 Acked Write 7 Done Sent Done false
1794 1581 Acked Write 9 Done Sent Done false
1795 1582 Acked Read 1 Done Sent Done false
1796 1583 Acked Read 6 Done Sent Done false
1797 1584 Acked Read 7 Done Sent Done false
1798 1585 Acked Write 10 Done Sent Done false
1799 1586 Acked Read 6 Done Sent Done false
1800 1587 Acked Read 6 Done Sent Done false
1801 1588 Acked Read 9 Done Sent Done false
1802 1589 Acked Write 1 Done Sent Done false
1803 1590 Acked Write 8 Done Sent Done false
1804 1591 Acked Write 4 Done Sent Done false
1805 1592 Acked Read 7 Done Sent Done false
1806 1593 Acked Read 2 Done Sent Done false
1807 1594 Acked Read 2 Done Sent Done false
1808 1595 Acked Read 9 Done Sent Done false
1809 1596 Acked Write 2 Done Sent Done false
1810 1597 Acked Write 6 Done Sent Done false
1811 1598 Acked Read 3 Done Sent Done false
1812 1599 Acked Write 7 Done Sent Done false
1813 1600 Acked Read 10 Done Sent Done false
1814 1601 Acked Read 1 Done Sent Done false
1815 STATES DS:0 DS:1 DS:2 TOTAL
1816 Sent 0 301 0 301
1817 Done 301 0 301 602
1818 Skipped 0 0 0 0
1819 Error 0 0 0 0
1820 Last Flush: Some(JobId(1574)) Some(JobId(1300)) Some(JobId(1574))
1821 Downstairs last five completed: 1300 1299 1298 1297 1296
1822 Upstairs last five completed: 1601 1600 1599 1598 1597
1823 CLIENT: Up:0 ds:301 act:2
18242025-02-25T20:47:02.098ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
18252025-02-25T20:47:03.098ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
18262025-02-25T20:47:04.099ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
1827 [8820] got start action
1828 [8820] I am going to restart
1829 Starting downstairs at port 8820
1830 Make output file at "/tmp/dsc/downstairs-8820.txt"
1831 [8820][1] reports Starting
1832 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1208)
1833 [8820][1] reports Running
1834 ----------------------------------------------------------------
1835 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1836 States: Active Offline Active
1837 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1838 1301 Acked Read 10 Done Sent Done false
1839 1302 Acked Flush 0 Done Sent Done false
1840 1303 Acked Flush 0 Done Sent Done false
1841 1304 Acked Read 7 Done Sent Done false
1842 1305 Acked Flush 0 Done Sent Done false
1843 1306 Acked Write 7 Done Sent Done false
1844 1307 Acked Write 2 Done Sent Done false
1845 1308 Acked Read 1 Done Sent Done false
1846 1309 Acked Read 3 Done Sent Done false
1847 1310 Acked Read 3 Done Sent Done false
1848 1311 Acked Write 9 Done Sent Done false
1849 1312 Acked Read 9 Done Sent Done false
1850 1313 Acked Read 1 Done Sent Done false
1851 1314 Acked Read 9 Done Sent Done false
1852 1315 Acked Read 6 Done Sent Done false
1853 1316 Acked Write 9 Done Sent Done false
1854 1317 Acked Write 10 Done Sent Done false
1855 1318 Acked Write 1 Done Sent Done false
1856 1319 Acked Read 5 Done Sent Done false
1857 1320 Acked Write 10 Done Sent Done false
1858 1321 Acked Write 4 Done Sent Done false
1859 1322 Acked Write 9 Done Sent Done false
1860 1323 Acked Write 4 Done Sent Done false
1861 1324 Acked Write 6 Done Sent Done false
1862 1325 Acked Read 6 Done Sent Done false
1863 1326 Acked Flush 0 Done Sent Done false
1864 1327 Acked Flush 0 Done Sent Done false
1865 1328 Acked Read 6 Done Sent Done false
1866 1329 Acked Read 8 Done Sent Done false
1867 1330 Acked Write 5 Done Sent Done false
1868 1331 Acked Write 1 Done Sent Done false
1869 1332 Acked Read 9 Done Sent Done false
1870 1333 Acked Read 5 Done Sent Done false
1871 1334 Acked Read 8 Done Sent Done false
1872 1335 Acked Read 9 Done Sent Done false
1873 1336 Acked Read 6 Done Sent Done false
1874 1337 Acked Read 10 Done Sent Done false
1875 1338 Acked Read 3 Done Sent Done false
1876 1339 Acked Write 1 Done Sent Done false
1877 1340 Acked Read 5 Done Sent Done false
1878 1341 Acked Read 6 Done Sent Done false
1879 1342 Acked Read 7 Done Sent Done false
1880 1343 Acked Write 2 Done Sent Done false
1881 1344 Acked Read 4 Done Sent Done false
1882 1345 Acked Write 8 Done Sent Done false
1883 1346 Acked Read 1 Done Sent Done false
1884 1347 Acked Read 9 Done Sent Done false
1885 1348 Acked Read 1 Done Sent Done false
1886 1349 Acked Read 5 Done Sent Done false
1887 1350 Acked Write 7 Done Sent Done false
1888 1351 Acked Write 10 Done Sent Done false
1889 1352 Acked Flush 0 Done Sent Done false
1890 1353 Acked Read 9 Done Sent Done false
1891 1354 Acked Read 8 Done Sent Done false
1892 1355 Acked Read 2 Done Sent Done false
1893 1356 Acked Read 6 Done Sent Done false
1894 1357 Acked Read 3 Done Sent Done false
1895 1358 Acked Flush 0 Done Sent Done false
1896 1359 Acked Read 2 Done Sent Done false
1897 1360 Acked Read 3 Done Sent Done false
1898 1361 Acked Read 7 Done Sent Done false
1899 1362 Acked Write 9 Done Sent Done false
1900 1363 Acked Read 7 Done Sent Done false
1901 1364 Acked Write 1 Done Sent Done false
1902 1365 Acked Flush 0 Done Sent Done false
1903 1366 Acked Read 10 Done Sent Done false
1904 1367 Acked Flush 0 Done Sent Done false
1905 1368 Acked Read 8 Done Sent Done false
1906 1369 Acked Write 5 Done Sent Done false
1907 1370 Acked Write 10 Done Sent Done false
1908 1371 Acked Read 2 Done Sent Done false
1909 1372 Acked Write 1 Done Sent Done false
1910 1373 Acked Read 5 Done Sent Done false
1911 1374 Acked Read 5 Done Sent Done false
1912 1375 Acked Write 1 Done Sent Done false
1913 1376 Acked Read 7 Done Sent Done false
1914 1377 Acked Read 6 Done Sent Done false
1915 1378 Acked Read 8 Done Sent Done false
1916 1379 Acked Read 10 Done Sent Done false
1917 1380 Acked Read 1 Done Sent Done false
1918 1381 Acked Read 2 Done Sent Done false
1919 1382 Acked Read 1 Done Sent Done false
1920 1383 Acked Read 5 Done Sent Done false
1921 1384 Acked Write 5 Done Sent Done false
1922 1385 Acked Write 6 Done Sent Done false
1923 1386 Acked Flush 0 Done Sent Done false
1924 1387 Acked Read 1 Done Sent Done false
1925 1388 Acked Write 6 Done Sent Done false
1926 1389 Acked Read 10 Done Sent Done false
1927 1390 Acked Read 1 Done Sent Done false
1928 1391 Acked Write 1 Done Sent Done false
1929 1392 Acked Write 1 Done Sent Done false
1930 1393 Acked Read 4 Done Sent Done false
1931 1394 Acked Read 3 Done Sent Done false
1932 1395 Acked Write 1 Done Sent Done false
1933 1396 Acked Write 1 Done Sent Done false
1934 1397 Acked Read 8 Done Sent Done false
1935 1398 Acked Read 7 Done Sent Done false
1936 1399 Acked Write 10 Done Sent Done false
1937 1400 Acked Write 3 Done Sent Done false
1938 1401 Acked Read 6 Done Sent Done false
1939 1402 Acked Write 3 Done Sent Done false
1940 1403 Acked Read 7 Done Sent Done false
1941 1404 Acked Read 6 Done Sent Done false
1942 1405 Acked Write 1 Done Sent Done false
1943 1406 Acked Read 4 Done Sent Done false
1944 1407 Acked Write 6 Done Sent Done false
1945 1408 Acked Write 7 Done Sent Done false
1946 1409 Acked Write 3 Done Sent Done false
1947 1410 Acked Write 4 Done Sent Done false
1948 1411 Acked Read 4 Done Sent Done false
1949 1412 Acked Write 9 Done Sent Done false
1950 1413 Acked Read 10 Done Sent Done false
1951 1414 Acked Write 8 Done Sent Done false
1952 1415 Acked Write 3 Done Sent Done false
1953 1416 Acked Write 2 Done Sent Done false
1954 1417 Acked Flush 0 Done Sent Done false
1955 1418 Acked Write 9 Done Sent Done false
1956 1419 Acked Write 3 Done Sent Done false
1957 1420 Acked Read 3 Done Sent Done false
1958 1421 Acked Write 10 Done Sent Done false
1959 1422 Acked Read 10 Done Sent Done false
1960 1423 Acked Flush 0 Done Sent Done false
1961 1424 Acked Read 10 Done Sent Done false
1962 1425 Acked Read 3 Done Sent Done false
1963 1426 Acked Read 1 Done Sent Done false
1964 1427 Acked Read 10 Done Sent Done false
1965 1428 Acked Write 2 Done Sent Done false
1966 1429 Acked Write 1 Done Sent Done false
1967 1430 Acked Read 9 Done Sent Done false
1968 1431 Acked Read 7 Done Sent Done false
1969 1432 Acked Write 5 Done Sent Done false
1970 1433 Acked Read 6 Done Sent Done false
1971 1434 Acked Write 7 Done Sent Done false
1972 1435 Acked Write 6 Done Sent Done false
1973 1436 Acked Read 8 Done Sent Done false
1974 1437 Acked Flush 0 Done Sent Done false
1975 1438 Acked Write 5 Done Sent Done false
1976 1439 Acked Write 3 Done Sent Done false
1977 1440 Acked Flush 0 Done Sent Done false
1978 1441 Acked Write 10 Done Sent Done false
1979 1442 Acked Read 9 Done Sent Done false
1980 1443 Acked Read 2 Done Sent Done false
1981 1444 Acked Write 1 Done Sent Done false
1982 1445 Acked Read 9 Done Sent Done false
1983 1446 Acked Read 8 Done Sent Done false
1984 1447 Acked Write 3 Done Sent Done false
1985 1448 Acked Write 3 Done Sent Done false
1986 1449 Acked Write 2 Done Sent Done false
1987 1450 Acked Read 7 Done Sent Done false
1988 1451 Acked Read 8 Done Sent Done false
1989 1452 Acked Read 7 Done Sent Done false
1990 1453 Acked Write 1 Done Sent Done false
1991 1454 Acked Write 10 Done Sent Done false
1992 1455 Acked Write 2 Done Sent Done false
1993 1456 Acked Write 1 Done Sent Done false
1994 1457 Acked Write 5 Done Sent Done false
1995 1458 Acked Read 3 Done Sent Done false
1996 1459 Acked Read 9 Done Sent Done false
1997 1460 Acked Write 1 Done Sent Done false
1998 1461 Acked Write 4 Done Sent Done false
1999 1462 Acked Write 6 Done Sent Done false
2000 1463 Acked Read 9 Done Sent Done false
2001 1464 Acked Write 1 Done Sent Done false
2002 1465 Acked Read 10 Done Sent Done false
2003 1466 Acked Write 1 Done Sent Done false
2004 1467 Acked Read 1 Done Sent Done false
2005 1468 Acked Write 5 Done Sent Done false
2006 1469 Acked Read 5 Done Sent Done false
2007 1470 Acked Read 7 Done Sent Done false
2008 1471 Acked Read 8 Done Sent Done false
2009 1472 Acked Flush 0 Done Sent Done false
2010 1473 Acked Read 2 Done Sent Done false
2011 1474 Acked Write 1 Done Sent Done false
2012 1475 Acked Flush 0 Done Sent Done false
2013 1476 Acked Read 8 Done Sent Done false
2014 1477 Acked Read 4 Done Sent Done false
2015 1478 Acked Read 1 Done Sent Done false
2016 1479 Acked Write 10 Done Sent Done false
2017 1480 Acked Read 10 Done Sent Done false
2018 1481 Acked Write 1 Done Sent Done false
2019 1482 Acked Read 3 Done Sent Done false
2020 1483 Acked Read 1 Done Sent Done false
2021 1484 Acked Write 2 Done Sent Done false
2022 1485 Acked Read 2 Done Sent Done false
2023 1486 Acked Read 4 Done Sent Done false
2024 1487 Acked Read 7 Done Sent Done false
2025 1488 Acked Read 5 Done Sent Done false
2026 1489 Acked Flush 0 Done Sent Done false
2027 1490 Acked Read 7 Done Sent Done false
2028 1491 Acked Write 10 Done Sent Done false
2029 1492 Acked Write 9 Done Sent Done false
2030 1493 Acked Read 7 Done Sent Done false
2031 1494 Acked Write 8 Done Sent Done false
2032 1495 Acked Read 2 Done Sent Done false
2033 1496 Acked Read 3 Done Sent Done false
2034 1497 Acked Write 6 Done Sent Done false
2035 1498 Acked Read 2 Done Sent Done false
2036 1499 Acked Write 10 Done Sent Done false
2037 1500 Acked Read 5 Done Sent Done false
2038 1501 Acked Write 4 Done Sent Done false
2039 1502 Acked Read 1 Done Sent Done false
2040 1503 Acked Flush 0 Done Sent Done false
2041 1504 Acked Write 4 Done Sent Done false
2042 1505 Acked Read 5 Done Sent Done false
2043 1506 Acked Write 9 Done Sent Done false
2044 1507 Acked Write 3 Done Sent Done false
2045 1508 Acked Flush 0 Done Sent Done false
2046 1509 Acked Write 10 Done Sent Done false
2047 1510 Acked Read 5 Done Sent Done false
2048 1511 Acked Write 3 Done Sent Done false
2049 1512 Acked Write 6 Done Sent Done false
2050 1513 Acked Read 8 Done Sent Done false
2051 1514 Acked Read 7 Done Sent Done false
2052 1515 Acked Write 2 Done Sent Done false
2053 1516 Acked Flush 0 Done Sent Done false
2054 1517 Acked Read 9 Done Sent Done false
2055 1518 Acked Read 3 Done Sent Done false
2056 1519 Acked Write 10 Done Sent Done false
2057 1520 Acked Write 3 Done Sent Done false
2058 1521 Acked Write 10 Done Sent Done false
2059 1522 Acked Write 7 Done Sent Done false
2060 1523 Acked Read 3 Done Sent Done false
2061 1524 Acked Read 5 Done Sent Done false
2062 1525 Acked Read 8 Done Sent Done false
2063 1526 Acked Write 1 Done Sent Done false
2064 1527 Acked Read 4 Done Sent Done false
2065 1528 Acked Read 9 Done Sent Done false
2066 1529 Acked Write 2 Done Sent Done false
2067 1530 Acked Read 5 Done Sent Done false
2068 1531 Acked Read 10 Done Sent Done false
2069 1532 Acked Read 9 Done Sent Done false
2070 1533 Acked Read 4 Done Sent Done false
2071 1534 Acked Write 5 Done Sent Done false
2072 1535 Acked Write 9 Done Sent Done false
2073 1536 Acked Flush 0 Done Sent Done false
2074 1537 Acked Flush 0 Done Sent Done false
2075 1538 Acked Write 6 Done Sent Done false
2076 1539 Acked Read 2 Done Sent Done false
2077 1540 Acked Read 8 Done Sent Done false
2078 1541 Acked Read 3 Done Sent Done false
2079 1542 Acked Read 5 Done Sent Done false
2080 1543 Acked Read 4 Done Sent Done false
2081 1544 Acked Read 8 Done Sent Done false
2082 1545 Acked Flush 0 Done Sent Done false
2083 1546 Acked Read 3 Done Sent Done false
2084 1547 Acked Write 7 Done Sent Done false
2085 1548 Acked Read 8 Done Sent Done false
2086 1549 Acked Read 8 Done Sent Done false
2087 1550 Acked Flush 0 Done Sent Done false
2088 1551 Acked Write 9 Done Sent Done false
2089 1552 Acked Read 6 Done Sent Done false
2090 1553 Acked Read 8 Done Sent Done false
2091 1554 Acked Read 4 Done Sent Done false
2092 1555 Acked Read 8 Done Sent Done false
2093 1556 Acked Write 8 Done Sent Done false
2094 1557 Acked Read 5 Done Sent Done false
2095 1558 Acked Read 8 Done Sent Done false
2096 1559 Acked Write 8 Done Sent Done false
2097 1560 Acked Flush 0 Done Sent Done false
2098 1561 Acked Read 4 Done Sent Done false
2099 1562 Acked Read 9 Done Sent Done false
2100 1563 Acked Flush 0 Done Sent Done false
2101 1564 Acked Flush 0 Done Sent Done false
2102 1565 Acked Read 6 Done Sent Done false
2103 1566 Acked Write 5 Done Sent Done false
2104 1567 Acked Write 7 Done Sent Done false
2105 1568 Acked Write 3 Done Sent Done false
2106 1569 Acked Write 3 Done Sent Done false
2107 1570 Acked Flush 0 Done Sent Done false
2108 1571 Acked Read 3 Done Sent Done false
2109 1572 Acked Read 7 Done Sent Done false
2110 1573 Acked Read 6 Done Sent Done false
2111 1574 Acked Flush 0 Done Sent Done false
2112 1575 Acked Write 1 Done Sent Done false
2113 1576 Acked Read 1 Done Sent Done false
2114 1577 Acked Write 5 Done Sent Done false
2115 1578 Acked Write 6 Done Sent Done false
2116 1579 Acked Read 3 Done Sent Done false
2117 1580 Acked Write 7 Done Sent Done false
2118 1581 Acked Write 9 Done Sent Done false
2119 1582 Acked Read 1 Done Sent Done false
2120 1583 Acked Read 6 Done Sent Done false
2121 1584 Acked Read 7 Done Sent Done false
2122 1585 Acked Write 10 Done Sent Done false
2123 1586 Acked Read 6 Done Sent Done false
2124 1587 Acked Read 6 Done Sent Done false
2125 1588 Acked Read 9 Done Sent Done false
2126 1589 Acked Write 1 Done Sent Done false
2127 1590 Acked Write 8 Done Sent Done false
2128 1591 Acked Write 4 Done Sent Done false
2129 1592 Acked Read 7 Done Sent Done false
2130 1593 Acked Read 2 Done Sent Done false
2131 1594 Acked Read 2 Done Sent Done false
2132 1595 Acked Read 9 Done Sent Done false
2133 1596 Acked Write 2 Done Sent Done false
2134 1597 Acked Write 6 Done Sent Done false
2135 1598 Acked Read 3 Done Sent Done false
2136 1599 Acked Write 7 Done Sent Done false
2137 1600 Acked Read 10 Done Sent Done false
2138 1601 Acked Read 1 Done Sent Done false
2139 STATES DS:0 DS:1 DS:2 TOTAL
2140 Sent 0 301 0 301
2141 Done 301 0 301 602
2142 Skipped 0 0 0 0
2143 Error 0 0 0 0
2144 Last Flush: Some(JobId(1574)) Some(JobId(1300)) Some(JobId(1574))
2145 Downstairs last five completed: 1300 1299 1298 1297 1296
2146 Upstairs last five completed: 1601 1600 1599 1598 1597
2147 CLIENT: Up:0 ds:301 act:2
21482025-02-25T20:47:06.102ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21492025-02-25T20:47:06.850ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21502025-02-25T20:47:06.851ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59574) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21512025-02-25T20:47:06.852ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21522025-02-25T20:47:06.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21532025-02-25T20:47:06.852ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21542025-02-25T20:47:06.852ZINFOcrucible: send last flush ID to this DS: Some(JobId(1300)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21552025-02-25T20:47:06.852ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21562025-02-25T20:47:06.852ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1300)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21572025-02-25T20:47:06.853ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1300)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2158 ----------------------------------------------------------------
2159 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2160 Downstairs last five completed: 1602 1601 1600 1599 1598
2161 Upstairs last five completed: 1602 1601 1600 1599 1598
2162 CLIENT: Up:0 ds:0 act:3
2163 Replay: All jobs finished
2164 Main task has work to do, go find it
2165 got dsc Stop(2)
2166 stop 2
2167 Feb 25 20:47:09.105 INFO request completed, latency_us: 397, response_code[8830] Got stop action so:false kr:false
2168 : 204, uri: /stop/cid/2, method: POST, req_id: 8c7ec8f7-6e39-4ecf-901c-f39e69cb359f, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
2169 Feb 25 20:47:09.106 INFO request completed, latency_us: 171, response_code: 200, uri: /state/cid/2, method{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:47:09.106068296Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
2170 : GET, req_id: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:47:09.106144331Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
2171 7509ea46-22c9-48ee-a3e3-d08d770cae3d, remote_addr: 127.0.0.1:43930, local_addr: 127{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:47:09.106174159Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
2172 .0.0.1:9998
21732025-02-25T20:47:09.106ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2174 [8830] Exited with: ExitStatus(unix_wait_status(9))
2175 [8830][2] reports Exit
2176 Feb 25 20:47:13.108 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2edd174c-7c16-4bc2-af48-8302db9b01dc, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
2177 001/300 Read block 4672 len 40960
2178 002/300 Read block 805 len 20480
2179 003/300 Read block 2836 len 4096
2180 004/300 Read block 4454 len 32768
2181 005/300 Read block 279 len 16384
2182 006/300 Write block 1555 len 4096 data: 2
2183 007/300 Read block 1266 len 36864
2184 008/300 Read block 293 len 28672
2185 009/300 Read block 3697 len 4096
2186 010/300 Read block 215 len 40960
2187 011/300 Read block 1881 len 24576
2188 012/300 Read block 4443 len 16384
2189 013/300 Write block 4695 len 32768 data: 3 3 3 3 2 2 2 2
2190 014/300 Read block 3204 len 16384
2191 015/300 Read block 724 len 16384
2192 016/300 Read block 2947 len 20480
2193 017/300 Read block 3317 len 24576
2194 018/300 Write block 4434 len 8192 data: 2 2
2195 019/300 Read block 429 len 40960
2196 020/300 Write block 4092 len 12288 data: 2 2 2
2197 021/300 Write block 2522 len 20480 data: 3 3 2 2 2
2198 022/300 Read block 2558 len 20480
2199 023/300 Write block 1909 len 28672 data: 2 3 3 3 3 3 3
2200 024/300 Read block 915 len 20480
2201 025/300 Read block 4975 len 8192
2202 026/300 Write block 3854 len 16384 data: 2 2 2 2
2203 027/300 Read block 2396 len 16384
2204 028/300 Read block 4736 len 4096
2205 029/300 Write block 1401 len 12288 data: 2 2 2
2206 030/300 Flush
2207 031/300 Read block 783 len 40960
2208 032/300 Read block 1506 len 20480
2209 033/300 Read block 1093 len 16384
2210 034/300 Read block 1364 len 24576
2211 035/300 Read block 2930 len 36864
2212 036/300 Flush
2213 037/300 Read block 3649 len 24576
2214 038/300 Read block 4246 len 8192
2215 039/300 Read block 1990 len 32768
2216 040/300 Write block 422 len 24576 data: 2 2 2 2 2 2
2217 041/300 Write block 4142 len 16384 data: 2 2 2 2
2218 042/300 Write block 4989 len 12288 data: 2 2 2
2219 043/300 Read block 4256 len 28672
2220 044/300 Read block 1204 len 32768
2221 045/300 Write block 6 len 24576 data: 2 2 2 2 2 2
2222 046/300 Write block 675 len 20480 data: 3 3 3 3 3
2223 047/300 Write block 3381 len 28672 data: 2 2 2 2 2 2 2
2224 048/300 Write block 579 len 28672 data: 3 3 2 2 2 2 2
2225 049/300 Read block 785 len 28672
2226 050/300 Write block 39 len 12288 data: 2 2 2
2227 051/300 Read block 339 len 20480
2228 052/300 Write block 350 len 32768 data: 2 2 2 2 2 2 2 2
2229 053/300 Read block 1898 len 4096
2230 054/300 Write block 483 len 28672 data: 2 2 2 2 2 2 2
2231 055/300 Read block 4747 len 12288
2232 056/300 Read block 114 len 24576
2233 057/300 Flush
2234 058/300 Write block 1583 len 24576 data: 3 3 2 2 2 2
2235 059/300 Read block 1624 len 8192
2236 060/300 Read block 1934 len 32768
2237 061/300 Read block 1634 len 24576
2238 062/300 Write block 1186 len 32768 data: 2 2 2 2 3 3 3 3
2239 063/300 Read block 1060 len 24576
2240 064/300 Flush
2241 065/300 Write block 576 len 40960 data: 3 3 3 4 4 3 3 3 3 3
2242 066/300 Write block 4916 len 24576 data: 2 2 2 2 2 2
2243 067/300 Read block 2937 len 4096
2244 068/300 Write block 3945 len 36864 data: 2 3 2 2 2 3 3 3 3
2245 069/300 Read block 4075 len 32768
2246 070/300 Read block 4935 len 8192
2247 071/300 Read block 847 len 40960
2248 072/300 Read block 662 len 12288
2249 073/300 Write block 3772 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2250 074/300 Read block 4042 len 12288
2251 075/300 Read block 1228 len 8192
2252 076/300 Read block 3723 len 28672
2253 077/300 Read block 1122 len 24576
2254 078/300 Read block 2561 len 12288
2255 079/300 Write block 3380 len 28672 data: 2 3 3 3 3 3 3
2256 080/300 Write block 2867 len 20480 data: 2 2 2 2 2
2257 081/300 Write block 2439 len 28672 data: 2 2 2 2 2 2 2
2258 082/300 Read block 4354 len 28672
2259 083/300 Read block 2971 len 16384
2260 084/300 Flush
2261 085/300 Write block 3810 len 16384 data: 2 2 2 2
2262 086/300 Flush
2263 087/300 Read block 1331 len 32768
2264 088/300 Write block 3745 len 4096 data: 2
2265 089/300 Write block 1217 len 28672 data: 2 3 2 2 2 2 2
2266 090/300 Read block 217 len 8192
2267 091/300 Write block 1112 len 4096 data: 2
2268 092/300 Write block 2426 len 28672 data: 3 3 3 3 3 3 2
2269 093/300 Read block 2223 len 20480
2270 094/300 Read block 1254 len 20480
2271 095/300 Read block 3249 len 16384
2272 096/300 Read block 860 len 4096
2273 097/300 Read block 3808 len 24576
2274 098/300 Flush
2275 099/300 Read block 1537 len 12288
2276 100/300 Write block 3417 len 28672 data: 2 2 2 2 2 2 2
2277 101/300 Write block 3270 len 4096 data: 2
2278 102/300 Read block 838 len 8192
2279 103/300 Write block 1719 len 28672 data: 2 2 2 2 2 2 2
2280 104/300 Write block 4813 len 12288 data: 2 2 2
2281 105/300 Write block 329 len 32768 data: 3 3 3 3 2 2 2 2
2282 106/300 Flush
2283 107/300 Read block 3878 len 20480
2284 108/300 Write block 2642 len 32768 data: 2 2 2 2 2 2 2 2
2285 109/300 Read block 1697 len 40960
2286 110/300 Write block 1570 len 20480 data: 2 2 2 2 2
2287 111/300 Flush
2288 112/300 Write block 4883 len 4096 data: 2
2289 113/300 Read block 3816 len 40960
2290 114/300 Write block 269 len 40960 data: 5 4 4 4 4 4 3 3 3 2
2291 115/300 Write block 2950 len 4096 data: 2
2292 116/300 Read block 4418 len 32768
2293 117/300 Flush
2294 118/300 Flush
2295 119/300 Read block 1893 len 28672
2296 120/300 Write block 4795 len 20480 data: 2 2 2 2 2
2297 121/300 Write block 690 len 24576 data: 2 2 2 2 2 2
2298 122/300 Flush
2299 123/300 Read block 3094 len 16384
2300 124/300 Write block 1535 len 32768 data: 4 3 2 2 2 2 2 2
2301 125/300 Write block 1217 len 20480 data: 3 4 3 3 3
2302 126/300 Read block 1100 len 36864
2303 127/300 Read block 782 len 28672
2304 128/300 Write block 3527 len 4096 data: 2
2305 129/300 Read block 1494 len 20480
2306 130/300 Read block 2908 len 8192
2307 131/300 Write block 2363 len 32768 data: 2 2 2 2 2 2 2 2
2308 132/300 Write block 1243 len 28672 data: 2 2 2 2 2 2 2
2309 133/300 Read block 671 len 16384
2310 134/300 Read block 4164 len 32768
2311 135/300 Read block 4197 len 28672
2312 136/300 Write block 4149 len 32768 data: 2 2 2 2 2 2 2 2
2313 137/300 Read block 4612 len 24576
2314 138/300 Read block 371 len 12288
2315 139/300 Read block 4814 len 32768
2316 140/300 Write block 3461 len 4096 data: 2
2317 141/300 Read block 3735 len 40960
2318 142/300 Read block 1207 len 40960
2319 143/300 Write block 2424 len 36864 data: 2 2 4 4 4 4 4 4 3
2320 144/300 Write block 4874 len 4096 data: 2
2321 145/300 Write block 1792 len 32768 data: 2 2 2 2 2 2 2 2
2322 146/300 Write block 1874 len 28672 data: 3 3 3 3 3 3 3
2323 147/300 Flush
2324 148/300 Write block 233 len 28672 data: 2 2 2 2 2 2 2
2325 149/300 Read block 2822 len 8192
2326 150/300 Write block 817 len 24576 data: 2 2 2 2 2 2
2327 151/300 Write block 1678 len 20480 data: 2 2 2 2 2
2328 152/300 Write block 1546 len 4096 data: 2
2329 153/300 Write block 3490 len 32768 data: 3 3 3 3 2 2 2 2
2330 154/300 Flush
2331 155/300 Write block 45 len 16384 data: 2 2 2 2
2332 156/300 Read block 475 len 32768
2333 157/300 Write block 2608 len 4096 data: 2
2334 158/300 Flush
2335 159/300 Read block 3135 len 8192
2336 160/300 Read block 4492 len 40960
2337 161/300 Write block 1621 len 12288 data: 2 2 2
2338 162/300 Write block 2530 len 12288 data: 2 2 2
2339 163/300 Write block 3541 len 12288 data: 2 2 2
2340 164/300 Write block 2959 len 36864 data: 2 2 2 2 2 2 2 2 2
2341 165/300 Read block 4670 len 40960
2342 166/300 Write block 4089 len 8192 data: 2 2
2343 167/300 Write block 4617 len 4096 data: 2
2344 168/300 Read block 403 len 4096
2345 169/300 Read block 3524 len 16384
2346 170/300 Write block 3476 len 12288 data: 2 2 2
2347 171/300 Flush
2348 172/300 Read block 1519 len 12288
2349 173/300 Write block 983 len 12288 data: 2 2 2
2350 174/300 Write block 3982 len 20480 data: 2 2 2 2 2
2351 175/300 Read block 2858 len 32768
2352 176/300 Write block 2906 len 12288 data: 2 2 2
2353 177/300 Read block 2078 len 40960
2354 178/300 Write block 4964 len 20480 data: 2 2 2 2 2
2355 179/300 Write block 3560 len 8192 data: 2 3
2356 180/300 Write block 1012 len 32768 data: 2 2 2 2 2 2 2 2
2357 181/300 Flush
2358 182/300 Read block 3536 len 8192
2359 183/300 Write block 729 len 12288 data: 4 3 3
2360 184/300 Write block 1954 len 24576 data: 3 2 2 2 2 2
2361 185/300 Write block 3611 len 36864 data: 2 2 2 2 2 2 2 2 3
2362 186/300 Write block 1669 len 8192 data: 2 2
2363 187/300 Flush
2364 188/300 Read block 4499 len 16384
2365 189/300 Write block 3374 len 40960 data: 2 2 2 2 2 2 3 4 4 4
2366 190/300 Flush
2367 191/300 Write block 1478 len 16384 data: 3 3 3 3
2368 192/300 Read block 3015 len 4096
2369 193/300 Flush
2370 194/300 Write block 2687 len 36864 data: 2 4 3 3 2 2 2 2 2
2371 195/300 Read block 224 len 4096
2372 196/300 Flush
2373 197/300 Read block 4298 len 36864
2374 198/300 Write block 1892 len 8192 data: 2 2
2375 199/300 Read block 1484 len 8192
2376 200/300 Read block 1967 len 36864
2377 201/300 Read block 12 len 40960
2378 202/300 Read block 3598 len 12288
2379 203/300 Write block 3452 len 24576 data: 2 2 2 2 2 2
2380 204/300 Read block 4405 len 8192
2381 205/300 Write block 4881 len 16384 data: 2 2 3 2
2382 206/300 Write block 2235 len 12288 data: 2 2 2
2383 207/300 Read block 4984 len 40960
2384 208/300 Flush
2385 209/300 Write block 2377 len 28672 data: 2 2 2 2 2 2 2
2386 210/300 Read block 4519 len 16384
2387 211/300 Flush
2388 212/300 Flush
2389 213/300 Read block 2903 len 4096
2390 214/300 Read block 2915 len 16384
2391 215/300 Write block 450 len 24576 data: 2 2 2 2 2 2
2392 216/300 Read block 3496 len 32768
2393 217/300 Read block 4188 len 28672
2394 218/300 Write block 2056 len 20480 data: 2 2 2 2 2
2395 219/300 Read block 1217 len 4096
2396 220/300 Write block 4440 len 32768 data: 2 2 2 2 2 2 2 2
2397 221/300 Write block 1849 len 20480 data: 3 3 3 2 2
2398 222/300 Flush
2399 223/300 Write block 3082 len 32768 data: 2 2 2 2 2 2 2 2
2400 224/300 Write block 4794 len 28672 data: 2 3 3 3 3 3 2
2401 225/300 Read block 4465 len 24576
2402 226/300 Read block 149 len 40960
2403 227/300 Read block 3796 len 16384
2404 228/300 Write block 660 len 36864 data: 2 2 2 2 2 2 2 2 2
2405 229/300 Write block 3746 len 8192 data: 2 2
2406 230/300 Read block 4969 len 24576
2407 231/300 Read block 4692 len 32768
2408 232/300 Read block 2689 len 4096
2409 233/300 Write block 2392 len 24576 data: 2 2 2 2 2 2
2410 234/300 Write block 4973 len 16384 data: 2 2 2 2
2411 235/300 Read block 284 len 20480
2412 236/300 Read block 323 len 36864
2413 237/300 Flush
2414 238/300 Write block 1671 len 40960 data: 2 2 2 2 2 3 3 3 3 3
2415 239/300 Flush
2416 240/300 Write block 968 len 32768 data: 2 2 2 2 2 2 2 3
2417 241/300 Read block 1786 len 12288
2418 242/300 Flush
2419 243/300 Flush
2420 244/300 Read block 2481 len 36864
2421 245/300 Write block 2680 len 20480 data: 3 3 3 3 3
2422 246/300 Read block 2033 len 12288
2423 247/300 Flush
2424 248/300 Read block 2787 len 4096
2425 249/300 Write block 3054 len 28672 data: 2 2 2 2 2 2 2
2426 250/300 Write block 3707 len 24576 data: 3 3 3 3 3 3
2427 251/300 Write block 2341 len 20480 data: 3 3 3 2 2
2428 252/300 Write block 4398 len 36864 data: 2 2 2 2 2 2 2 2 2
2429 253/300 Write block 2207 len 12288 data: 3 4 3
2430 254/300 Write block 39 len 12288 data: 3 3 3
2431 255/300 Read block 3703 len 24576
2432 256/300 Read block 2584 len 12288
2433 257/300 Read block 4720 len 12288
2434 258/300 Read block 2600 len 40960
2435 259/300 Flush
2436 260/300 Read block 4238 len 32768
2437 261/300 Flush
2438 262/300 Read block 1859 len 24576
2439 263/300 Write block 1889 len 12288 data: 2 2 2
2440 264/300 Read block 3355 len 4096
2441 265/300 Read block 495 len 40960
2442 266/300 Write block 1812 len 32768 data: 2 2 2 2 2 2 2 2
2443 267/300 Write block 2255 len 28672 data: 3 3 2 2 2 2 2
2444 268/300 Write block 3064 len 20480 data: 2 2 2 2 2
2445 269/300 Read block 1333 len 12288
2446 270/300 Write block 3165 len 8192 data: 2 2
2447 271/300 Read block 1108 len 4096
2448 272/300 Read block 1601 len 20480
2449 273/300 Read block 1250 len 28672
2450 274/300 Read block 1516 len 16384
2451 275/300 Read block 1325 len 4096
2452 276/300 Write block 984 len 24576 data: 3 3 3 3 3 3
2453 277/300 Read block 4045 len 40960
2454 278/300 Read block 2159 len 20480
2455 279/300 Write block 2827 len 12288 data: 3 3 3
2456 280/300 Read block 1051 len 40960
2457 281/300 Read block 400 len 32768
2458 282/300 Read block 1364 len 8192
2459 283/300 Read block 259 len 8192
2460 284/300 Write block 218 len 28672 data: 3 3 3 3 3 2 2
2461 285/300 Read block 1 len 12288
2462 286/300 Write block 4720 len 36864 data: 2 2 2 2 2 2 2 2 2
2463 287/300 Read block 4487 len 8192
2464 288/300 Read block 1839 len 24576
2465 289/300 Write block 2401 len 28672 data: 3 3 2 2 2 2 2
2466 290/300 Write block 4984 len 40960 data: 2 2 2 2 2 3 3 3 2 2
2467 291/300 Flush
2468 292/300 Flush
2469 293/300 Write block 4665 len 24576 data: 2 2 2 2 2 2
2470 294/300 Read block 3097 len 28672
2471 295/300 Read block 4544 len 32768
2472 296/300 Read block 4082 len 8192
2473 297/300 Read block 884 len 12288
2474 298/300 Write block 4165 len 36864 data: 3 3 3 3 2 2 2 2 2
2475 299/300 Read block 1231 len 20480
2476 300/300 Read block 1749 len 32768
2477 301/300 Write block 2032 len 4096 data: 2
2478 Main task has work to do, go find it
2479 got dsc Start(2)
2480 start 2
2481 Feb 25 20:47:13.350 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 74d1f205-bf95-43c6-a5d2-6eb03aafd3ed, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
2482 Replay: started 2, returned:Ok(())
2483 ----------------------------------------------------------------
2484 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2485 States: Active Active Offline
2486 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2487 1603 Acked Read 10 Done Done Sent false
2488 1604 Acked Read 5 Done Done Sent false
2489 1605 Acked Read 1 Done Done Sent false
2490 1606 Acked Read 8 Done Done Sent false
2491 1607 Acked Read 4 Done Done Sent false
2492 1608 Acked Write 1 Done Done Sent false
2493 1609 Acked Read 9 Done Done Sent false
2494 1610 Acked Read 7 Done Done Sent false
2495 1611 Acked Read 1 Done Done Sent false
2496 1612 Acked Read 10 Done Done Sent false
2497 1613 Acked Read 6 Done Done Sent false
2498 1614 Acked Read 4 Done Done Sent false
2499 1615 Acked Write 8 Done Done Sent false
2500 1616 Acked Read 4 Done Done Sent false
2501 1617 Acked Read 4 Done Done Sent false
2502 1618 Acked Read 5 Done Done Sent false
2503 1619 Acked Read 6 Done Done Sent false
2504 1620 Acked Write 2 Done Done Sent false
2505 1621 Acked Read 10 Done Done Sent false
2506 1622 Acked Write 3 Done Done Sent false
2507 1623 Acked Write 5 Done Done Sent false
2508 1624 Acked Read 5 Done Done Sent false
2509 1625 Acked Write 7 Done Done Sent false
2510 1626 Acked Read 5 Done Done Sent false
2511 1627 Acked Read 2 Done Done Sent false
2512 1628 Acked Write 4 Done Done Sent false
2513 1629 Acked Read 4 Done Done Sent false
2514 1630 Acked Read 1 Done Done Sent false
2515 1631 Acked Write 3 Done Done Sent false
2516 1632 Acked Flush 0 Done Done Sent false
2517 1633 Acked Read 10 Done Done Sent false
2518 1634 Acked Read 5 Done Done Sent false
2519 1635 Acked Read 4 Done Done Sent false
2520 1636 Acked Read 6 Done Done Sent false
2521 1637 Acked Read 9 Done Done Sent false
2522 1638 Acked Flush 0 Done Done Sent false
2523 1639 Acked Read 6 Done Done Sent false
2524 1640 Acked Read 2 Done Done Sent false
2525 1641 Acked Read 8 Done Done Sent false
2526 1642 Acked Write 6 Done Done Sent false
2527 1643 Acked Write 4 Done Done Sent false
2528 1644 Acked Write 3 Done Done Sent false
2529 1645 Acked Read 7 Done Done Sent false
2530 1646 Acked Read 8 Done Done Sent false
2531 1647 Acked Write 6 Done Done Sent false
2532 1648 Acked Write 5 Done Done Sent false
2533 1649 Acked Write 7 Done Done Sent false
2534 1650 Acked Write 7 Done Done Sent false
2535 1651 Acked Read 7 Done Done Sent false
2536 1652 Acked Write 3 Done Done Sent false
2537 1653 Acked Read 5 Done Done Sent false
2538 1654 Acked Write 8 Done Done Sent false
2539 1655 Acked Read 1 Done Done Sent false
2540 1656 Acked Write 7 Done Done Sent false
2541 1657 Acked Read 3 Done Done Sent false
2542 1658 Acked Read 6 Done Done Sent false
2543 1659 Acked Flush 0 Done Done Sent false
2544 1660 Acked Write 6 Done Done Sent false
2545 1661 Acked Read 2 Done Done Sent false
2546 1662 Acked Read 8 Done Done Sent false
2547 1663 Acked Read 6 Done Done Sent false
2548 1664 Acked Write 8 Done Done Sent false
2549 1665 Acked Read 6 Done Done Sent false
2550 1666 Acked Flush 0 Done Done Sent false
2551 1667 Acked Write 10 Done Done Sent false
2552 1668 Acked Write 6 Done Done Sent false
2553 1669 Acked Read 1 Done Done Sent false
2554 1670 Acked Write 9 Done Done Sent false
2555 1671 Acked Read 8 Done Done Sent false
2556 1672 Acked Read 2 Done Done Sent false
2557 1673 Acked Read 10 Done Done Sent false
2558 1674 Acked Read 3 Done Done Sent false
2559 1675 Acked Write 10 Done Done Sent false
2560 1676 Acked Read 3 Done Done Sent false
2561 1677 Acked Read 2 Done Done Sent false
2562 1678 Acked Read 7 Done Done Sent false
2563 1679 Acked Read 6 Done Done Sent false
2564 1680 Acked Read 3 Done Done Sent false
2565 1681 Acked Write 7 Done Done Sent false
2566 1682 Acked Write 5 Done Done Sent false
2567 1683 Acked Write 7 Done Done Sent false
2568 1684 Acked Read 7 Done Done Sent false
2569 1685 Acked Read 4 Done Done Sent false
2570 1686 Acked Flush 0 Done Done Sent false
2571 1687 Acked Write 4 Done Done Sent false
2572 1688 Acked Flush 0 Done Done Sent false
2573 1689 Acked Read 8 Done Done Sent false
2574 1690 Acked Write 1 Done Done Sent false
2575 1691 Acked Write 7 Done Done Sent false
2576 1692 Acked Read 2 Done Done Sent false
2577 1693 Acked Write 1 Done Done Sent false
2578 1694 Acked Write 7 Done Done Sent false
2579 1695 Acked Read 5 Done Done Sent false
2580 1696 Acked Read 5 Done Done Sent false
2581 1697 Acked Read 4 Done Done Sent false
2582 1698 Acked Read 1 Done Done Sent false
2583 1699 Acked Read 6 Done Done Sent false
2584 1700 Acked Flush 0 Done Done Sent false
2585 1701 Acked Read 3 Done Done Sent false
2586 1702 Acked Write 7 Done Done Sent false
2587 1703 Acked Write 1 Done Done Sent false
2588 1704 Acked Read 2 Done Done Sent false
2589 1705 Acked Write 7 Done Done Sent false
2590 1706 Acked Write 3 Done Done Sent false
2591 1707 Acked Write 8 Done Done Sent false
2592 1708 Acked Flush 0 Done Done Sent false
2593 1709 Acked Read 5 Done Done Sent false
2594 1710 Acked Write 8 Done Done Sent false
2595 1711 Acked Read 10 Done Done Sent false
2596 1712 Acked Write 5 Done Done Sent false
2597 1713 Acked Flush 0 Done Done Sent false
2598 1714 Acked Write 1 Done Done Sent false
2599 1715 Acked Read 10 Done Done Sent false
2600 1716 Acked Write 10 Done Done Sent false
2601 1717 Acked Write 1 Done Done Sent false
2602 1718 Acked Read 8 Done Done Sent false
2603 1719 Acked Flush 0 Done Done Sent false
2604 1720 Acked Flush 0 Done Done Sent false
2605 1721 Acked Read 7 Done Done Sent false
2606 1722 Acked Write 5 Done Done Sent false
2607 1723 Acked Write 6 Done Done Sent false
2608 1724 Acked Flush 0 Done Done Sent false
2609 1725 Acked Read 4 Done Done Sent false
2610 1726 Acked Write 8 Done Done Sent false
2611 1727 Acked Write 5 Done Done Sent false
2612 1728 Acked Read 9 Done Done Sent false
2613 1729 Acked Read 7 Done Done Sent false
2614 1730 Acked Write 1 Done Done Sent false
2615 1731 Acked Read 5 Done Done Sent false
2616 1732 Acked Read 2 Done Done Sent false
2617 1733 Acked Write 8 Done Done Sent false
2618 1734 Acked Write 7 Done Done Sent false
2619 1735 Acked Read 4 Done Done Sent false
2620 1736 Acked Read 8 Done Done Sent false
2621 1737 Acked Read 7 Done Done Sent false
2622 1738 Acked Write 8 Done Done Sent false
2623 1739 Acked Read 6 Done Done Sent false
2624 1740 Acked Read 3 Done Done Sent false
2625 1741 Acked Read 8 Done Done Sent false
2626 1742 Acked Write 1 Done Done Sent false
2627 1743 Acked Read 10 Done Done Sent false
2628 1744 Acked Read 10 Done Done Sent false
2629 1745 Acked Write 9 Done Done Sent false
2630 1746 Acked Write 1 Done Done Sent false
2631 1747 Acked Write 8 Done Done Sent false
2632 1748 Acked Write 7 Done Done Sent false
2633 1749 Acked Flush 0 Done Done Sent false
2634 1750 Acked Write 7 Done Done Sent false
2635 1751 Acked Read 2 Done Done Sent false
2636 1752 Acked Write 6 Done Done Sent false
2637 1753 Acked Write 5 Done Done Sent false
2638 1754 Acked Write 1 Done Done Sent false
2639 1755 Acked Write 8 Done Done Sent false
2640 1756 Acked Flush 0 Done Done Sent false
2641 1757 Acked Write 4 Done Done Sent false
2642 1758 Acked Read 8 Done Done Sent false
2643 1759 Acked Write 1 Done Done Sent false
2644 1760 Acked Flush 0 Done Done Sent false
2645 1761 Acked Read 2 Done Done Sent false
2646 1762 Acked Read 10 Done Done Sent false
2647 1763 Acked Write 3 Done Done Sent false
2648 1764 Acked Write 3 Done Done Sent false
2649 1765 Acked Write 3 Done Done Sent false
2650 1766 Acked Write 9 Done Done Sent false
2651 1767 Acked Read 10 Done Done Sent false
2652 1768 Acked Write 2 Done Done Sent false
2653 1769 Acked Write 1 Done Done Sent false
2654 1770 Acked Read 1 Done Done Sent false
2655 1771 Acked Read 4 Done Done Sent false
2656 1772 Acked Write 3 Done Done Sent false
2657 1773 Acked Flush 0 Done Done Sent false
2658 1774 Acked Read 3 Done Done Sent false
2659 1775 Acked Write 3 Done Done Sent false
2660 1776 Acked Write 5 Done Done Sent false
2661 1777 Acked Read 8 Done Done Sent false
2662 1778 Acked Write 3 Done Done Sent false
2663 1779 Acked Read 10 Done Done Sent false
2664 1780 Acked Write 5 Done Done Sent false
2665 1781 Acked Write 2 Done Done Sent false
2666 1782 Acked Write 8 Done Done Sent false
2667 1783 Acked Flush 0 Done Done Sent false
2668 1784 Acked Read 2 Done Done Sent false
2669 1785 Acked Write 3 Done Done Sent false
2670 1786 Acked Write 6 Done Done Sent false
2671 1787 Acked Write 9 Done Done Sent false
2672 1788 Acked Write 2 Done Done Sent false
2673 1789 Acked Flush 0 Done Done Sent false
2674 1790 Acked Read 4 Done Done Sent false
2675 1791 Acked Write 10 Done Done Sent false
2676 1792 Acked Flush 0 Done Done Sent false
2677 1793 Acked Write 4 Done Done Sent false
2678 1794 Acked Read 1 Done Done Sent false
2679 1795 Acked Flush 0 Done Done Sent false
2680 1796 Acked Write 9 Done Done Sent false
2681 1797 Acked Read 1 Done Done Sent false
2682 1798 Acked Flush 0 Done Done Sent false
2683 1799 Acked Read 9 Done Done Sent false
2684 1800 Acked Write 2 Done Done Sent false
2685 1801 Acked Read 2 Done Done Sent false
2686 1802 Acked Read 9 Done Done Sent false
2687 1803 Acked Read 10 Done Done Sent false
2688 1804 Acked Read 3 Done Done Sent false
2689 1805 Acked Write 6 Done Done Sent false
2690 1806 Acked Read 2 Done Done Sent false
2691 1807 Acked Write 4 Done Done Sent false
2692 1808 Acked Write 3 Done Done Sent false
2693 1809 Acked Read 10 Done Done Sent false
2694 1810 Acked Flush 0 Done Done Sent false
2695 1811 Acked Write 7 Done Done Sent false
2696 1812 Acked Read 4 Done Done Sent false
2697 1813 Acked Flush 0 Done Done Sent false
2698 1814 Acked Flush 0 Done Done Sent false
2699 1815 Acked Read 1 Done Done Sent false
2700 1816 Acked Read 4 Done Done Sent false
2701 1817 Acked Write 6 Done Done Sent false
2702 1818 Acked Read 8 Done Done Sent false
2703 1819 Acked Read 7 Done Done Sent false
2704 1820 Acked Write 5 Done Done Sent false
2705 1821 Acked Read 1 Done Done Sent false
2706 1822 Acked Write 8 Done Done Sent false
2707 1823 Acked Write 5 Done Done Sent false
2708 1824 Acked Flush 0 Done Done Sent false
2709 1825 Acked Write 8 Done Done Sent false
2710 1826 Acked Write 7 Done Done Sent false
2711 1827 Acked Read 6 Done Done Sent false
2712 1828 Acked Read 10 Done Done Sent false
2713 1829 Acked Read 4 Done Done Sent false
2714 1830 Acked Write 9 Done Done Sent false
2715 1831 Acked Write 2 Done Done Sent false
2716 1832 Acked Read 6 Done Done Sent false
2717 1833 Acked Read 8 Done Done Sent false
2718 1834 Acked Read 1 Done Done Sent false
2719 1835 Acked Write 6 Done Done Sent false
2720 1836 Acked Write 4 Done Done Sent false
2721 1837 Acked Read 5 Done Done Sent false
2722 1838 Acked Read 9 Done Done Sent false
2723 1839 Acked Flush 0 Done Done Sent false
2724 1840 Acked Write 10 Done Done Sent false
2725 1841 Acked Flush 0 Done Done Sent false
2726 1842 Acked Write 8 Done Done Sent false
2727 1843 Acked Read 3 Done Done Sent false
2728 1844 Acked Flush 0 Done Done Sent false
2729 1845 Acked Flush 0 Done Done Sent false
2730 1846 Acked Read 9 Done Done Sent false
2731 1847 Acked Write 5 Done Done Sent false
2732 1848 Acked Read 3 Done Done Sent false
2733 1849 Acked Flush 0 Done Done Sent false
2734 1850 Acked Read 1 Done Done Sent false
2735 1851 Acked Write 7 Done Done Sent false
2736 1852 Acked Write 6 Done Done Sent false
2737 1853 Acked Write 5 Done Done Sent false
2738 1854 Acked Write 9 Done Done Sent false
2739 1855 Acked Write 3 Done Done Sent false
2740 1856 Acked Write 3 Done Done Sent false
2741 1857 Acked Read 6 Done Done Sent false
2742 1858 Acked Read 3 Done Done Sent false
2743 1859 Acked Read 3 Done Done Sent false
2744 1860 Acked Read 10 Done Done Sent false
2745 1861 Acked Flush 0 Done Done Sent false
2746 1862 Acked Read 8 Done Done Sent false
2747 1863 Acked Flush 0 Done Done Sent false
2748 1864 Acked Read 6 Done Done Sent false
2749 1865 Acked Write 3 Done Done Sent false
2750 1866 Acked Read 1 Done Done Sent false
2751 1867 Acked Read 10 Done Done Sent false
2752 1868 Acked Write 8 Done Done Sent false
2753 1869 Acked Write 7 Done Done Sent false
2754 1870 Acked Write 5 Done Done Sent false
2755 1871 Acked Read 3 Done Done Sent false
2756 1872 Acked Write 2 Done Done Sent false
2757 1873 Acked Read 1 Done Done Sent false
2758 1874 Acked Read 5 Done Done Sent false
2759 1875 Acked Read 7 Done Done Sent false
2760 1876 Acked Read 4 Done Done Sent false
2761 1877 Acked Read 1 Done Done Sent false
2762 1878 Acked Write 6 Done Done Sent false
2763 1879 Acked Read 10 Done Done Sent false
2764 1880 Acked Read 5 Done Done Sent false
2765 1881 Acked Write 3 Done Done Sent false
2766 1882 Acked Read 10 Done Done Sent false
2767 1883 Acked Read 8 Done Done Sent false
2768 1884 Acked Read 2 Done Done Sent false
2769 1885 Acked Read 2 Done Done Sent false
2770 1886 Acked Write 7 Done Done Sent false
2771 1887 Acked Read 3 Done Done Sent false
2772 1888 Acked Write 9 Done Done Sent false
2773 1889 Acked Read 2 Done Done Sent false
2774 1890 Acked Read 6 Done Done Sent false
2775 1891 Acked Write 7 Done Done Sent false
2776 1892 Acked Write 10 Done Done Sent false
2777 1893 Acked Flush 0 Done Done Sent false
2778 1894 Acked Flush 0 Done Done Sent false
2779 1895 Acked Write 6 Done Done Sent false
2780 1896 Acked Read 7 Done Done Sent false
2781 1897 Acked Read 8 Done Done Sent false
2782 1898 Acked Read 2 Done Done Sent false
2783 1899 Acked Read 3 Done Done Sent false
2784 1900 Acked Write 9 Done Done Sent false
2785 1901 Acked Read 5 Done Done Sent false
2786 1902 Acked Read 8 Done Done Sent false
2787 1903 Acked Write 1 Done Done Sent false
2788 STATES DS:0 DS:1 DS:2 TOTAL
2789 Sent 0 0 301 301
2790 Done 301 301 0 602
2791 Skipped 0 0 0 0
2792 Error 0 0 0 0
2793 Last Flush: Some(JobId(1894)) Some(JobId(1894)) Some(JobId(1602))
2794 Downstairs last five completed: 1602 1601 1600 1599 1598
2795 Upstairs last five completed: 1903 1902 1901 1900 1899
2796 CLIENT: Up:0 ds:301 act:2
27972025-02-25T20:47:14.114ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
27982025-02-25T20:47:15.114ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
27992025-02-25T20:47:16.116ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
2800 ----------------------------------------------------------------
2801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2802 States: Active Active Offline
2803 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2804 1603 Acked Read 10 Done Done Sent false
2805 1604 Acked Read 5 Done Done Sent false
2806 1605 Acked Read 1 Done Done Sent false
2807 1606 Acked Read 8 Done Done Sent false
2808 1607 Acked Read 4 Done Done Sent false
2809 1608 Acked Write 1 Done Done Sent false
2810 1609 Acked Read 9 Done Done Sent false
2811 1610 Acked Read 7 Done Done Sent false
2812 1611 Acked Read 1 Done Done Sent false
2813 1612 Acked Read 10 Done Done Sent false
2814 1613 Acked Read 6 Done Done Sent false
2815 1614 Acked Read 4 Done Done Sent false
2816 1615 Acked Write 8 Done Done Sent false
2817 1616 Acked Read 4 Done Done Sent false
2818 1617 Acked Read 4 Done Done Sent false
2819 1618 Acked Read 5 Done Done Sent false
2820 1619 Acked Read 6 Done Done Sent false
2821 1620 Acked Write 2 Done Done Sent false
2822 1621 Acked Read 10 Done Done Sent false
2823 1622 Acked Write 3 Done Done Sent false
2824 1623 Acked Write 5 Done Done Sent false
2825 1624 Acked Read 5 Done Done Sent false
2826 1625 Acked Write 7 Done Done Sent false
2827 1626 Acked Read 5 Done Done Sent false
2828 1627 Acked Read 2 Done Done Sent false
2829 1628 Acked Write 4 Done Done Sent false
2830 1629 Acked Read 4 Done Done Sent false
2831 1630 Acked Read 1 Done Done Sent false
2832 1631 Acked Write 3 Done Done Sent false
2833 1632 Acked Flush 0 Done Done Sent false
2834 1633 Acked Read 10 Done Done Sent false
2835 1634 Acked Read 5 Done Done Sent false
2836 1635 Acked Read 4 Done Done Sent false
2837 1636 Acked Read 6 Done Done Sent false
2838 1637 Acked Read 9 Done Done Sent false
2839 1638 Acked Flush 0 Done Done Sent false
2840 1639 Acked Read 6 Done Done Sent false
2841 1640 Acked Read 2 Done Done Sent false
2842 1641 Acked Read 8 Done Done Sent false
2843 1642 Acked Write 6 Done Done Sent false
2844 1643 Acked Write 4 Done Done Sent false
2845 1644 Acked Write 3 Done Done Sent false
2846 1645 Acked Read 7 Done Done Sent false
2847 1646 Acked Read 8 Done Done Sent false
2848 1647 Acked Write 6 Done Done Sent false
2849 1648 Acked Write 5 Done Done Sent false
2850 1649 Acked Write 7 Done Done Sent false
2851 1650 Acked Write 7 Done Done Sent false
2852 1651 Acked Read 7 Done Done Sent false
2853 1652 Acked Write 3 Done Done Sent false
2854 1653 Acked Read 5 Done Done Sent false
2855 1654 Acked Write 8 Done Done Sent false
2856 1655 Acked Read 1 Done Done Sent false
2857 1656 Acked Write 7 Done Done Sent false
2858 1657 Acked Read 3 Done Done Sent false
2859 1658 Acked Read 6 Done Done Sent false
2860 1659 Acked Flush 0 Done Done Sent false
2861 1660 Acked Write 6 Done Done Sent false
2862 1661 Acked Read 2 Done Done Sent false
2863 1662 Acked Read 8 Done Done Sent false
2864 1663 Acked Read 6 Done Done Sent false
2865 1664 Acked Write 8 Done Done Sent false
2866 1665 Acked Read 6 Done Done Sent false
2867 1666 Acked Flush 0 Done Done Sent false
2868 1667 Acked Write 10 Done Done Sent false
2869 1668 Acked Write 6 Done Done Sent false
2870 1669 Acked Read 1 Done Done Sent false
2871 1670 Acked Write 9 Done Done Sent false
2872 1671 Acked Read 8 Done Done Sent false
2873 1672 Acked Read 2 Done Done Sent false
2874 1673 Acked Read 10 Done Done Sent false
2875 1674 Acked Read 3 Done Done Sent false
2876 1675 Acked Write 10 Done Done Sent false
2877 1676 Acked Read 3 Done Done Sent false
2878 1677 Acked Read 2 Done Done Sent false
2879 1678 Acked Read 7 Done Done Sent false
2880 1679 Acked Read 6 Done Done Sent false
2881 1680 Acked Read 3 Done Done Sent false
2882 1681 Acked Write 7 Done Done Sent false
2883 1682 Acked Write 5 Done Done Sent false
2884 1683 Acked Write 7 Done Done Sent false
2885 1684 Acked Read 7 Done Done Sent false
2886 1685 Acked Read 4 Done Done Sent false
2887 1686 Acked Flush 0 Done Done Sent false
2888 1687 Acked Write 4 Done Done Sent false
2889 1688 Acked Flush 0 Done Done Sent false
2890 1689 Acked Read 8 Done Done Sent false
2891 1690 Acked Write 1 Done Done Sent false
2892 1691 Acked Write 7 Done Done Sent false
2893 1692 Acked Read 2 Done Done Sent false
2894 1693 Acked Write 1 Done Done Sent false
2895 1694 Acked Write 7 Done Done Sent false
2896 1695 Acked Read 5 Done Done Sent false
2897 1696 Acked Read 5 Done Done Sent false
2898 1697 Acked Read 4 Done Done Sent false
2899 1698 Acked Read 1 Done Done Sent false
2900 1699 Acked Read 6 Done Done Sent false
2901 1700 Acked Flush 0 Done Done Sent false
2902 1701 Acked Read 3 Done Done Sent false
2903 1702 Acked Write 7 Done Done Sent false
2904 1703 Acked Write 1 Done Done Sent false
2905 1704 Acked Read 2 Done Done Sent false
2906 1705 Acked Write 7 Done Done Sent false
2907 1706 Acked Write 3 Done Done Sent false
2908 1707 Acked Write 8 Done Done Sent false
2909 1708 Acked Flush 0 Done Done Sent false
2910 1709 Acked Read 5 Done Done Sent false
2911 1710 Acked Write 8 Done Done Sent false
2912 1711 Acked Read 10 Done Done Sent false
2913 1712 Acked Write 5 Done Done Sent false
2914 1713 Acked Flush 0 Done Done Sent false
2915 1714 Acked Write 1 Done Done Sent false
2916 1715 Acked Read 10 Done Done Sent false
2917 1716 Acked Write 10 Done Done Sent false
2918 1717 Acked Write 1 Done Done Sent false
2919 1718 Acked Read 8 Done Done Sent false
2920 1719 Acked Flush 0 Done Done Sent false
2921 1720 Acked Flush 0 Done Done Sent false
2922 1721 Acked Read 7 Done Done Sent false
2923 1722 Acked Write 5 Done Done Sent false
2924 1723 Acked Write 6 Done Done Sent false
2925 1724 Acked Flush 0 Done Done Sent false
2926 1725 Acked Read 4 Done Done Sent false
2927 1726 Acked Write 8 Done Done Sent false
2928 1727 Acked Write 5 Done Done Sent false
2929 1728 Acked Read 9 Done Done Sent false
2930 1729 Acked Read 7 Done Done Sent false
2931 1730 Acked Write 1 Done Done Sent false
2932 1731 Acked Read 5 Done Done Sent false
2933 1732 Acked Read 2 Done Done Sent false
2934 1733 Acked Write 8 Done Done Sent false
2935 1734 Acked Write 7 Done Done Sent false
2936 1735 Acked Read 4 Done Done Sent false
2937 1736 Acked Read 8 Done Done Sent false
2938 1737 Acked Read 7 Done Done Sent false
2939 1738 Acked Write 8 Done Done Sent false
2940 1739 Acked Read 6 Done Done Sent false
2941 1740 Acked Read 3 Done Done Sent false
2942 1741 Acked Read 8 Done Done Sent false
2943 1742 Acked Write 1 Done Done Sent false
2944 1743 Acked Read 10 Done Done Sent false
2945 1744 Acked Read 10 Done Done Sent false
2946 1745 Acked Write 9 Done Done Sent false
2947 1746 Acked Write 1 Done Done Sent false
2948 1747 Acked Write 8 Done Done Sent false
2949 1748 Acked Write 7 Done Done Sent false
2950 1749 Acked Flush 0 Done Done Sent false
2951 1750 Acked Write 7 Done Done Sent false
2952 1751 Acked Read 2 Done Done Sent false
2953 1752 Acked Write 6 Done Done Sent false
2954 1753 Acked Write 5 Done Done Sent false
2955 1754 Acked Write 1 Done Done Sent false
2956 1755 Acked Write 8 Done Done Sent false
2957 1756 Acked Flush 0 Done Done Sent false
2958 1757 Acked Write 4 Done Done Sent false
2959 1758 Acked Read 8 Done Done Sent false
2960 1759 Acked Write 1 Done Done Sent false
2961 1760 Acked Flush 0 Done Done Sent false
2962 1761 Acked Read 2 Done Done Sent false
2963 1762 Acked Read 10 Done Done Sent false
2964 1763 Acked Write 3 Done Done Sent false
2965 1764 Acked Write 3 Done Done Sent false
2966 1765 Acked Write 3 Done Done Sent false
2967 1766 Acked Write 9 Done Done Sent false
2968 1767 Acked Read 10 Done Done Sent false
2969 1768 Acked Write 2 Done Done Sent false
2970 1769 Acked Write 1 Done Done Sent false
2971 1770 Acked Read 1 Done Done Sent false
2972 1771 Acked Read 4 Done Done Sent false
2973 1772 Acked Write 3 Done Done Sent false
2974 1773 Acked Flush 0 Done Done Sent false
2975 1774 Acked Read 3 Done Done Sent false
2976 1775 Acked Write 3 Done Done Sent false
2977 1776 Acked Write 5 Done Done Sent false
2978 1777 Acked Read 8 Done Done Sent false
2979 1778 Acked Write 3 Done Done Sent false
2980 1779 Acked Read 10 Done Done Sent false
2981 1780 Acked Write 5 Done Done Sent false
2982 1781 Acked Write 2 Done Done Sent false
2983 1782 Acked Write 8 Done Done Sent false
2984 1783 Acked Flush 0 Done Done Sent false
2985 1784 Acked Read 2 Done Done Sent false
2986 1785 Acked Write 3 Done Done Sent false
2987 1786 Acked Write 6 Done Done Sent false
2988 1787 Acked Write 9 Done Done Sent false
2989 1788 Acked Write 2 Done Done Sent false
2990 1789 Acked Flush 0 Done Done Sent false
2991 1790 Acked Read 4 Done Done Sent false
2992 1791 Acked Write 10 Done Done Sent false
2993 1792 Acked Flush 0 Done Done Sent false
2994 1793 Acked Write 4 Done Done Sent false
2995 1794 Acked Read 1 Done Done Sent false
2996 1795 Acked Flush 0 Done Done Sent false
2997 1796 Acked Write 9 Done Done Sent false
2998 1797 Acked Read 1 Done Done Sent false
2999 1798 Acked Flush 0 Done Done Sent false
3000 1799 Acked Read 9 Done Done Sent false
3001 1800 Acked Write 2 Done Done Sent false
3002 1801 Acked Read 2 Done Done Sent false
3003 1802 Acked Read 9 Done Done Sent false
3004 1803 Acked Read 10 Done Done Sent false
3005 1804 Acked Read 3 Done Done Sent false
3006 1805 Acked Write 6 Done Done Sent false
3007 1806 Acked Read 2 Done Done Sent false
3008 1807 Acked Write 4 Done Done Sent false
3009 1808 Acked Write 3 Done Done Sent false
3010 1809 Acked Read 10 Done Done Sent false
3011 1810 Acked Flush 0 Done Done Sent false
3012 1811 Acked Write 7 Done Done Sent false
3013 1812 Acked Read 4 Done Done Sent false
3014 1813 Acked Flush 0 Done Done Sent false
3015 1814 Acked Flush 0 Done Done Sent false
3016 1815 Acked Read 1 Done Done Sent false
3017 1816 Acked Read 4 Done Done Sent false
3018 1817 Acked Write 6 Done Done Sent false
3019 1818 Acked Read 8 Done Done Sent false
3020 1819 Acked Read 7 Done Done Sent false
3021 1820 Acked Write 5 Done Done Sent false
3022 1821 Acked Read 1 Done Done Sent false
3023 1822 Acked Write 8 Done Done Sent false
3024 1823 Acked Write 5 Done Done Sent false
3025 1824 Acked Flush 0 Done Done Sent false
3026 1825 Acked Write 8 Done Done Sent false
3027 1826 Acked Write 7 Done Done Sent false
3028 1827 Acked Read 6 Done Done Sent false
3029 1828 Acked Read 10 Done Done Sent false
3030 1829 Acked Read 4 Done Done Sent false
3031 1830 Acked Write 9 Done Done Sent false
3032 1831 Acked Write 2 Done Done Sent false
3033 1832 Acked Read 6 Done Done Sent false
3034 1833 Acked Read 8 Done Done Sent false
3035 1834 Acked Read 1 Done Done Sent false
3036 1835 Acked Write 6 Done Done Sent false
3037 1836 Acked Write 4 Done Done Sent false
3038 1837 Acked Read 5 Done Done Sent false
3039 1838 Acked Read 9 Done Done Sent false
3040 1839 Acked Flush 0 Done Done Sent false
3041 1840 Acked Write 10 Done Done Sent false
3042 1841 Acked Flush 0 Done Done Sent false
3043 1842 Acked Write 8 Done Done Sent false
3044 1843 Acked Read 3 Done Done Sent false
3045 1844 Acked Flush 0 Done Done Sent false
3046 1845 Acked Flush 0 Done Done Sent false
3047 1846 Acked Read 9 Done Done Sent false
3048 1847 Acked Write 5 Done Done Sent false
3049 1848 Acked Read 3 Done Done Sent false
3050 1849 Acked Flush 0 Done Done Sent false
3051 1850 Acked Read 1 Done Done Sent false
3052 1851 Acked Write 7 Done Done Sent false
3053 1852 Acked Write 6 Done Done Sent false
3054 1853 Acked Write 5 Done Done Sent false
3055 1854 Acked Write 9 Done Done Sent false
3056 1855 Acked Write 3 Done Done Sent false
3057 1856 Acked Write 3 Done Done Sent false
3058 1857 Acked Read 6 Done Done Sent false
3059 1858 Acked Read 3 Done Done Sent false
3060 1859 Acked Read 3 Done Done Sent false
3061 1860 Acked Read 10 Done Done Sent false
3062 1861 Acked Flush 0 Done Done Sent false
3063 1862 Acked Read 8 Done Done Sent false
3064 1863 Acked Flush 0 Done Done Sent false
3065 1864 Acked Read 6 Done Done Sent false
3066 1865 Acked Write 3 Done Done Sent false
3067 1866 Acked Read 1 Done Done Sent false
3068 1867 Acked Read 10 Done Done Sent false
3069 1868 Acked Write 8 Done Done Sent false
3070 1869 Acked Write 7 Done Done Sent false
3071 1870 Acked Write 5 Done Done Sent false
3072 1871 Acked Read 3 Done Done Sent false
3073 1872 Acked Write 2 Done Done Sent false
3074 1873 Acked Read 1 Done Done Sent false
3075 1874 Acked Read 5 Done Done Sent false
3076 1875 Acked Read 7 Done Done Sent false
3077 1876 Acked Read 4 Done Done Sent false
3078 1877 Acked Read 1 Done Done Sent false
3079 1878 Acked Write 6 Done Done Sent false
3080 1879 Acked Read 10 Done Done Sent false
3081 1880 Acked Read 5 Done Done Sent false
3082 1881 Acked Write 3 Done Done Sent false
3083 1882 Acked Read 10 Done Done Sent false
3084 1883 Acked Read 8 Done Done Sent false
3085 1884 Acked Read 2 Done Done Sent false
3086 1885 Acked Read 2 Done Done Sent false
3087 1886 Acked Write 7 Done Done Sent false
3088 1887 Acked Read 3 Done Done Sent false
3089 1888 Acked Write 9 Done Done Sent false
3090 1889 Acked Read 2 Done Done Sent false
3091 1890 Acked Read 6 Done Done Sent false
3092 1891 Acked Write 7 Done Done Sent false
3093 1892 Acked Write 10 Done Done Sent false
3094 1893 Acked Flush 0 Done Done Sent false
3095 1894 Acked Flush 0 Done Done Sent false
3096 1895 Acked Write 6 Done Done Sent false
3097 1896 Acked Read 7 Done Done Sent false
3098 1897 Acked Read 8 Done Done Sent false
3099 1898 Acked Read 2 Done Done Sent false
3100 1899 Acked Read 3 Done Done Sent false
3101 1900 Acked Write 9 Done Done Sent false
3102 1901 Acked Read 5 Done Done Sent false
3103 1902 Acked Read 8 Done Done Sent false
3104 1903 Acked Write 1 Done Done Sent false
3105 STATES DS:0 DS:1 DS:2 TOTAL
3106 Sent 0 0 301 301
3107 Done 301 301 0 602
3108 Skipped 0 0 0 0
3109 Error 0 0 0 0
3110 Last Flush: Some(JobId(1894)) Some(JobId(1894)) Some(JobId(1602))
3111 Downstairs last five completed: 1602 1601 1600 1599 1598
3112 Upstairs last five completed: 1903 1902 1901 1900 1899
3113 CLIENT: Up:0 ds:301 act:2
31142025-02-25T20:47:18.119ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31152025-02-25T20:47:19.108ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31162025-02-25T20:47:19.108ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31172025-02-25T20:47:19.108ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31182025-02-25T20:47:19.108ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31192025-02-25T20:47:19.108ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31202025-02-25T20:47:20.123ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31212025-02-25T20:47:21.126ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3122 ----------------------------------------------------------------
3123 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3124 States: Active Active Offline
3125 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3126 1603 Acked Read 10 Done Done Sent false
3127 1604 Acked Read 5 Done Done Sent false
3128 1605 Acked Read 1 Done Done Sent false
3129 1606 Acked Read 8 Done Done Sent false
3130 1607 Acked Read 4 Done Done Sent false
3131 1608 Acked Write 1 Done Done Sent false
3132 1609 Acked Read 9 Done Done Sent false
3133 1610 Acked Read 7 Done Done Sent false
3134 1611 Acked Read 1 Done Done Sent false
3135 1612 Acked Read 10 Done Done Sent false
3136 1613 Acked Read 6 Done Done Sent false
3137 1614 Acked Read 4 Done Done Sent false
3138 1615 Acked Write 8 Done Done Sent false
3139 1616 Acked Read 4 Done Done Sent false
3140 1617 Acked Read 4 Done Done Sent false
3141 1618 Acked Read 5 Done Done Sent false
3142 1619 Acked Read 6 Done Done Sent false
3143 1620 Acked Write 2 Done Done Sent false
3144 1621 Acked Read 10 Done Done Sent false
3145 1622 Acked Write 3 Done Done Sent false
3146 1623 Acked Write 5 Done Done Sent false
3147 1624 Acked Read 5 Done Done Sent false
3148 1625 Acked Write 7 Done Done Sent false
3149 1626 Acked Read 5 Done Done Sent false
3150 1627 Acked Read 2 Done Done Sent false
3151 1628 Acked Write 4 Done Done Sent false
3152 1629 Acked Read 4 Done Done Sent false
3153 1630 Acked Read 1 Done Done Sent false
3154 1631 Acked Write 3 Done Done Sent false
3155 1632 Acked Flush 0 Done Done Sent false
3156 1633 Acked Read 10 Done Done Sent false
3157 1634 Acked Read 5 Done Done Sent false
3158 1635 Acked Read 4 Done Done Sent false
3159 1636 Acked Read 6 Done Done Sent false
3160 1637 Acked Read 9 Done Done Sent false
3161 1638 Acked Flush 0 Done Done Sent false
3162 1639 Acked Read 6 Done Done Sent false
3163 1640 Acked Read 2 Done Done Sent false
3164 1641 Acked Read 8 Done Done Sent false
3165 1642 Acked Write 6 Done Done Sent false
3166 1643 Acked Write 4 Done Done Sent false
3167 1644 Acked Write 3 Done Done Sent false
3168 1645 Acked Read 7 Done Done Sent false
3169 1646 Acked Read 8 Done Done Sent false
3170 1647 Acked Write 6 Done Done Sent false
3171 1648 Acked Write 5 Done Done Sent false
3172 1649 Acked Write 7 Done Done Sent false
3173 1650 Acked Write 7 Done Done Sent false
3174 1651 Acked Read 7 Done Done Sent false
3175 1652 Acked Write 3 Done Done Sent false
3176 1653 Acked Read 5 Done Done Sent false
3177 1654 Acked Write 8 Done Done Sent false
3178 1655 Acked Read 1 Done Done Sent false
3179 1656 Acked Write 7 Done Done Sent false
3180 1657 Acked Read 3 Done Done Sent false
3181 1658 Acked Read 6 Done Done Sent false
3182 1659 Acked Flush 0 Done Done Sent false
3183 1660 Acked Write 6 Done Done Sent false
3184 1661 Acked Read 2 Done Done Sent false
3185 1662 Acked Read 8 Done Done Sent false
3186 1663 Acked Read 6 Done Done Sent false
3187 1664 Acked Write 8 Done Done Sent false
3188 1665 Acked Read 6 Done Done Sent false
3189 1666 Acked Flush 0 Done Done Sent false
3190 1667 Acked Write 10 Done Done Sent false
3191 1668 Acked Write 6 Done Done Sent false
3192 1669 Acked Read 1 Done Done Sent false
3193 1670 Acked Write 9 Done Done Sent false
3194 1671 Acked Read 8 Done Done Sent false
3195 1672 Acked Read 2 Done Done Sent false
3196 1673 Acked Read 10 Done Done Sent false
3197 1674 Acked Read 3 Done Done Sent false
3198 1675 Acked Write 10 Done Done Sent false
3199 1676 Acked Read 3 Done Done Sent false
3200 1677 Acked Read 2 Done Done Sent false
3201 1678 Acked Read 7 Done Done Sent false
3202 1679 Acked Read 6 Done Done Sent false
3203 1680 Acked Read 3 Done Done Sent false
3204 1681 Acked Write 7 Done Done Sent false
3205 1682 Acked Write 5 Done Done Sent false
3206 1683 Acked Write 7 Done Done Sent false
3207 1684 Acked Read 7 Done Done Sent false
3208 1685 Acked Read 4 Done Done Sent false
3209 1686 Acked Flush 0 Done Done Sent false
3210 1687 Acked Write 4 Done Done Sent false
3211 1688 Acked Flush 0 Done Done Sent false
3212 1689 Acked Read 8 Done Done Sent false
3213 1690 Acked Write 1 Done Done Sent false
3214 1691 Acked Write 7 Done Done Sent false
3215 1692 Acked Read 2 Done Done Sent false
3216 1693 Acked Write 1 Done Done Sent false
3217 1694 Acked Write 7 Done Done Sent false
3218 1695 Acked Read 5 Done Done Sent false
3219 1696 Acked Read 5 Done Done Sent false
3220 1697 Acked Read 4 Done Done Sent false
3221 1698 Acked Read 1 Done Done Sent false
3222 1699 Acked Read 6 Done Done Sent false
3223 1700 Acked Flush 0 Done Done Sent false
3224 1701 Acked Read 3 Done Done Sent false
3225 1702 Acked Write 7 Done Done Sent false
3226 1703 Acked Write 1 Done Done Sent false
3227 1704 Acked Read 2 Done Done Sent false
3228 1705 Acked Write 7 Done Done Sent false
3229 1706 Acked Write 3 Done Done Sent false
3230 1707 Acked Write 8 Done Done Sent false
3231 1708 Acked Flush 0 Done Done Sent false
3232 1709 Acked Read 5 Done Done Sent false
3233 1710 Acked Write 8 Done Done Sent false
3234 1711 Acked Read 10 Done Done Sent false
3235 1712 Acked Write 5 Done Done Sent false
3236 1713 Acked Flush 0 Done Done Sent false
3237 1714 Acked Write 1 Done Done Sent false
3238 1715 Acked Read 10 Done Done Sent false
3239 1716 Acked Write 10 Done Done Sent false
3240 1717 Acked Write 1 Done Done Sent false
3241 1718 Acked Read 8 Done Done Sent false
3242 1719 Acked Flush 0 Done Done Sent false
3243 1720 Acked Flush 0 Done Done Sent false
3244 1721 Acked Read 7 Done Done Sent false
3245 1722 Acked Write 5 Done Done Sent false
3246 1723 Acked Write 6 Done Done Sent false
3247 1724 Acked Flush 0 Done Done Sent false
3248 1725 Acked Read 4 Done Done Sent false
3249 1726 Acked Write 8 Done Done Sent false
3250 1727 Acked Write 5 Done Done Sent false
3251 1728 Acked Read 9 Done Done Sent false
3252 1729 Acked Read 7 Done Done Sent false
3253 1730 Acked Write 1 Done Done Sent false
3254 1731 Acked Read 5 Done Done Sent false
3255 1732 Acked Read 2 Done Done Sent false
3256 1733 Acked Write 8 Done Done Sent false
3257 1734 Acked Write 7 Done Done Sent false
3258 1735 Acked Read 4 Done Done Sent false
3259 1736 Acked Read 8 Done Done Sent false
3260 1737 Acked Read 7 Done Done Sent false
3261 1738 Acked Write 8 Done Done Sent false
3262 1739 Acked Read 6 Done Done Sent false
3263 1740 Acked Read 3 Done Done Sent false
3264 1741 Acked Read 8 Done Done Sent false
3265 1742 Acked Write 1 Done Done Sent false
3266 1743 Acked Read 10 Done Done Sent false
3267 1744 Acked Read 10 Done Done Sent false
3268 1745 Acked Write 9 Done Done Sent false
3269 1746 Acked Write 1 Done Done Sent false
3270 1747 Acked Write 8 Done Done Sent false
3271 1748 Acked Write 7 Done Done Sent false
3272 1749 Acked Flush 0 Done Done Sent false
3273 1750 Acked Write 7 Done Done Sent false
3274 1751 Acked Read 2 Done Done Sent false
3275 1752 Acked Write 6 Done Done Sent false
3276 1753 Acked Write 5 Done Done Sent false
3277 1754 Acked Write 1 Done Done Sent false
3278 1755 Acked Write 8 Done Done Sent false
3279 1756 Acked Flush 0 Done Done Sent false
3280 1757 Acked Write 4 Done Done Sent false
3281 1758 Acked Read 8 Done Done Sent false
3282 1759 Acked Write 1 Done Done Sent false
3283 1760 Acked Flush 0 Done Done Sent false
3284 1761 Acked Read 2 Done Done Sent false
3285 1762 Acked Read 10 Done Done Sent false
3286 1763 Acked Write 3 Done Done Sent false
3287 1764 Acked Write 3 Done Done Sent false
3288 1765 Acked Write 3 Done Done Sent false
3289 1766 Acked Write 9 Done Done Sent false
3290 1767 Acked Read 10 Done Done Sent false
3291 1768 Acked Write 2 Done Done Sent false
3292 1769 Acked Write 1 Done Done Sent false
3293 1770 Acked Read 1 Done Done Sent false
3294 1771 Acked Read 4 Done Done Sent false
3295 1772 Acked Write 3 Done Done Sent false
3296 1773 Acked Flush 0 Done Done Sent false
3297 1774 Acked Read 3 Done Done Sent false
3298 1775 Acked Write 3 Done Done Sent false
3299 1776 Acked Write 5 Done Done Sent false
3300 1777 Acked Read 8 Done Done Sent false
3301 1778 Acked Write 3 Done Done Sent false
3302 1779 Acked Read 10 Done Done Sent false
3303 1780 Acked Write 5 Done Done Sent false
3304 1781 Acked Write 2 Done Done Sent false
3305 1782 Acked Write 8 Done Done Sent false
3306 1783 Acked Flush 0 Done Done Sent false
3307 1784 Acked Read 2 Done Done Sent false
3308 1785 Acked Write 3 Done Done Sent false
3309 1786 Acked Write 6 Done Done Sent false
3310 1787 Acked Write 9 Done Done Sent false
3311 1788 Acked Write 2 Done Done Sent false
3312 1789 Acked Flush 0 Done Done Sent false
3313 1790 Acked Read 4 Done Done Sent false
3314 1791 Acked Write 10 Done Done Sent false
3315 1792 Acked Flush 0 Done Done Sent false
3316 1793 Acked Write 4 Done Done Sent false
3317 1794 Acked Read 1 Done Done Sent false
3318 1795 Acked Flush 0 Done Done Sent false
3319 1796 Acked Write 9 Done Done Sent false
3320 1797 Acked Read 1 Done Done Sent false
3321 1798 Acked Flush 0 Done Done Sent false
3322 1799 Acked Read 9 Done Done Sent false
3323 1800 Acked Write 2 Done Done Sent false
3324 1801 Acked Read 2 Done Done Sent false
3325 1802 Acked Read 9 Done Done Sent false
3326 1803 Acked Read 10 Done Done Sent false
3327 1804 Acked Read 3 Done Done Sent false
3328 1805 Acked Write 6 Done Done Sent false
3329 1806 Acked Read 2 Done Done Sent false
3330 1807 Acked Write 4 Done Done Sent false
3331 1808 Acked Write 3 Done Done Sent false
3332 1809 Acked Read 10 Done Done Sent false
3333 1810 Acked Flush 0 Done Done Sent false
3334 1811 Acked Write 7 Done Done Sent false
3335 1812 Acked Read 4 Done Done Sent false
3336 1813 Acked Flush 0 Done Done Sent false
3337 1814 Acked Flush 0 Done Done Sent false
3338 1815 Acked Read 1 Done Done Sent false
3339 1816 Acked Read 4 Done Done Sent false
3340 1817 Acked Write 6 Done Done Sent false
3341 1818 Acked Read 8 Done Done Sent false
3342 1819 Acked Read 7 Done Done Sent false
3343 1820 Acked Write 5 Done Done Sent false
3344 1821 Acked Read 1 Done Done Sent false
3345 1822 Acked Write 8 Done Done Sent false
3346 1823 Acked Write 5 Done Done Sent false
3347 1824 Acked Flush 0 Done Done Sent false
3348 1825 Acked Write 8 Done Done Sent false
3349 1826 Acked Write 7 Done Done Sent false
3350 1827 Acked Read 6 Done Done Sent false
3351 1828 Acked Read 10 Done Done Sent false
3352 1829 Acked Read 4 Done Done Sent false
3353 1830 Acked Write 9 Done Done Sent false
3354 1831 Acked Write 2 Done Done Sent false
3355 1832 Acked Read 6 Done Done Sent false
3356 1833 Acked Read 8 Done Done Sent false
3357 1834 Acked Read 1 Done Done Sent false
3358 1835 Acked Write 6 Done Done Sent false
3359 1836 Acked Write 4 Done Done Sent false
3360 1837 Acked Read 5 Done Done Sent false
3361 1838 Acked Read 9 Done Done Sent false
3362 1839 Acked Flush 0 Done Done Sent false
3363 1840 Acked Write 10 Done Done Sent false
3364 1841 Acked Flush 0 Done Done Sent false
3365 1842 Acked Write 8 Done Done Sent false
3366 1843 Acked Read 3 Done Done Sent false
3367 1844 Acked Flush 0 Done Done Sent false
3368 1845 Acked Flush 0 Done Done Sent false
3369 1846 Acked Read 9 Done Done Sent false
3370 1847 Acked Write 5 Done Done Sent false
3371 1848 Acked Read 3 Done Done Sent false
3372 1849 Acked Flush 0 Done Done Sent false
3373 1850 Acked Read 1 Done Done Sent false
3374 1851 Acked Write 7 Done Done Sent false
3375 1852 Acked Write 6 Done Done Sent false
3376 1853 Acked Write 5 Done Done Sent false
3377 1854 Acked Write 9 Done Done Sent false
3378 1855 Acked Write 3 Done Done Sent false
3379 1856 Acked Write 3 Done Done Sent false
3380 1857 Acked Read 6 Done Done Sent false
3381 1858 Acked Read 3 Done Done Sent false
3382 1859 Acked Read 3 Done Done Sent false
3383 1860 Acked Read 10 Done Done Sent false
3384 1861 Acked Flush 0 Done Done Sent false
3385 1862 Acked Read 8 Done Done Sent false
3386 1863 Acked Flush 0 Done Done Sent false
3387 1864 Acked Read 6 Done Done Sent false
3388 1865 Acked Write 3 Done Done Sent false
3389 1866 Acked Read 1 Done Done Sent false
3390 1867 Acked Read 10 Done Done Sent false
3391 1868 Acked Write 8 Done Done Sent false
3392 1869 Acked Write 7 Done Done Sent false
3393 1870 Acked Write 5 Done Done Sent false
3394 1871 Acked Read 3 Done Done Sent false
3395 1872 Acked Write 2 Done Done Sent false
3396 1873 Acked Read 1 Done Done Sent false
3397 1874 Acked Read 5 Done Done Sent false
3398 1875 Acked Read 7 Done Done Sent false
3399 1876 Acked Read 4 Done Done Sent false
3400 1877 Acked Read 1 Done Done Sent false
3401 1878 Acked Write 6 Done Done Sent false
3402 1879 Acked Read 10 Done Done Sent false
3403 1880 Acked Read 5 Done Done Sent false
3404 1881 Acked Write 3 Done Done Sent false
3405 1882 Acked Read 10 Done Done Sent false
3406 1883 Acked Read 8 Done Done Sent false
3407 1884 Acked Read 2 Done Done Sent false
3408 1885 Acked Read 2 Done Done Sent false
3409 1886 Acked Write 7 Done Done Sent false
3410 1887 Acked Read 3 Done Done Sent false
3411 1888 Acked Write 9 Done Done Sent false
3412 1889 Acked Read 2 Done Done Sent false
3413 1890 Acked Read 6 Done Done Sent false
3414 1891 Acked Write 7 Done Done Sent false
3415 1892 Acked Write 10 Done Done Sent false
3416 1893 Acked Flush 0 Done Done Sent false
3417 1894 Acked Flush 0 Done Done Sent false
3418 1895 Acked Write 6 Done Done Sent false
3419 1896 Acked Read 7 Done Done Sent false
3420 1897 Acked Read 8 Done Done Sent false
3421 1898 Acked Read 2 Done Done Sent false
3422 1899 Acked Read 3 Done Done Sent false
3423 1900 Acked Write 9 Done Done Sent false
3424 1901 Acked Read 5 Done Done Sent false
3425 1902 Acked Read 8 Done Done Sent false
3426 1903 Acked Write 1 Done Done Sent false
3427 STATES DS:0 DS:1 DS:2 TOTAL
3428 Sent 0 0 301 301
3429 Done 301 301 0 602
3430 Skipped 0 0 0 0
3431 Error 0 0 0 0
3432 Last Flush: Some(JobId(1894)) Some(JobId(1894)) Some(JobId(1602))
3433 Downstairs last five completed: 1602 1601 1600 1599 1598
3434 Upstairs last five completed: 1903 1902 1901 1900 1899
3435 CLIENT: Up:0 ds:301 act:2
34362025-02-25T20:47:22.355ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3437 [8830] got start action
3438 [8830] I am going to restart
3439 Starting downstairs at port 8830
3440 Make output file at "/tmp/dsc/downstairs-8830.txt"
3441 [8830][2] reports Starting
3442 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1209)
3443 [8830][2] reports Running
34442025-02-25T20:47:23.129ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34452025-02-25T20:47:24.131ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34462025-02-25T20:47:25.133ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3447 ----------------------------------------------------------------
3448 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3449 States: Active Active Offline
3450 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3451 1603 Acked Read 10 Done Done Sent false
3452 1604 Acked Read 5 Done Done Sent false
3453 1605 Acked Read 1 Done Done Sent false
3454 1606 Acked Read 8 Done Done Sent false
3455 1607 Acked Read 4 Done Done Sent false
3456 1608 Acked Write 1 Done Done Sent false
3457 1609 Acked Read 9 Done Done Sent false
3458 1610 Acked Read 7 Done Done Sent false
3459 1611 Acked Read 1 Done Done Sent false
3460 1612 Acked Read 10 Done Done Sent false
3461 1613 Acked Read 6 Done Done Sent false
3462 1614 Acked Read 4 Done Done Sent false
3463 1615 Acked Write 8 Done Done Sent false
3464 1616 Acked Read 4 Done Done Sent false
3465 1617 Acked Read 4 Done Done Sent false
3466 1618 Acked Read 5 Done Done Sent false
3467 1619 Acked Read 6 Done Done Sent false
3468 1620 Acked Write 2 Done Done Sent false
3469 1621 Acked Read 10 Done Done Sent false
3470 1622 Acked Write 3 Done Done Sent false
3471 1623 Acked Write 5 Done Done Sent false
3472 1624 Acked Read 5 Done Done Sent false
3473 1625 Acked Write 7 Done Done Sent false
3474 1626 Acked Read 5 Done Done Sent false
3475 1627 Acked Read 2 Done Done Sent false
3476 1628 Acked Write 4 Done Done Sent false
3477 1629 Acked Read 4 Done Done Sent false
3478 1630 Acked Read 1 Done Done Sent false
3479 1631 Acked Write 3 Done Done Sent false
3480 1632 Acked Flush 0 Done Done Sent false
3481 1633 Acked Read 10 Done Done Sent false
3482 1634 Acked Read 5 Done Done Sent false
3483 1635 Acked Read 4 Done Done Sent false
3484 1636 Acked Read 6 Done Done Sent false
3485 1637 Acked Read 9 Done Done Sent false
3486 1638 Acked Flush 0 Done Done Sent false
3487 1639 Acked Read 6 Done Done Sent false
3488 1640 Acked Read 2 Done Done Sent false
3489 1641 Acked Read 8 Done Done Sent false
3490 1642 Acked Write 6 Done Done Sent false
3491 1643 Acked Write 4 Done Done Sent false
3492 1644 Acked Write 3 Done Done Sent false
3493 1645 Acked Read 7 Done Done Sent false
3494 1646 Acked Read 8 Done Done Sent false
3495 1647 Acked Write 6 Done Done Sent false
3496 1648 Acked Write 5 Done Done Sent false
3497 1649 Acked Write 7 Done Done Sent false
3498 1650 Acked Write 7 Done Done Sent false
3499 1651 Acked Read 7 Done Done Sent false
3500 1652 Acked Write 3 Done Done Sent false
3501 1653 Acked Read 5 Done Done Sent false
3502 1654 Acked Write 8 Done Done Sent false
3503 1655 Acked Read 1 Done Done Sent false
3504 1656 Acked Write 7 Done Done Sent false
3505 1657 Acked Read 3 Done Done Sent false
3506 1658 Acked Read 6 Done Done Sent false
3507 1659 Acked Flush 0 Done Done Sent false
3508 1660 Acked Write 6 Done Done Sent false
3509 1661 Acked Read 2 Done Done Sent false
3510 1662 Acked Read 8 Done Done Sent false
3511 1663 Acked Read 6 Done Done Sent false
3512 1664 Acked Write 8 Done Done Sent false
3513 1665 Acked Read 6 Done Done Sent false
3514 1666 Acked Flush 0 Done Done Sent false
3515 1667 Acked Write 10 Done Done Sent false
3516 1668 Acked Write 6 Done Done Sent false
3517 1669 Acked Read 1 Done Done Sent false
3518 1670 Acked Write 9 Done Done Sent false
3519 1671 Acked Read 8 Done Done Sent false
3520 1672 Acked Read 2 Done Done Sent false
3521 1673 Acked Read 10 Done Done Sent false
3522 1674 Acked Read 3 Done Done Sent false
3523 1675 Acked Write 10 Done Done Sent false
3524 1676 Acked Read 3 Done Done Sent false
3525 1677 Acked Read 2 Done Done Sent false
3526 1678 Acked Read 7 Done Done Sent false
3527 1679 Acked Read 6 Done Done Sent false
3528 1680 Acked Read 3 Done Done Sent false
3529 1681 Acked Write 7 Done Done Sent false
3530 1682 Acked Write 5 Done Done Sent false
3531 1683 Acked Write 7 Done Done Sent false
3532 1684 Acked Read 7 Done Done Sent false
3533 1685 Acked Read 4 Done Done Sent false
3534 1686 Acked Flush 0 Done Done Sent false
3535 1687 Acked Write 4 Done Done Sent false
3536 1688 Acked Flush 0 Done Done Sent false
3537 1689 Acked Read 8 Done Done Sent false
3538 1690 Acked Write 1 Done Done Sent false
3539 1691 Acked Write 7 Done Done Sent false
3540 1692 Acked Read 2 Done Done Sent false
3541 1693 Acked Write 1 Done Done Sent false
3542 1694 Acked Write 7 Done Done Sent false
3543 1695 Acked Read 5 Done Done Sent false
3544 1696 Acked Read 5 Done Done Sent false
3545 1697 Acked Read 4 Done Done Sent false
3546 1698 Acked Read 1 Done Done Sent false
3547 1699 Acked Read 6 Done Done Sent false
3548 1700 Acked Flush 0 Done Done Sent false
3549 1701 Acked Read 3 Done Done Sent false
3550 1702 Acked Write 7 Done Done Sent false
3551 1703 Acked Write 1 Done Done Sent false
3552 1704 Acked Read 2 Done Done Sent false
3553 1705 Acked Write 7 Done Done Sent false
3554 1706 Acked Write 3 Done Done Sent false
3555 1707 Acked Write 8 Done Done Sent false
3556 1708 Acked Flush 0 Done Done Sent false
3557 1709 Acked Read 5 Done Done Sent false
3558 1710 Acked Write 8 Done Done Sent false
3559 1711 Acked Read 10 Done Done Sent false
3560 1712 Acked Write 5 Done Done Sent false
3561 1713 Acked Flush 0 Done Done Sent false
3562 1714 Acked Write 1 Done Done Sent false
3563 1715 Acked Read 10 Done Done Sent false
3564 1716 Acked Write 10 Done Done Sent false
3565 1717 Acked Write 1 Done Done Sent false
3566 1718 Acked Read 8 Done Done Sent false
3567 1719 Acked Flush 0 Done Done Sent false
3568 1720 Acked Flush 0 Done Done Sent false
3569 1721 Acked Read 7 Done Done Sent false
3570 1722 Acked Write 5 Done Done Sent false
3571 1723 Acked Write 6 Done Done Sent false
3572 1724 Acked Flush 0 Done Done Sent false
3573 1725 Acked Read 4 Done Done Sent false
3574 1726 Acked Write 8 Done Done Sent false
3575 1727 Acked Write 5 Done Done Sent false
3576 1728 Acked Read 9 Done Done Sent false
3577 1729 Acked Read 7 Done Done Sent false
3578 1730 Acked Write 1 Done Done Sent false
3579 1731 Acked Read 5 Done Done Sent false
3580 1732 Acked Read 2 Done Done Sent false
3581 1733 Acked Write 8 Done Done Sent false
3582 1734 Acked Write 7 Done Done Sent false
3583 1735 Acked Read 4 Done Done Sent false
3584 1736 Acked Read 8 Done Done Sent false
3585 1737 Acked Read 7 Done Done Sent false
3586 1738 Acked Write 8 Done Done Sent false
3587 1739 Acked Read 6 Done Done Sent false
3588 1740 Acked Read 3 Done Done Sent false
3589 1741 Acked Read 8 Done Done Sent false
3590 1742 Acked Write 1 Done Done Sent false
3591 1743 Acked Read 10 Done Done Sent false
3592 1744 Acked Read 10 Done Done Sent false
3593 1745 Acked Write 9 Done Done Sent false
3594 1746 Acked Write 1 Done Done Sent false
3595 1747 Acked Write 8 Done Done Sent false
3596 1748 Acked Write 7 Done Done Sent false
3597 1749 Acked Flush 0 Done Done Sent false
3598 1750 Acked Write 7 Done Done Sent false
3599 1751 Acked Read 2 Done Done Sent false
3600 1752 Acked Write 6 Done Done Sent false
3601 1753 Acked Write 5 Done Done Sent false
3602 1754 Acked Write 1 Done Done Sent false
3603 1755 Acked Write 8 Done Done Sent false
3604 1756 Acked Flush 0 Done Done Sent false
3605 1757 Acked Write 4 Done Done Sent false
3606 1758 Acked Read 8 Done Done Sent false
3607 1759 Acked Write 1 Done Done Sent false
3608 1760 Acked Flush 0 Done Done Sent false
3609 1761 Acked Read 2 Done Done Sent false
3610 1762 Acked Read 10 Done Done Sent false
3611 1763 Acked Write 3 Done Done Sent false
3612 1764 Acked Write 3 Done Done Sent false
3613 1765 Acked Write 3 Done Done Sent false
3614 1766 Acked Write 9 Done Done Sent false
3615 1767 Acked Read 10 Done Done Sent false
3616 1768 Acked Write 2 Done Done Sent false
3617 1769 Acked Write 1 Done Done Sent false
3618 1770 Acked Read 1 Done Done Sent false
3619 1771 Acked Read 4 Done Done Sent false
3620 1772 Acked Write 3 Done Done Sent false
3621 1773 Acked Flush 0 Done Done Sent false
3622 1774 Acked Read 3 Done Done Sent false
3623 1775 Acked Write 3 Done Done Sent false
3624 1776 Acked Write 5 Done Done Sent false
3625 1777 Acked Read 8 Done Done Sent false
3626 1778 Acked Write 3 Done Done Sent false
3627 1779 Acked Read 10 Done Done Sent false
3628 1780 Acked Write 5 Done Done Sent false
3629 1781 Acked Write 2 Done Done Sent false
3630 1782 Acked Write 8 Done Done Sent false
3631 1783 Acked Flush 0 Done Done Sent false
3632 1784 Acked Read 2 Done Done Sent false
3633 1785 Acked Write 3 Done Done Sent false
3634 1786 Acked Write 6 Done Done Sent false
3635 1787 Acked Write 9 Done Done Sent false
3636 1788 Acked Write 2 Done Done Sent false
3637 1789 Acked Flush 0 Done Done Sent false
3638 1790 Acked Read 4 Done Done Sent false
3639 1791 Acked Write 10 Done Done Sent false
3640 1792 Acked Flush 0 Done Done Sent false
3641 1793 Acked Write 4 Done Done Sent false
3642 1794 Acked Read 1 Done Done Sent false
3643 1795 Acked Flush 0 Done Done Sent false
3644 1796 Acked Write 9 Done Done Sent false
3645 1797 Acked Read 1 Done Done Sent false
3646 1798 Acked Flush 0 Done Done Sent false
3647 1799 Acked Read 9 Done Done Sent false
3648 1800 Acked Write 2 Done Done Sent false
3649 1801 Acked Read 2 Done Done Sent false
3650 1802 Acked Read 9 Done Done Sent false
3651 1803 Acked Read 10 Done Done Sent false
3652 1804 Acked Read 3 Done Done Sent false
3653 1805 Acked Write 6 Done Done Sent false
3654 1806 Acked Read 2 Done Done Sent false
3655 1807 Acked Write 4 Done Done Sent false
3656 1808 Acked Write 3 Done Done Sent false
3657 1809 Acked Read 10 Done Done Sent false
3658 1810 Acked Flush 0 Done Done Sent false
3659 1811 Acked Write 7 Done Done Sent false
3660 1812 Acked Read 4 Done Done Sent false
3661 1813 Acked Flush 0 Done Done Sent false
3662 1814 Acked Flush 0 Done Done Sent false
3663 1815 Acked Read 1 Done Done Sent false
3664 1816 Acked Read 4 Done Done Sent false
3665 1817 Acked Write 6 Done Done Sent false
3666 1818 Acked Read 8 Done Done Sent false
3667 1819 Acked Read 7 Done Done Sent false
3668 1820 Acked Write 5 Done Done Sent false
3669 1821 Acked Read 1 Done Done Sent false
3670 1822 Acked Write 8 Done Done Sent false
3671 1823 Acked Write 5 Done Done Sent false
3672 1824 Acked Flush 0 Done Done Sent false
3673 1825 Acked Write 8 Done Done Sent false
3674 1826 Acked Write 7 Done Done Sent false
3675 1827 Acked Read 6 Done Done Sent false
3676 1828 Acked Read 10 Done Done Sent false
3677 1829 Acked Read 4 Done Done Sent false
3678 1830 Acked Write 9 Done Done Sent false
3679 1831 Acked Write 2 Done Done Sent false
3680 1832 Acked Read 6 Done Done Sent false
3681 1833 Acked Read 8 Done Done Sent false
3682 1834 Acked Read 1 Done Done Sent false
3683 1835 Acked Write 6 Done Done Sent false
3684 1836 Acked Write 4 Done Done Sent false
3685 1837 Acked Read 5 Done Done Sent false
3686 1838 Acked Read 9 Done Done Sent false
3687 1839 Acked Flush 0 Done Done Sent false
3688 1840 Acked Write 10 Done Done Sent false
3689 1841 Acked Flush 0 Done Done Sent false
3690 1842 Acked Write 8 Done Done Sent false
3691 1843 Acked Read 3 Done Done Sent false
3692 1844 Acked Flush 0 Done Done Sent false
3693 1845 Acked Flush 0 Done Done Sent false
3694 1846 Acked Read 9 Done Done Sent false
3695 1847 Acked Write 5 Done Done Sent false
3696 1848 Acked Read 3 Done Done Sent false
3697 1849 Acked Flush 0 Done Done Sent false
3698 1850 Acked Read 1 Done Done Sent false
3699 1851 Acked Write 7 Done Done Sent false
3700 1852 Acked Write 6 Done Done Sent false
3701 1853 Acked Write 5 Done Done Sent false
3702 1854 Acked Write 9 Done Done Sent false
3703 1855 Acked Write 3 Done Done Sent false
3704 1856 Acked Write 3 Done Done Sent false
3705 1857 Acked Read 6 Done Done Sent false
3706 1858 Acked Read 3 Done Done Sent false
3707 1859 Acked Read 3 Done Done Sent false
3708 1860 Acked Read 10 Done Done Sent false
3709 1861 Acked Flush 0 Done Done Sent false
3710 1862 Acked Read 8 Done Done Sent false
3711 1863 Acked Flush 0 Done Done Sent false
3712 1864 Acked Read 6 Done Done Sent false
3713 1865 Acked Write 3 Done Done Sent false
3714 1866 Acked Read 1 Done Done Sent false
3715 1867 Acked Read 10 Done Done Sent false
3716 1868 Acked Write 8 Done Done Sent false
3717 1869 Acked Write 7 Done Done Sent false
3718 1870 Acked Write 5 Done Done Sent false
3719 1871 Acked Read 3 Done Done Sent false
3720 1872 Acked Write 2 Done Done Sent false
3721 1873 Acked Read 1 Done Done Sent false
3722 1874 Acked Read 5 Done Done Sent false
3723 1875 Acked Read 7 Done Done Sent false
3724 1876 Acked Read 4 Done Done Sent false
3725 1877 Acked Read 1 Done Done Sent false
3726 1878 Acked Write 6 Done Done Sent false
3727 1879 Acked Read 10 Done Done Sent false
3728 1880 Acked Read 5 Done Done Sent false
3729 1881 Acked Write 3 Done Done Sent false
3730 1882 Acked Read 10 Done Done Sent false
3731 1883 Acked Read 8 Done Done Sent false
3732 1884 Acked Read 2 Done Done Sent false
3733 1885 Acked Read 2 Done Done Sent false
3734 1886 Acked Write 7 Done Done Sent false
3735 1887 Acked Read 3 Done Done Sent false
3736 1888 Acked Write 9 Done Done Sent false
3737 1889 Acked Read 2 Done Done Sent false
3738 1890 Acked Read 6 Done Done Sent false
3739 1891 Acked Write 7 Done Done Sent false
3740 1892 Acked Write 10 Done Done Sent false
3741 1893 Acked Flush 0 Done Done Sent false
3742 1894 Acked Flush 0 Done Done Sent false
3743 1895 Acked Write 6 Done Done Sent false
3744 1896 Acked Read 7 Done Done Sent false
3745 1897 Acked Read 8 Done Done Sent false
3746 1898 Acked Read 2 Done Done Sent false
3747 1899 Acked Read 3 Done Done Sent false
3748 1900 Acked Write 9 Done Done Sent false
3749 1901 Acked Read 5 Done Done Sent false
3750 1902 Acked Read 8 Done Done Sent false
3751 1903 Acked Write 1 Done Done Sent false
3752 STATES DS:0 DS:1 DS:2 TOTAL
3753 Sent 0 0 301 301
3754 Done 301 301 0 602
3755 Skipped 0 0 0 0
3756 Error 0 0 0 0
3757 Last Flush: Some(JobId(1894)) Some(JobId(1894)) Some(JobId(1602))
3758 Downstairs last five completed: 1602 1601 1600 1599 1598
3759 Upstairs last five completed: 1903 1902 1901 1900 1899
3760 CLIENT: Up:0 ds:301 act:2
37612025-02-25T20:47:26.135ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37622025-02-25T20:47:27.356ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37632025-02-25T20:47:28.139ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37642025-02-25T20:47:29.110ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37652025-02-25T20:47:29.110ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36804) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37662025-02-25T20:47:29.111ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37672025-02-25T20:47:29.111ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37682025-02-25T20:47:29.111ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37692025-02-25T20:47:29.111ZINFOcrucible: send last flush ID to this DS: Some(JobId(1602)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37702025-02-25T20:47:29.111ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37712025-02-25T20:47:29.111ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1602)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
37722025-02-25T20:47:29.112ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1602)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3773 ----------------------------------------------------------------
3774 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
3775 States: Active Active Active
3776 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3777 1895 Acked Write 6 Done Done Done true
3778 1896 Acked Read 7 Done Done Done true
3779 1897 Acked Read 8 Done Done Done true
3780 1898 Acked Read 2 Done Done Done true
3781 1899 Acked Read 3 Done Done Done true
3782 1900 Acked Write 9 Done Done Done true
3783 1901 Acked Read 5 Done Done Done true
3784 1902 Acked Read 8 Done Done Done true
3785 1903 Acked Write 1 Done Done Done true
3786 STATES DS:0 DS:1 DS:2 TOTAL
3787 Sent 0 0 0 0
3788 Done 9 9 9 27
3789 Skipped 0 0 0 0
3790 Error 0 0 0 0
3791 Last Flush: Some(JobId(1894)) Some(JobId(1894)) Some(JobId(1894))
3792 Downstairs last five completed: 1894 1893 1892 1891 1890
3793 Upstairs last five completed: 1903 1902 1901 1900 1899
3794 CLIENT: Up:0 ds:9 act:3
3795 ----------------------------------------------------------------
3796 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3797 Downstairs last five completed: 1904 1903 1902 1901 1900
3798 Upstairs last five completed: 1904 1903 1902 1901 1900
3799 CLIENT: Up:0 ds:0 act:3
3800 Replay: All jobs finished
3801 Main task has work to do, go find it
3802 got dsc Stop(2)
3803 stop 2
3804 Feb 25 20:47:33.362 INFO request completed, latency_us: 643, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
3805 , req_id: b8bd24fc-d219-41ce-a44b-2dad63bee685, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
38062025-02-25T20:47:33.363ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
38072025-02-25T20:47:33.363ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
38082025-02-25T20:47:33.363ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3809 Feb 25 20:47:33.363 INFO request completed, latency_us: 146, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1d3d2373-7dd0-4518-8933-d4b98b0469d7, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
38102025-02-25T20:47:33.363ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
3811 [8830] Exited with: ExitStatus(unix_wait_status(9))
3812 [8830][2] reports Exit
3813 Feb 25 20:47:37.365 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/2, method: GET, req_id: a43ff1b5-64c3-4c56-8155-2e3eee4e7818, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
3814 001/300 Write block 3933 len 8192 data: 2 2
3815 002/300 Write block 1804 len 16384 data: 2 2 3 3
3816 003/300 Read block 2704 len 24576
3817 004/300 Read block 3880 len 8192
3818 005/300 Read block 4895 len 24576
3819 006/300 Read block 3007 len 8192
3820 007/300 Read block 4820 len 4096
3821 008/300 Write block 4896 len 24576 data: 3 3 3 2 2 2
3822 009/300 Read block 1103 len 24576
3823 010/300 Flush
3824 011/300 Write block 2298 len 8192 data: 2 2
3825 012/300 Read block 690 len 36864
3826 013/300 Read block 3915 len 8192
3827 014/300 Read block 1715 len 36864
3828 015/300 Read block 1199 len 4096
3829 016/300 Read block 1711 len 16384
3830 017/300 Write block 4161 len 24576 data: 2 2 3 3 4 4
3831 018/300 Read block 1617 len 24576
3832 019/300 Read block 2257 len 28672
3833 020/300 Read block 2446 len 24576
3834 021/300 Write block 672 len 20480 data: 2 2 3 4 4
3835 022/300 Read block 706 len 12288
3836 023/300 Read block 1969 len 40960
3837 024/300 Write block 973 len 24576 data: 3 3 4 3 3 3
3838 025/300 Read block 1969 len 24576
3839 026/300 Read block 4303 len 32768
3840 027/300 Write block 2647 len 20480 data: 3 3 3 2 2
3841 028/300 Read block 4197 len 28672
3842 029/300 Write block 3166 len 4096 data: 3
3843 030/300 Read block 3084 len 40960
3844 031/300 Write block 931 len 16384 data: 2 2 3 3
3845 032/300 Read block 3858 len 12288
3846 033/300 Write block 1825 len 20480 data: 2 2 2 2 2
3847 034/300 Read block 4321 len 24576
3848 035/300 Write block 1456 len 28672 data: 2 2 2 2 2 2 3
3849 036/300 Read block 1430 len 8192
3850 037/300 Read block 886 len 8192
3851 038/300 Read block 821 len 32768
3852 039/300 Write block 3276 len 24576 data: 3 3 3 2 2 2
3853 040/300 Read block 1723 len 36864
3854 041/300 Write block 4614 len 20480 data: 2 2 2 3 2
3855 042/300 Read block 4059 len 32768
3856 043/300 Read block 815 len 24576
3857 044/300 Flush
3858 045/300 Read block 1198 len 28672
3859 046/300 Read block 1750 len 24576
3860 047/300 Write block 3966 len 24576 data: 2 2 2 2 2 2
3861 048/300 Read block 1311 len 8192
3862 049/300 Write block 2919 len 24576 data: 2 2 2 2 3 3
3863 050/300 Read block 1177 len 40960
3864 051/300 Write block 1325 len 40960 data: 3 3 2 2 2 2 2 2 2 2
3865 052/300 Read block 4627 len 12288
3866 053/300 Read block 2920 len 4096
3867 054/300 Write block 1775 len 32768 data: 2 2 2 2 3 2 2 2
3868 055/300 Flush
3869 056/300 Write block 3817 len 40960 data: 2 2 2 2 2 2 2 2 3 3
3870 057/300 Write block 2705 len 36864 data: 2 2 2 3 3 4 3 2 2
3871 058/300 Flush
3872 059/300 Write block 1128 len 16384 data: 3 3 2 2
3873 060/300 Flush
3874 061/300 Write block 1760 len 24576 data: 2 2 2 2 2 2
3875 062/300 Write block 2992 len 36864 data: 3 3 3 3 3 3 2 2 2
3876 063/300 Read block 1635 len 40960
3877 064/300 Write block 3841 len 24576 data: 2 2 2 2 2 2
3878 065/300 Read block 211 len 40960
3879 066/300 Write block 3251 len 12288 data: 2 2 2
3880 067/300 Read block 3700 len 8192
3881 068/300 Read block 3653 len 12288
3882 069/300 Read block 748 len 20480
3883 070/300 Read block 1841 len 12288
3884 071/300 Read block 2691 len 40960
3885 072/300 Read block 2109 len 16384
3886 073/300 Flush
3887 074/300 Write block 4785 len 16384 data: 3 3 3 3
3888 075/300 Read block 1120 len 32768
3889 076/300 Read block 535 len 4096
3890 077/300 Write block 4296 len 36864 data: 2 2 2 2 2 2 2 2 2
3891 078/300 Write block 1120 len 32768 data: 2 2 2 2 2 2 3 3
3892 079/300 Write block 3789 len 12288 data: 2 2 2
3893 080/300 Read block 2253 len 4096
3894 081/300 Write block 3112 len 8192 data: 2 2
3895 082/300 Write block 1255 len 16384 data: 2 2 2 2
3896 083/300 Read block 3819 len 28672
3897 084/300 Read block 4116 len 12288
3898 085/300 Read block 290 len 16384
3899 086/300 Read block 1889 len 16384
3900 087/300 Read block 2807 len 4096
3901 088/300 Read block 1999 len 36864
3902 089/300 Write block 2681 len 16384 data: 4 4 4 4
3903 090/300 Read block 4355 len 32768
3904 091/300 Flush
3905 092/300 Read block 3305 len 8192
3906 093/300 Write block 954 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3907 094/300 Read block 338 len 24576
3908 095/300 Write block 3560 len 12288 data: 3 4 3
3909 096/300 Read block 4342 len 40960
3910 097/300 Write block 4058 len 8192 data: 3 3
3911 098/300 Write block 2258 len 8192 data: 3 3
3912 099/300 Read block 2041 len 40960
3913 100/300 Read block 1749 len 16384
3914 101/300 Write block 3422 len 32768 data: 3 3 2 2 2 2 2 2
3915 102/300 Read block 2517 len 28672
3916 103/300 Read block 3890 len 12288
3917 104/300 Write block 35 len 12288 data: 2 2 2
3918 105/300 Write block 3049 len 16384 data: 2 2 2 2
3919 106/300 Read block 3230 len 20480
3920 107/300 Write block 2769 len 8192 data: 3 3
3921 108/300 Read block 4495 len 8192
3922 109/300 Read block 2486 len 4096
3923 110/300 Write block 199 len 32768 data: 2 2 2 2 2 2 3 3
3924 111/300 Write block 2673 len 12288 data: 2 2 2
3925 112/300 Read block 3409 len 36864
3926 113/300 Flush
3927 114/300 Write block 710 len 16384 data: 2 3 3 3
3928 115/300 Read block 1260 len 12288
3929 116/300 Write block 126 len 20480 data: 2 2 2 2 2
3930 117/300 Read block 4036 len 32768
3931 118/300 Write block 4992 len 28672 data: 3 3 2 2 2 2 2
3932 119/300 Read block 1047 len 12288
3933 120/300 Write block 4964 len 32768 data: 3 3 3 3 3 2 2 2
3934 121/300 Write block 882 len 16384 data: 2 2 3 3
3935 122/300 Read block 3646 len 40960
3936 123/300 Read block 3343 len 40960
3937 124/300 Write block 4422 len 4096 data: 4
3938 125/300 Write block 3797 len 16384 data: 2 2 2 2
3939 126/300 Read block 1374 len 20480
3940 127/300 Write block 2756 len 4096 data: 2
3941 128/300 Read block 2803 len 20480
3942 129/300 Flush
3943 130/300 Read block 4434 len 40960
3944 131/300 Read block 4199 len 16384
3945 132/300 Write block 1948 len 32768 data: 3 3 3 3 3 3 4 3
3946 133/300 Read block 3678 len 36864
3947 134/300 Read block 253 len 36864
3948 135/300 Read block 1162 len 4096
3949 136/300 Write block 303 len 4096 data: 2
3950 137/300 Read block 2634 len 12288
3951 138/300 Read block 74 len 12288
3952 139/300 Read block 2401 len 40960
3953 140/300 Write block 4411 len 32768 data: 2 2 2 2 2 2 2 3
3954 141/300 Write block 4406 len 40960 data: 3 2 2 2 2 3 3 3 3 3
3955 142/300 Write block 2797 len 4096 data: 2
3956 143/300 Read block 4138 len 24576
3957 144/300 Write block 4526 len 32768 data: 2 2 2 2 2 2 2 2
3958 145/300 Write block 4661 len 40960 data: 2 2 2 2 3 3 3 3 3 3
3959 146/300 Write block 74 len 40960 data: 2 2 2 2 2 2 2 3 3 3
3960 147/300 Flush
3961 148/300 Read block 1490 len 4096
3962 149/300 Read block 922 len 4096
3963 150/300 Read block 678 len 4096
3964 151/300 Read block 970 len 32768
3965 152/300 Read block 4775 len 4096
3966 153/300 Write block 4300 len 4096 data: 3
3967 154/300 Read block 2870 len 12288
3968 155/300 Read block 4324 len 36864
3969 156/300 Write block 4167 len 40960 data: 4 4 3 3 3 3 3 2 2 2
3970 157/300 Write block 1531 len 16384 data: 4 4 4 4
3971 158/300 Write block 2903 len 40960 data: 2 2 2 3 3 3 2 2 2 2
3972 159/300 Write block 3315 len 40960 data: 2 2 2 2 2 2 2 2 3 3
3973 160/300 Write block 3496 len 12288 data: 3 3 2
3974 161/300 Flush
3975 162/300 Read block 1989 len 16384
3976 163/300 Write block 3769 len 32768 data: 2 2 2 3 3 3 3 3
3977 164/300 Write block 842 len 28672 data: 2 2 2 2 2 2 2
3978 165/300 Write block 1208 len 12288 data: 3 3 3
3979 166/300 Read block 2795 len 20480
3980 167/300 Write block 770 len 16384 data: 2 2 2 2
3981 168/300 Read block 3079 len 12288
3982 169/300 Read block 1287 len 40960
3983 170/300 Read block 3464 len 40960
3984 171/300 Read block 235 len 20480
3985 172/300 Read block 4562 len 4096
3986 173/300 Flush
3987 174/300 Read block 2032 len 24576
3988 175/300 Read block 2839 len 8192
3989 176/300 Read block 4665 len 16384
3990 177/300 Write block 1166 len 20480 data: 3 3 3 3 2
3991 178/300 Write block 3479 len 12288 data: 2 2 2
3992 179/300 Write block 1577 len 36864 data: 2 2 2 3 2 3 4 4 3
3993 180/300 Read block 228 len 12288
3994 181/300 Write block 3912 len 8192 data: 2 2
3995 182/300 Read block 4335 len 24576
3996 183/300 Flush
3997 184/300 Read block 2102 len 32768
3998 185/300 Write block 1073 len 16384 data: 2 2 2 2
3999 186/300 Read block 1858 len 28672
4000 187/300 Flush
4001 188/300 Read block 901 len 4096
4002 189/300 Read block 3481 len 16384
4003 190/300 Write block 297 len 4096 data: 2
4004 191/300 Read block 298 len 12288
4005 192/300 Write block 4524 len 20480 data: 2 2 3 3 3
4006 193/300 Write block 2447 len 16384 data: 2 2 2 3
4007 194/300 Write block 3563 len 8192 data: 3 3
4008 195/300 Read block 4390 len 12288
4009 196/300 Read block 2808 len 16384
4010 197/300 Read block 137 len 40960
4011 198/300 Flush
4012 199/300 Read block 4771 len 4096
4013 200/300 Write block 4424 len 40960 data: 3 3 4 4 4 4 4 4 2 2
4014 201/300 Read block 2070 len 24576
4015 202/300 Read block 3260 len 28672
4016 203/300 Flush
4017 204/300 Read block 50 len 36864
4018 205/300 Read block 1806 len 16384
4019 206/300 Read block 2767 len 40960
4020 207/300 Read block 1022 len 40960
4021 208/300 Write block 4885 len 4096 data: 2
4022 209/300 Read block 2015 len 36864
4023 210/300 Read block 19 len 24576
4024 211/300 Write block 3164 len 24576 data: 2 3 4 2 2 2
4025 212/300 Write block 4077 len 24576 data: 2 2 2 2 2 2
4026 213/300 Write block 1875 len 40960 data: 4 4 4 4 4 4 2 2 2 2
4027 214/300 Read block 2218 len 20480
4028 215/300 Read block 1209 len 12288
4029 216/300 Write block 842 len 28672 data: 3 3 3 3 3 3 3
4030 217/300 Write block 3250 len 4096 data: 2
4031 218/300 Write block 1385 len 4096 data: 2
4032 219/300 Write block 2347 len 20480 data: 2 3 3 3 3
4033 220/300 Read block 2764 len 36864
4034 221/300 Write block 3248 len 32768 data: 2 2 3 3 3 3 2 2
4035 222/300 Read block 1426 len 40960
4036 223/300 Read block 3477 len 24576
4037 224/300 Write block 49 len 4096 data: 2
4038 225/300 Read block 777 len 36864
4039 226/300 Read block 4470 len 12288
4040 227/300 Write block 4683 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4041 228/300 Write block 3705 len 8192 data: 2 2
4042 229/300 Read block 2135 len 32768
4043 230/300 Read block 1770 len 8192
4044 231/300 Read block 814 len 24576
4045 232/300 Read block 769 len 24576
4046 233/300 Read block 1713 len 4096
4047 234/300 Read block 4269 len 36864
4048 235/300 Read block 1483 len 8192
4049 236/300 Write block 2770 len 20480 data: 4 3 3 3 3
4050 237/300 Write block 3281 len 16384 data: 3 2 2 2
4051 238/300 Read block 846 len 24576
4052 239/300 Read block 1675 len 36864
4053 240/300 Read block 2965 len 24576
4054 241/300 Write block 3348 len 12288 data: 2 2 2
4055 242/300 Read block 1876 len 8192
4056 243/300 Flush
4057 244/300 Flush
4058 245/300 Read block 3815 len 32768
4059 246/300 Read block 1637 len 16384
4060 247/300 Read block 2157 len 16384
4061 248/300 Flush
4062 249/300 Write block 2329 len 20480 data: 2 2 2 2 2
4063 250/300 Write block 3346 len 32768 data: 3 2 3 3 3 2 2 2
4064 251/300 Read block 1123 len 12288
4065 252/300 Read block 815 len 36864
4066 253/300 Write block 288 len 12288 data: 3 3 2
4067 254/300 Read block 226 len 8192
4068 255/300 Read block 1943 len 40960
4069 256/300 Read block 387 len 4096
4070 257/300 Write block 489 len 4096 data: 3
4071 258/300 Read block 4636 len 32768
4072 259/300 Read block 3230 len 4096
4073 260/300 Read block 3761 len 36864
4074 261/300 Read block 3256 len 20480
4075 262/300 Read block 65 len 40960
4076 263/300 Read block 4578 len 40960
4077 264/300 Read block 3730 len 8192
4078 265/300 Read block 4981 len 24576
4079 266/300 Write block 4751 len 36864 data: 2 2 2 2 2 2 2 2 2
4080 267/300 Read block 141 len 36864
4081 268/300 Write block 103 len 36864 data: 2 2 2 2 2 2 2 2 2
4082 269/300 Write block 4896 len 36864 data: 4 4 4 3 3 3 2 2 2
4083 270/300 Read block 1253 len 40960
4084 271/300 Read block 4282 len 20480
4085 272/300 Read block 2273 len 28672
4086 273/300 Flush
4087 274/300 Read block 1137 len 24576
4088 275/300 Read block 4444 len 12288
4089 276/300 Write block 4166 len 40960 data: 5 5 5 4 4 4 4 4 3 3
4090 277/300 Write block 919 len 24576 data: 2 2 2 2 2 2
4091 278/300 Read block 4304 len 4096
4092 279/300 Write block 1052 len 24576 data: 2 2 2 2 2 2
4093 280/300 Read block 2418 len 16384
4094 281/300 Read block 2101 len 40960
4095 282/300 Write block 4173 len 8192 data: 5 4
4096 283/300 Read block 4183 len 28672
4097 284/300 Write block 1361 len 4096 data: 3
4098 285/300 Write block 1067 len 28672 data: 2 2 2 2 2 2 3
4099 286/300 Write block 1019 len 40960 data: 3 2 2 2 2 2 3 2 2 2
4100 287/300 Write block 4915 len 4096 data: 3
4101 288/300 Read block 4692 len 4096
4102 289/300 Read block 3839 len 36864
4103 290/300 Write block 2449 len 36864 data: 3 4 3 3 4 4 4 2 2
4104 291/300 Read block 3386 len 8192
4105 292/300 Write block 2453 len 32768 data: 5 5 5 3 3 2 2 2
4106 293/300 Write block 1987 len 32768 data: 3 3 3 3 3 3 3 3
4107 294/300 Read block 2027 len 8192
4108 295/300 Read block 3349 len 36864
4109 296/300 Read block 1129 len 24576
4110 297/300 Read block 1713 len 8192
4111 298/300 Read block 1618 len 16384
4112 299/300 Write block 586 len 12288 data: 2 2 2
4113 300/300 Read block 4475 len 28672
4114 301/300 Read block 2407 len 4096
4115 Main task has work to do, go find it
4116 got dsc Start(2)
4117 start 2
4118 Feb 25 20:47:37.620 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8bbb732c-ea5b-4fa7-b8db-68626c28708e, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
4119 Replay: started 2, returned:Ok(())
4120 ----------------------------------------------------------------
4121 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4122 States: Active Active Offline
4123 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4124 1905 Acked Write 2 Done Done Sent false
4125 1906 Acked Write 4 Done Done Sent false
4126 1907 Acked Read 6 Done Done Sent false
4127 1908 Acked Read 2 Done Done Sent false
4128 1909 Acked Read 6 Done Done Sent false
4129 1910 Acked Read 2 Done Done Sent false
4130 1911 Acked Read 1 Done Done Sent false
4131 1912 Acked Write 6 Done Done Sent false
4132 1913 Acked Read 6 Done Done Sent false
4133 1914 Acked Flush 0 Done Done Sent false
4134 1915 Acked Write 2 Done Done Sent false
4135 1916 Acked Read 9 Done Done Sent false
4136 1917 Acked Read 2 Done Done Sent false
4137 1918 Acked Read 9 Done Done Sent false
4138 1919 Acked Read 1 Done Done Sent false
4139 1920 Acked Read 4 Done Done Sent false
4140 1921 Acked Write 6 Done Done Sent false
4141 1922 Acked Read 6 Done Done Sent false
4142 1923 Acked Read 7 Done Done Sent false
4143 1924 Acked Read 6 Done Done Sent false
4144 1925 Acked Write 5 Done Done Sent false
4145 1926 Acked Read 3 Done Done Sent false
4146 1927 Acked Read 10 Done Done Sent false
4147 1928 Acked Write 6 Done Done Sent false
4148 1929 Acked Read 6 Done Done Sent false
4149 1930 Acked Read 8 Done Done Sent false
4150 1931 Acked Write 5 Done Done Sent false
4151 1932 Acked Read 7 Done Done Sent false
4152 1933 Acked Write 1 Done Done Sent false
4153 1934 Acked Read 10 Done Done Sent false
4154 1935 Acked Write 4 Done Done Sent false
4155 1936 Acked Read 3 Done Done Sent false
4156 1937 Acked Write 5 Done Done Sent false
4157 1938 Acked Read 6 Done Done Sent false
4158 1939 Acked Write 7 Done Done Sent false
4159 1940 Acked Read 2 Done Done Sent false
4160 1941 Acked Read 2 Done Done Sent false
4161 1942 Acked Read 8 Done Done Sent false
4162 1943 Acked Write 6 Done Done Sent false
4163 1944 Acked Read 9 Done Done Sent false
4164 1945 Acked Write 5 Done Done Sent false
4165 1946 Acked Read 8 Done Done Sent false
4166 1947 Acked Read 6 Done Done Sent false
4167 1948 Acked Flush 0 Done Done Sent false
4168 1949 Acked Read 7 Done Done Sent false
4169 1950 Acked Read 6 Done Done Sent false
4170 1951 Acked Write 6 Done Done Sent false
4171 1952 Acked Read 2 Done Done Sent false
4172 1953 Acked Write 6 Done Done Sent false
4173 1954 Acked Read 10 Done Done Sent false
4174 1955 Acked Write 10 Done Done Sent false
4175 1956 Acked Read 3 Done Done Sent false
4176 1957 Acked Read 1 Done Done Sent false
4177 1958 Acked Write 8 Done Done Sent false
4178 1959 Acked Flush 0 Done Done Sent false
4179 1960 Acked Write 10 Done Done Sent false
4180 1961 Acked Write 9 Done Done Sent false
4181 1962 Acked Flush 0 Done Done Sent false
4182 1963 Acked Write 4 Done Done Sent false
4183 1964 Acked Flush 0 Done Done Sent false
4184 1965 Acked Write 6 Done Done Sent false
4185 1966 Acked Write 9 Done Done Sent false
4186 1967 Acked Read 10 Done Done Sent false
4187 1968 Acked Write 6 Done Done Sent false
4188 1969 Acked Read 10 Done Done Sent false
4189 1970 Acked Write 3 Done Done Sent false
4190 1971 Acked Read 2 Done Done Sent false
4191 1972 Acked Read 3 Done Done Sent false
4192 1973 Acked Read 5 Done Done Sent false
4193 1974 Acked Read 3 Done Done Sent false
4194 1975 Acked Read 10 Done Done Sent false
4195 1976 Acked Read 4 Done Done Sent false
4196 1977 Acked Flush 0 Done Done Sent false
4197 1978 Acked Write 4 Done Done Sent false
4198 1979 Acked Read 8 Done Done Sent false
4199 1980 Acked Read 1 Done Done Sent false
4200 1981 Acked Write 9 Done Done Sent false
4201 1982 Acked Write 8 Done Done Sent false
4202 1983 Acked Write 3 Done Done Sent false
4203 1984 Acked Read 1 Done Done Sent false
4204 1985 Acked Write 2 Done Done Sent false
4205 1986 Acked Write 4 Done Done Sent false
4206 1987 Acked Read 7 Done Done Sent false
4207 1988 Acked Read 3 Done Done Sent false
4208 1989 Acked Read 4 Done Done Sent false
4209 1990 Acked Read 4 Done Done Sent false
4210 1991 Acked Read 1 Done Done Sent false
4211 1992 Acked Read 9 Done Done Sent false
4212 1993 Acked Write 4 Done Done Sent false
4213 1994 Acked Read 8 Done Done Sent false
4214 1995 Acked Flush 0 Done Done Sent false
4215 1996 Acked Read 2 Done Done Sent false
4216 1997 Acked Write 10 Done Done Sent false
4217 1998 Acked Read 6 Done Done Sent false
4218 1999 Acked Write 3 Done Done Sent false
4219 2000 Acked Read 10 Done Done Sent false
4220 2001 Acked Write 2 Done Done Sent false
4221 2002 Acked Write 2 Done Done Sent false
4222 2003 Acked Read 10 Done Done Sent false
4223 2004 Acked Read 4 Done Done Sent false
4224 2005 Acked Write 8 Done Done Sent false
4225 2006 Acked Read 7 Done Done Sent false
4226 2007 Acked Read 3 Done Done Sent false
4227 2008 Acked Write 3 Done Done Sent false
4228 2009 Acked Write 4 Done Done Sent false
4229 2010 Acked Read 5 Done Done Sent false
4230 2011 Acked Write 2 Done Done Sent false
4231 2012 Acked Read 2 Done Done Sent false
4232 2013 Acked Read 1 Done Done Sent false
4233 2014 Acked Write 8 Done Done Sent false
4234 2015 Acked Write 3 Done Done Sent false
4235 2016 Acked Read 9 Done Done Sent false
4236 2017 Acked Flush 0 Done Done Sent false
4237 2018 Acked Write 4 Done Done Sent false
4238 2019 Acked Read 3 Done Done Sent false
4239 2020 Acked Write 5 Done Done Sent false
4240 2021 Acked Read 8 Done Done Sent false
4241 2022 Acked Write 7 Done Done Sent false
4242 2023 Acked Read 3 Done Done Sent false
4243 2024 Acked Write 8 Done Done Sent false
4244 2025 Acked Write 4 Done Done Sent false
4245 2026 Acked Read 10 Done Done Sent false
4246 2027 Acked Read 10 Done Done Sent false
4247 2028 Acked Write 1 Done Done Sent false
4248 2029 Acked Write 4 Done Done Sent false
4249 2030 Acked Read 5 Done Done Sent false
4250 2031 Acked Write 1 Done Done Sent false
4251 2032 Acked Read 5 Done Done Sent false
4252 2033 Acked Flush 0 Done Done Sent false
4253 2034 Acked Read 10 Done Done Sent false
4254 2035 Acked Read 4 Done Done Sent false
4255 2036 Acked Write 8 Done Done Sent false
4256 2037 Acked Read 9 Done Done Sent false
4257 2038 Acked Read 9 Done Done Sent false
4258 2039 Acked Read 1 Done Done Sent false
4259 2040 Acked Write 1 Done Done Sent false
4260 2041 Acked Read 3 Done Done Sent false
4261 2042 Acked Read 3 Done Done Sent false
4262 2043 Acked Read 10 Done Done Sent false
4263 2044 Acked Write 8 Done Done Sent false
4264 2045 Acked Write 10 Done Done Sent false
4265 2046 Acked Write 1 Done Done Sent false
4266 2047 Acked Read 6 Done Done Sent false
4267 2048 Acked Write 8 Done Done Sent false
4268 2049 Acked Write 10 Done Done Sent false
4269 2050 Acked Write 10 Done Done Sent false
4270 2051 Acked Flush 0 Done Done Sent false
4271 2052 Acked Read 1 Done Done Sent false
4272 2053 Acked Read 1 Done Done Sent false
4273 2054 Acked Read 1 Done Done Sent false
4274 2055 Acked Read 8 Done Done Sent false
4275 2056 Acked Read 1 Done Done Sent false
4276 2057 Acked Write 1 Done Done Sent false
4277 2058 Acked Read 3 Done Done Sent false
4278 2059 Acked Read 9 Done Done Sent false
4279 2060 Acked Write 10 Done Done Sent false
4280 2061 Acked Write 4 Done Done Sent false
4281 2062 Acked Write 10 Done Done Sent false
4282 2063 Acked Write 10 Done Done Sent false
4283 2064 Acked Write 3 Done Done Sent false
4284 2065 Acked Flush 0 Done Done Sent false
4285 2066 Acked Read 4 Done Done Sent false
4286 2067 Acked Write 8 Done Done Sent false
4287 2068 Acked Write 7 Done Done Sent false
4288 2069 Acked Write 3 Done Done Sent false
4289 2070 Acked Read 5 Done Done Sent false
4290 2071 Acked Write 4 Done Done Sent false
4291 2072 Acked Read 3 Done Done Sent false
4292 2073 Acked Read 10 Done Done Sent false
4293 2074 Acked Read 10 Done Done Sent false
4294 2075 Acked Read 5 Done Done Sent false
4295 2076 Acked Read 1 Done Done Sent false
4296 2077 Acked Flush 0 Done Done Sent false
4297 2078 Acked Read 6 Done Done Sent false
4298 2079 Acked Read 2 Done Done Sent false
4299 2080 Acked Read 4 Done Done Sent false
4300 2081 Acked Write 5 Done Done Sent false
4301 2082 Acked Write 3 Done Done Sent false
4302 2083 Acked Write 9 Done Done Sent false
4303 2084 Acked Read 3 Done Done Sent false
4304 2085 Acked Write 2 Done Done Sent false
4305 2086 Acked Read 6 Done Done Sent false
4306 2087 Acked Flush 0 Done Done Sent false
4307 2088 Acked Read 8 Done Done Sent false
4308 2089 Acked Write 4 Done Done Sent false
4309 2090 Acked Read 7 Done Done Sent false
4310 2091 Acked Flush 0 Done Done Sent false
4311 2092 Acked Read 1 Done Done Sent false
4312 2093 Acked Read 4 Done Done Sent false
4313 2094 Acked Write 1 Done Done Sent false
4314 2095 Acked Read 3 Done Done Sent false
4315 2096 Acked Write 5 Done Done Sent false
4316 2097 Acked Write 4 Done Done Sent false
4317 2098 Acked Write 2 Done Done Sent false
4318 2099 Acked Read 3 Done Done Sent false
4319 2100 Acked Read 4 Done Done Sent false
4320 2101 Acked Read 10 Done Done Sent false
4321 2102 Acked Flush 0 Done Done Sent false
4322 2103 Acked Read 1 Done Done Sent false
4323 2104 Acked Write 10 Done Done Sent false
4324 2105 Acked Read 6 Done Done Sent false
4325 2106 Acked Read 7 Done Done Sent false
4326 2107 Acked Flush 0 Done Done Sent false
4327 2108 Acked Read 9 Done Done Sent false
4328 2109 Acked Read 4 Done Done Sent false
4329 2110 Acked Read 10 Done Done Sent false
4330 2111 Acked Read 10 Done Done Sent false
4331 2112 Acked Write 1 Done Done Sent false
4332 2113 Acked Read 9 Done Done Sent false
4333 2114 Acked Read 6 Done Done Sent false
4334 2115 Acked Write 6 Done Done Sent false
4335 2116 Acked Write 6 Done Done Sent false
4336 2117 Acked Write 10 Done Done Sent false
4337 2118 Acked Read 5 Done Done Sent false
4338 2119 Acked Read 3 Done Done Sent false
4339 2120 Acked Write 7 Done Done Sent false
4340 2121 Acked Write 1 Done Done Sent false
4341 2122 Acked Write 1 Done Done Sent false
4342 2123 Acked Write 5 Done Done Sent false
4343 2124 Acked Read 9 Done Done Sent false
4344 2125 Acked Write 8 Done Done Sent false
4345 2126 Acked Read 10 Done Done Sent false
4346 2127 Acked Read 6 Done Done Sent false
4347 2128 Acked Write 1 Done Done Sent false
4348 2129 Acked Read 9 Done Done Sent false
4349 2130 Acked Read 3 Done Done Sent false
4350 2131 Acked Write 10 Done Done Sent false
4351 2132 Acked Write 2 Done Done Sent false
4352 2133 Acked Read 8 Done Done Sent false
4353 2134 Acked Read 2 Done Done Sent false
4354 2135 Acked Read 6 Done Done Sent false
4355 2136 Acked Read 6 Done Done Sent false
4356 2137 Acked Read 1 Done Done Sent false
4357 2138 Acked Read 9 Done Done Sent false
4358 2139 Acked Read 2 Done Done Sent false
4359 2140 Acked Write 5 Done Done Sent false
4360 2141 Acked Write 4 Done Done Sent false
4361 2142 Acked Read 6 Done Done Sent false
4362 2143 Acked Read 9 Done Done Sent false
4363 2144 Acked Read 6 Done Done Sent false
4364 2145 Acked Write 3 Done Done Sent false
4365 2146 Acked Read 2 Done Done Sent false
4366 2147 Acked Flush 0 Done Done Sent false
4367 2148 Acked Flush 0 Done Done Sent false
4368 2149 Acked Read 8 Done Done Sent false
4369 2150 Acked Read 4 Done Done Sent false
4370 2151 Acked Read 4 Done Done Sent false
4371 2152 Acked Flush 0 Done Done Sent false
4372 2153 Acked Write 5 Done Done Sent false
4373 2154 Acked Write 8 Done Done Sent false
4374 2155 Acked Read 3 Done Done Sent false
4375 2156 Acked Read 9 Done Done Sent false
4376 2157 Acked Write 3 Done Done Sent false
4377 2158 Acked Read 2 Done Done Sent false
4378 2159 Acked Read 10 Done Done Sent false
4379 2160 Acked Read 1 Done Done Sent false
4380 2161 Acked Write 1 Done Done Sent false
4381 2162 Acked Read 8 Done Done Sent false
4382 2163 Acked Read 1 Done Done Sent false
4383 2164 Acked Read 9 Done Done Sent false
4384 2165 Acked Read 5 Done Done Sent false
4385 2166 Acked Read 10 Done Done Sent false
4386 2167 Acked Read 10 Done Done Sent false
4387 2168 Acked Read 2 Done Done Sent false
4388 2169 Acked Read 6 Done Done Sent false
4389 2170 Acked Write 9 Done Done Sent false
4390 2171 Acked Read 9 Done Done Sent false
4391 2172 Acked Write 9 Done Done Sent false
4392 2173 Acked Write 9 Done Done Sent false
4393 2174 Acked Read 10 Done Done Sent false
4394 2175 Acked Read 5 Done Done Sent false
4395 2176 Acked Read 7 Done Done Sent false
4396 2177 Acked Flush 0 Done Done Sent false
4397 2178 Acked Read 6 Done Done Sent false
4398 2179 Acked Read 3 Done Done Sent false
4399 2180 Acked Write 10 Done Done Sent false
4400 2181 Acked Write 6 Done Done Sent false
4401 2182 Acked Read 1 Done Done Sent false
4402 2183 Acked Write 6 Done Done Sent false
4403 2184 Acked Read 4 Done Done Sent false
4404 2185 Acked Read 10 Done Done Sent false
4405 2186 Acked Write 2 Done Done Sent false
4406 2187 Acked Read 7 Done Done Sent false
4407 2188 Acked Write 1 Done Done Sent false
4408 2189 Acked Write 7 Done Done Sent false
4409 2190 Acked Write 10 Done Done Sent false
4410 2191 Acked Write 1 Done Done Sent false
4411 2192 Acked Read 1 Done Done Sent false
4412 2193 Acked Read 9 Done Done Sent false
4413 2194 Acked Write 9 Done Done Sent false
4414 2195 Acked Read 2 Done Done Sent false
4415 2196 Acked Write 8 Done Done Sent false
4416 2197 Acked Write 8 Done Done Sent false
4417 2198 Acked Read 2 Done Done Sent false
4418 2199 Acked Read 9 Done Done Sent false
4419 2200 Acked Read 6 Done Done Sent false
4420 2201 Acked Read 2 Done Done Sent false
4421 2202 Acked Read 4 Done Done Sent false
4422 2203 Acked Write 3 Done Done Sent false
4423 2204 Acked Read 7 Done Done Sent false
4424 2205 Acked Read 1 Done Done Sent false
4425 STATES DS:0 DS:1 DS:2 TOTAL
4426 Sent 0 0 301 301
4427 Done 301 301 0 602
4428 Skipped 0 0 0 0
4429 Error 0 0 0 0
4430 Last Flush: Some(JobId(2177)) Some(JobId(2177)) Some(JobId(1904))
4431 Downstairs last five completed: 1904 1903 1902 1901 1900
4432 Upstairs last five completed: 2205 2204 2203 2202 2201
4433 CLIENT: Up:0 ds:301 act:2
44342025-02-25T20:47:38.156ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
44352025-02-25T20:47:39.156ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
4436 [8830] got start action
4437 [8830] I am going to restart
4438 Starting downstairs at port 8830
4439 Make output file at "/tmp/dsc/downstairs-8830.txt"
4440 [8830][2] reports Starting
4441 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1210)
4442 [8830][2] reports Running
44432025-02-25T20:47:40.158ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
44442025-02-25T20:47:41.159ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
4445 ----------------------------------------------------------------
4446 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4447 States: Active Active Offline
4448 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4449 1905 Acked Write 2 Done Done Sent false
4450 1906 Acked Write 4 Done Done Sent false
4451 1907 Acked Read 6 Done Done Sent false
4452 1908 Acked Read 2 Done Done Sent false
4453 1909 Acked Read 6 Done Done Sent false
4454 1910 Acked Read 2 Done Done Sent false
4455 1911 Acked Read 1 Done Done Sent false
4456 1912 Acked Write 6 Done Done Sent false
4457 1913 Acked Read 6 Done Done Sent false
4458 1914 Acked Flush 0 Done Done Sent false
4459 1915 Acked Write 2 Done Done Sent false
4460 1916 Acked Read 9 Done Done Sent false
4461 1917 Acked Read 2 Done Done Sent false
4462 1918 Acked Read 9 Done Done Sent false
4463 1919 Acked Read 1 Done Done Sent false
4464 1920 Acked Read 4 Done Done Sent false
4465 1921 Acked Write 6 Done Done Sent false
4466 1922 Acked Read 6 Done Done Sent false
4467 1923 Acked Read 7 Done Done Sent false
4468 1924 Acked Read 6 Done Done Sent false
4469 1925 Acked Write 5 Done Done Sent false
4470 1926 Acked Read 3 Done Done Sent false
4471 1927 Acked Read 10 Done Done Sent false
4472 1928 Acked Write 6 Done Done Sent false
4473 1929 Acked Read 6 Done Done Sent false
4474 1930 Acked Read 8 Done Done Sent false
4475 1931 Acked Write 5 Done Done Sent false
4476 1932 Acked Read 7 Done Done Sent false
4477 1933 Acked Write 1 Done Done Sent false
4478 1934 Acked Read 10 Done Done Sent false
4479 1935 Acked Write 4 Done Done Sent false
4480 1936 Acked Read 3 Done Done Sent false
4481 1937 Acked Write 5 Done Done Sent false
4482 1938 Acked Read 6 Done Done Sent false
4483 1939 Acked Write 7 Done Done Sent false
4484 1940 Acked Read 2 Done Done Sent false
4485 1941 Acked Read 2 Done Done Sent false
4486 1942 Acked Read 8 Done Done Sent false
4487 1943 Acked Write 6 Done Done Sent false
4488 1944 Acked Read 9 Done Done Sent false
4489 1945 Acked Write 5 Done Done Sent false
4490 1946 Acked Read 8 Done Done Sent false
4491 1947 Acked Read 6 Done Done Sent false
4492 1948 Acked Flush 0 Done Done Sent false
4493 1949 Acked Read 7 Done Done Sent false
4494 1950 Acked Read 6 Done Done Sent false
4495 1951 Acked Write 6 Done Done Sent false
4496 1952 Acked Read 2 Done Done Sent false
4497 1953 Acked Write 6 Done Done Sent false
4498 1954 Acked Read 10 Done Done Sent false
4499 1955 Acked Write 10 Done Done Sent false
4500 1956 Acked Read 3 Done Done Sent false
4501 1957 Acked Read 1 Done Done Sent false
4502 1958 Acked Write 8 Done Done Sent false
4503 1959 Acked Flush 0 Done Done Sent false
4504 1960 Acked Write 10 Done Done Sent false
4505 1961 Acked Write 9 Done Done Sent false
4506 1962 Acked Flush 0 Done Done Sent false
4507 1963 Acked Write 4 Done Done Sent false
4508 1964 Acked Flush 0 Done Done Sent false
4509 1965 Acked Write 6 Done Done Sent false
4510 1966 Acked Write 9 Done Done Sent false
4511 1967 Acked Read 10 Done Done Sent false
4512 1968 Acked Write 6 Done Done Sent false
4513 1969 Acked Read 10 Done Done Sent false
4514 1970 Acked Write 3 Done Done Sent false
4515 1971 Acked Read 2 Done Done Sent false
4516 1972 Acked Read 3 Done Done Sent false
4517 1973 Acked Read 5 Done Done Sent false
4518 1974 Acked Read 3 Done Done Sent false
4519 1975 Acked Read 10 Done Done Sent false
4520 1976 Acked Read 4 Done Done Sent false
4521 1977 Acked Flush 0 Done Done Sent false
4522 1978 Acked Write 4 Done Done Sent false
4523 1979 Acked Read 8 Done Done Sent false
4524 1980 Acked Read 1 Done Done Sent false
4525 1981 Acked Write 9 Done Done Sent false
4526 1982 Acked Write 8 Done Done Sent false
4527 1983 Acked Write 3 Done Done Sent false
4528 1984 Acked Read 1 Done Done Sent false
4529 1985 Acked Write 2 Done Done Sent false
4530 1986 Acked Write 4 Done Done Sent false
4531 1987 Acked Read 7 Done Done Sent false
4532 1988 Acked Read 3 Done Done Sent false
4533 1989 Acked Read 4 Done Done Sent false
4534 1990 Acked Read 4 Done Done Sent false
4535 1991 Acked Read 1 Done Done Sent false
4536 1992 Acked Read 9 Done Done Sent false
4537 1993 Acked Write 4 Done Done Sent false
4538 1994 Acked Read 8 Done Done Sent false
4539 1995 Acked Flush 0 Done Done Sent false
4540 1996 Acked Read 2 Done Done Sent false
4541 1997 Acked Write 10 Done Done Sent false
4542 1998 Acked Read 6 Done Done Sent false
4543 1999 Acked Write 3 Done Done Sent false
4544 2000 Acked Read 10 Done Done Sent false
4545 2001 Acked Write 2 Done Done Sent false
4546 2002 Acked Write 2 Done Done Sent false
4547 2003 Acked Read 10 Done Done Sent false
4548 2004 Acked Read 4 Done Done Sent false
4549 2005 Acked Write 8 Done Done Sent false
4550 2006 Acked Read 7 Done Done Sent false
4551 2007 Acked Read 3 Done Done Sent false
4552 2008 Acked Write 3 Done Done Sent false
4553 2009 Acked Write 4 Done Done Sent false
4554 2010 Acked Read 5 Done Done Sent false
4555 2011 Acked Write 2 Done Done Sent false
4556 2012 Acked Read 2 Done Done Sent false
4557 2013 Acked Read 1 Done Done Sent false
4558 2014 Acked Write 8 Done Done Sent false
4559 2015 Acked Write 3 Done Done Sent false
4560 2016 Acked Read 9 Done Done Sent false
4561 2017 Acked Flush 0 Done Done Sent false
4562 2018 Acked Write 4 Done Done Sent false
4563 2019 Acked Read 3 Done Done Sent false
4564 2020 Acked Write 5 Done Done Sent false
4565 2021 Acked Read 8 Done Done Sent false
4566 2022 Acked Write 7 Done Done Sent false
4567 2023 Acked Read 3 Done Done Sent false
4568 2024 Acked Write 8 Done Done Sent false
4569 2025 Acked Write 4 Done Done Sent false
4570 2026 Acked Read 10 Done Done Sent false
4571 2027 Acked Read 10 Done Done Sent false
4572 2028 Acked Write 1 Done Done Sent false
4573 2029 Acked Write 4 Done Done Sent false
4574 2030 Acked Read 5 Done Done Sent false
4575 2031 Acked Write 1 Done Done Sent false
4576 2032 Acked Read 5 Done Done Sent false
4577 2033 Acked Flush 0 Done Done Sent false
4578 2034 Acked Read 10 Done Done Sent false
4579 2035 Acked Read 4 Done Done Sent false
4580 2036 Acked Write 8 Done Done Sent false
4581 2037 Acked Read 9 Done Done Sent false
4582 2038 Acked Read 9 Done Done Sent false
4583 2039 Acked Read 1 Done Done Sent false
4584 2040 Acked Write 1 Done Done Sent false
4585 2041 Acked Read 3 Done Done Sent false
4586 2042 Acked Read 3 Done Done Sent false
4587 2043 Acked Read 10 Done Done Sent false
4588 2044 Acked Write 8 Done Done Sent false
4589 2045 Acked Write 10 Done Done Sent false
4590 2046 Acked Write 1 Done Done Sent false
4591 2047 Acked Read 6 Done Done Sent false
4592 2048 Acked Write 8 Done Done Sent false
4593 2049 Acked Write 10 Done Done Sent false
4594 2050 Acked Write 10 Done Done Sent false
4595 2051 Acked Flush 0 Done Done Sent false
4596 2052 Acked Read 1 Done Done Sent false
4597 2053 Acked Read 1 Done Done Sent false
4598 2054 Acked Read 1 Done Done Sent false
4599 2055 Acked Read 8 Done Done Sent false
4600 2056 Acked Read 1 Done Done Sent false
4601 2057 Acked Write 1 Done Done Sent false
4602 2058 Acked Read 3 Done Done Sent false
4603 2059 Acked Read 9 Done Done Sent false
4604 2060 Acked Write 10 Done Done Sent false
4605 2061 Acked Write 4 Done Done Sent false
4606 2062 Acked Write 10 Done Done Sent false
4607 2063 Acked Write 10 Done Done Sent false
4608 2064 Acked Write 3 Done Done Sent false
4609 2065 Acked Flush 0 Done Done Sent false
4610 2066 Acked Read 4 Done Done Sent false
4611 2067 Acked Write 8 Done Done Sent false
4612 2068 Acked Write 7 Done Done Sent false
4613 2069 Acked Write 3 Done Done Sent false
4614 2070 Acked Read 5 Done Done Sent false
4615 2071 Acked Write 4 Done Done Sent false
4616 2072 Acked Read 3 Done Done Sent false
4617 2073 Acked Read 10 Done Done Sent false
4618 2074 Acked Read 10 Done Done Sent false
4619 2075 Acked Read 5 Done Done Sent false
4620 2076 Acked Read 1 Done Done Sent false
4621 2077 Acked Flush 0 Done Done Sent false
4622 2078 Acked Read 6 Done Done Sent false
4623 2079 Acked Read 2 Done Done Sent false
4624 2080 Acked Read 4 Done Done Sent false
4625 2081 Acked Write 5 Done Done Sent false
4626 2082 Acked Write 3 Done Done Sent false
4627 2083 Acked Write 9 Done Done Sent false
4628 2084 Acked Read 3 Done Done Sent false
4629 2085 Acked Write 2 Done Done Sent false
4630 2086 Acked Read 6 Done Done Sent false
4631 2087 Acked Flush 0 Done Done Sent false
4632 2088 Acked Read 8 Done Done Sent false
4633 2089 Acked Write 4 Done Done Sent false
4634 2090 Acked Read 7 Done Done Sent false
4635 2091 Acked Flush 0 Done Done Sent false
4636 2092 Acked Read 1 Done Done Sent false
4637 2093 Acked Read 4 Done Done Sent false
4638 2094 Acked Write 1 Done Done Sent false
4639 2095 Acked Read 3 Done Done Sent false
4640 2096 Acked Write 5 Done Done Sent false
4641 2097 Acked Write 4 Done Done Sent false
4642 2098 Acked Write 2 Done Done Sent false
4643 2099 Acked Read 3 Done Done Sent false
4644 2100 Acked Read 4 Done Done Sent false
4645 2101 Acked Read 10 Done Done Sent false
4646 2102 Acked Flush 0 Done Done Sent false
4647 2103 Acked Read 1 Done Done Sent false
4648 2104 Acked Write 10 Done Done Sent false
4649 2105 Acked Read 6 Done Done Sent false
4650 2106 Acked Read 7 Done Done Sent false
4651 2107 Acked Flush 0 Done Done Sent false
4652 2108 Acked Read 9 Done Done Sent false
4653 2109 Acked Read 4 Done Done Sent false
4654 2110 Acked Read 10 Done Done Sent false
4655 2111 Acked Read 10 Done Done Sent false
4656 2112 Acked Write 1 Done Done Sent false
4657 2113 Acked Read 9 Done Done Sent false
4658 2114 Acked Read 6 Done Done Sent false
4659 2115 Acked Write 6 Done Done Sent false
4660 2116 Acked Write 6 Done Done Sent false
4661 2117 Acked Write 10 Done Done Sent false
4662 2118 Acked Read 5 Done Done Sent false
4663 2119 Acked Read 3 Done Done Sent false
4664 2120 Acked Write 7 Done Done Sent false
4665 2121 Acked Write 1 Done Done Sent false
4666 2122 Acked Write 1 Done Done Sent false
4667 2123 Acked Write 5 Done Done Sent false
4668 2124 Acked Read 9 Done Done Sent false
4669 2125 Acked Write 8 Done Done Sent false
4670 2126 Acked Read 10 Done Done Sent false
4671 2127 Acked Read 6 Done Done Sent false
4672 2128 Acked Write 1 Done Done Sent false
4673 2129 Acked Read 9 Done Done Sent false
4674 2130 Acked Read 3 Done Done Sent false
4675 2131 Acked Write 10 Done Done Sent false
4676 2132 Acked Write 2 Done Done Sent false
4677 2133 Acked Read 8 Done Done Sent false
4678 2134 Acked Read 2 Done Done Sent false
4679 2135 Acked Read 6 Done Done Sent false
4680 2136 Acked Read 6 Done Done Sent false
4681 2137 Acked Read 1 Done Done Sent false
4682 2138 Acked Read 9 Done Done Sent false
4683 2139 Acked Read 2 Done Done Sent false
4684 2140 Acked Write 5 Done Done Sent false
4685 2141 Acked Write 4 Done Done Sent false
4686 2142 Acked Read 6 Done Done Sent false
4687 2143 Acked Read 9 Done Done Sent false
4688 2144 Acked Read 6 Done Done Sent false
4689 2145 Acked Write 3 Done Done Sent false
4690 2146 Acked Read 2 Done Done Sent false
4691 2147 Acked Flush 0 Done Done Sent false
4692 2148 Acked Flush 0 Done Done Sent false
4693 2149 Acked Read 8 Done Done Sent false
4694 2150 Acked Read 4 Done Done Sent false
4695 2151 Acked Read 4 Done Done Sent false
4696 2152 Acked Flush 0 Done Done Sent false
4697 2153 Acked Write 5 Done Done Sent false
4698 2154 Acked Write 8 Done Done Sent false
4699 2155 Acked Read 3 Done Done Sent false
4700 2156 Acked Read 9 Done Done Sent false
4701 2157 Acked Write 3 Done Done Sent false
4702 2158 Acked Read 2 Done Done Sent false
4703 2159 Acked Read 10 Done Done Sent false
4704 2160 Acked Read 1 Done Done Sent false
4705 2161 Acked Write 1 Done Done Sent false
4706 2162 Acked Read 8 Done Done Sent false
4707 2163 Acked Read 1 Done Done Sent false
4708 2164 Acked Read 9 Done Done Sent false
4709 2165 Acked Read 5 Done Done Sent false
4710 2166 Acked Read 10 Done Done Sent false
4711 2167 Acked Read 10 Done Done Sent false
4712 2168 Acked Read 2 Done Done Sent false
4713 2169 Acked Read 6 Done Done Sent false
4714 2170 Acked Write 9 Done Done Sent false
4715 2171 Acked Read 9 Done Done Sent false
4716 2172 Acked Write 9 Done Done Sent false
4717 2173 Acked Write 9 Done Done Sent false
4718 2174 Acked Read 10 Done Done Sent false
4719 2175 Acked Read 5 Done Done Sent false
4720 2176 Acked Read 7 Done Done Sent false
4721 2177 Acked Flush 0 Done Done Sent false
4722 2178 Acked Read 6 Done Done Sent false
4723 2179 Acked Read 3 Done Done Sent false
4724 2180 Acked Write 10 Done Done Sent false
4725 2181 Acked Write 6 Done Done Sent false
4726 2182 Acked Read 1 Done Done Sent false
4727 2183 Acked Write 6 Done Done Sent false
4728 2184 Acked Read 4 Done Done Sent false
4729 2185 Acked Read 10 Done Done Sent false
4730 2186 Acked Write 2 Done Done Sent false
4731 2187 Acked Read 7 Done Done Sent false
4732 2188 Acked Write 1 Done Done Sent false
4733 2189 Acked Write 7 Done Done Sent false
4734 2190 Acked Write 10 Done Done Sent false
4735 2191 Acked Write 1 Done Done Sent false
4736 2192 Acked Read 1 Done Done Sent false
4737 2193 Acked Read 9 Done Done Sent false
4738 2194 Acked Write 9 Done Done Sent false
4739 2195 Acked Read 2 Done Done Sent false
4740 2196 Acked Write 8 Done Done Sent false
4741 2197 Acked Write 8 Done Done Sent false
4742 2198 Acked Read 2 Done Done Sent false
4743 2199 Acked Read 9 Done Done Sent false
4744 2200 Acked Read 6 Done Done Sent false
4745 2201 Acked Read 2 Done Done Sent false
4746 2202 Acked Read 4 Done Done Sent false
4747 2203 Acked Write 3 Done Done Sent false
4748 2204 Acked Read 7 Done Done Sent false
4749 2205 Acked Read 1 Done Done Sent false
4750 STATES DS:0 DS:1 DS:2 TOTAL
4751 Sent 0 0 301 301
4752 Done 301 301 0 602
4753 Skipped 0 0 0 0
4754 Error 0 0 0 0
4755 Last Flush: Some(JobId(2177)) Some(JobId(2177)) Some(JobId(1904))
4756 Downstairs last five completed: 1904 1903 1902 1901 1900
4757 Upstairs last five completed: 2205 2204 2203 2202 2201
4758 CLIENT: Up:0 ds:301 act:2
47592025-02-25T20:47:42.359ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47602025-02-25T20:47:43.163ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47612025-02-25T20:47:43.365ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47622025-02-25T20:47:43.365ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58149) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47632025-02-25T20:47:43.366ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47642025-02-25T20:47:43.367ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47652025-02-25T20:47:43.367ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47662025-02-25T20:47:43.367ZINFOcrucible: send last flush ID to this DS: Some(JobId(1904)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47672025-02-25T20:47:43.367ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47682025-02-25T20:47:43.367ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1904)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47692025-02-25T20:47:43.367ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1904)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
4770 ----------------------------------------------------------------
4771 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4772 Downstairs last five completed: 2206 2205 2204 2203 2202
4773 Upstairs last five completed: 2206 2205 2204 2203 2202
4774 CLIENT: Up:0 ds:0 act:3
4775 Replay: All jobs finished
4776 Main task has work to do, go find it
4777 got dsc Stop(0)
4778 stop 0
4779 Feb 25 20:47:45.624 INFO request completed, latency_us: 366, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 80cfd86a-050a-4972-bfc5-9dc2a18e0a33, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
4780 [8810] Got stop action so:false kr:false
47812025-02-25T20:47:45.625ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47822025-02-25T20:47:45.625ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
47832025-02-25T20:47:45.625ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
4784 Feb 25 20:47:45.625 INFO request completed, latency_us: 182{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:47:45.625865476Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
4785 , response_code: 200, uri: /state/cid/0, method: GET, req_id: c5807bd5-f576-4426-9b28-4ccd2f972503, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
4786 [8810] Exited with: ExitStatus(unix_wait_status(9))
4787 [8810][0] reports Exit
4788 Feb 25 20:47:49.628 INFO request completed, latency_us: 360, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8dc83ea6-ff10-4c03-919b-146fd178938b, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
4789 001/300 Read block 1866 len 36864
4790 002/300 Write block 69 len 28672 data: 2 2 2 2 2 3 3
4791 003/300 Write block 3574 len 16384 data: 3 2 2 2
4792 004/300 Read block 4375 len 28672
4793 005/300 Write block 4185 len 12288 data: 2 2 2
4794 006/300 Read block 4621 len 40960
4795 007/300 Read block 2783 len 32768
4796 008/300 Write block 4953 len 4096 data: 3
4797 009/300 Read block 3528 len 28672
4798 010/300 Read block 3790 len 4096
4799 011/300 Read block 1835 len 4096
4800 012/300 Read block 180 len 12288
4801 013/300 Write block 1323 len 12288 data: 3 3 4
4802 014/300 Read block 2562 len 8192
4803 015/300 Read block 1051 len 40960
4804 016/300 Read block 925 len 36864
4805 017/300 Read block 2453 len 32768
4806 018/300 Write block 2718 len 24576 data: 2 2 2 2 2 2
4807 019/300 Write block 3368 len 28672 data: 2 2 3 3 3 3 3
4808 020/300 Read block 3803 len 20480
4809 021/300 Write block 3750 len 28672 data: 2 2 2 2 2 2 2
4810 022/300 Write block 3720 len 20480 data: 2 2 2 2 2
4811 023/300 Read block 4053 len 36864
4812 024/300 Flush
4813 025/300 Write block 1623 len 40960 data: 3 3 3 3 3 3 3 3 3 3
4814 026/300 Read block 2855 len 40960
4815 027/300 Flush
4816 028/300 Read block 4357 len 36864
4817 029/300 Read block 4423 len 20480
4818 030/300 Flush
4819 031/300 Read block 3678 len 32768
4820 032/300 Write block 3943 len 40960 data: 2 2 3 4 3 3 3 4 4 4
4821 033/300 Write block 54 len 36864 data: 2 2 2 2 2 2 2 2 2
4822 034/300 Read block 3496 len 28672
4823 035/300 Write block 2222 len 20480 data: 2 2 2 2 2
4824 036/300 Read block 2609 len 24576
4825 037/300 Write block 374 len 20480 data: 2 2 2 2 2
4826 038/300 Write block 3611 len 32768 data: 3 3 3 3 3 3 3 3
4827 039/300 Read block 4087 len 16384
4828 040/300 Write block 993 len 16384 data: 2 2 2 2
4829 041/300 Read block 1860 len 40960
4830 042/300 Read block 4979 len 4096
4831 043/300 Flush
4832 044/300 Read block 284 len 36864
4833 045/300 Flush
4834 046/300 Write block 2099 len 40960 data: 2 2 2 2 2 2 2 2 3 3
4835 047/300 Flush
4836 048/300 Read block 331 len 32768
4837 049/300 Read block 2471 len 24576
4838 050/300 Read block 4251 len 12288
4839 051/300 Write block 3264 len 16384 data: 2 2 2 2
4840 052/300 Read block 3811 len 24576
4841 053/300 Read block 2748 len 4096
4842 054/300 Read block 1190 len 20480
4843 055/300 Read block 3351 len 20480
4844 056/300 Read block 3040 len 4096
4845 057/300 Read block 2625 len 8192
4846 058/300 Read block 701 len 20480
4847 059/300 Read block 4384 len 40960
4848 060/300 Write block 2706 len 20480 data: 3 3 4 4 5
4849 061/300 Write block 125 len 12288 data: 3 3 3
4850 062/300 Read block 3769 len 4096
4851 063/300 Read block 23 len 16384
4852 064/300 Read block 1750 len 20480
4853 065/300 Write block 1696 len 36864 data: 2 2 2 2 2 2 2 2 2
4854 066/300 Flush
4855 067/300 Flush
4856 068/300 Write block 3345 len 20480 data: 2 4 3 4 4
4857 069/300 Write block 4586 len 4096 data: 2
4858 070/300 Write block 1354 len 40960 data: 2 2 2 2 2 2 2 4 3 3
4859 071/300 Read block 264 len 24576
4860 072/300 Read block 1659 len 40960
4861 073/300 Read block 3626 len 4096
4862 074/300 Read block 2072 len 20480
4863 075/300 Write block 2605 len 12288 data: 2 2 2
4864 076/300 Flush
4865 077/300 Read block 3162 len 8192
4866 078/300 Read block 4141 len 40960
4867 079/300 Flush
4868 080/300 Read block 1765 len 12288
4869 081/300 Write block 1298 len 12288 data: 2 2 2
4870 082/300 Write block 4949 len 36864 data: 3 3 3 3 4 2 2 2 2
4871 083/300 Read block 555 len 16384
4872 084/300 Read block 2934 len 28672
4873 085/300 Flush
4874 086/300 Write block 4884 len 28672 data: 3 3 2 2 2 2 2
4875 087/300 Read block 1640 len 12288
4876 088/300 Write block 2718 len 12288 data: 3 3 3
4877 089/300 Flush
4878 090/300 Read block 1159 len 12288
4879 091/300 Flush
4880 092/300 Read block 4796 len 12288
4881 093/300 Read block 2889 len 28672
4882 094/300 Write block 1924 len 16384 data: 2 2 2 2
4883 095/300 Write block 1813 len 12288 data: 3 3 3
4884 096/300 Read block 1516 len 32768
4885 097/300 Write block 3972 len 4096 data: 2
4886 098/300 Flush
4887 099/300 Flush
4888 100/300 Write block 993 len 16384 data: 3 3 3 3
4889 101/300 Write block 2076 len 28672 data: 2 2 2 2 2 3 3
4890 102/300 Read block 3853 len 4096
4891 103/300 Read block 4951 len 36864
4892 104/300 Write block 2956 len 32768 data: 2 2 2 3 3 3 3 3
4893 105/300 Flush
4894 106/300 Write block 3827 len 28672 data: 3 3 3 3 3 2 2
4895 107/300 Read block 322 len 16384
4896 108/300 Read block 483 len 8192
4897 109/300 Write block 4485 len 8192 data: 2 2
4898 110/300 Write block 4141 len 8192 data: 2 3
4899 111/300 Flush
4900 112/300 Write block 4657 len 36864 data: 2 2 2 2 3 3 3 3 4
4901 113/300 Write block 2606 len 24576 data: 3 3 3 2 2 2
4902 114/300 Flush
4903 115/300 Write block 422 len 20480 data: 3 3 3 3 3
4904 116/300 Write block 409 len 4096 data: 2
4905 117/300 Read block 3336 len 28672
4906 118/300 Write block 4468 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4907 119/300 Read block 3642 len 8192
4908 120/300 Read block 519 len 24576
4909 121/300 Read block 3973 len 16384
4910 122/300 Write block 4578 len 12288 data: 2 2 2
4911 123/300 Write block 165 len 8192 data: 3 3
4912 124/300 Read block 175 len 16384
4913 125/300 Write block 4505 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4914 126/300 Write block 2511 len 16384 data: 2 2 2 3
4915 127/300 Read block 30 len 4096
4916 128/300 Write block 3529 len 24576 data: 2 2 2 2 2 2
4917 129/300 Write block 2445 len 24576 data: 3 2 3 3 4 5
4918 130/300 Read block 3329 len 36864
4919 131/300 Read block 4924 len 16384
4920 132/300 Write block 2117 len 16384 data: 2 2 2 2
4921 133/300 Write block 4036 len 24576 data: 2 3 3 3 3 3
4922 134/300 Write block 1067 len 8192 data: 3 3
4923 135/300 Write block 2009 len 8192 data: 3 3
4924 136/300 Read block 2885 len 24576
4925 137/300 Write block 2178 len 24576 data: 2 2 2 2 2 2
4926 138/300 Write block 4339 len 8192 data: 3 3
4927 139/300 Write block 1934 len 24576 data: 2 2 2 2 2 2
4928 140/300 Read block 3316 len 28672
4929 141/300 Read block 4200 len 8192
4930 142/300 Read block 1025 len 20480
4931 143/300 Read block 1238 len 4096
4932 144/300 Write block 331 len 24576 data: 4 4 3 3 3 3
4933 145/300 Read block 4727 len 16384
4934 146/300 Read block 2312 len 40960
4935 147/300 Write block 3183 len 8192 data: 3 2
4936 148/300 Read block 2981 len 40960
4937 149/300 Write block 3680 len 36864 data: 3 3 2 2 2 2 2 2 2
4938 150/300 Read block 3726 len 36864
4939 151/300 Write block 2760 len 12288 data: 2 2 2
4940 152/300 Write block 2893 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4941 153/300 Write block 4217 len 20480 data: 2 2 2 2 2
4942 154/300 Read block 2625 len 40960
4943 155/300 Write block 4117 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4944 156/300 Read block 2704 len 40960
4945 157/300 Write block 3111 len 24576 data: 2 3 3 2 2 2
4946 158/300 Flush
4947 159/300 Read block 2861 len 8192
4948 160/300 Read block 4058 len 8192
4949 161/300 Read block 2743 len 36864
4950 162/300 Read block 2830 len 36864
4951 163/300 Write block 3134 len 16384 data: 2 2 2 2
4952 164/300 Write block 3142 len 32768 data: 2 2 2 2 2 2 2 2
4953 165/300 Read block 4866 len 32768
4954 166/300 Write block 1602 len 28672 data: 2 2 2 2 2 2 2
4955 167/300 Read block 2293 len 8192
4956 168/300 Write block 4874 len 32768 data: 3 2 2 2 2 2 2 3
4957 169/300 Read block 480 len 40960
4958 170/300 Read block 120 len 28672
4959 171/300 Write block 703 len 20480 data: 3 4 4 4 3
4960 172/300 Read block 2090 len 36864
4961 173/300 Flush
4962 174/300 Flush
4963 175/300 Read block 3997 len 12288
4964 176/300 Flush
4965 177/300 Read block 4041 len 40960
4966 178/300 Flush
4967 179/300 Write block 2743 len 36864 data: 2 2 2 2 2 2 2 2 2
4968 180/300 Write block 3242 len 20480 data: 3 2 2 2 2
4969 181/300 Read block 1083 len 8192
4970 182/300 Write block 810 len 28672 data: 2 2 2 2 2 2 2
4971 183/300 Flush
4972 184/300 Read block 2101 len 24576
4973 185/300 Read block 4544 len 20480
4974 186/300 Read block 710 len 40960
4975 187/300 Read block 4017 len 24576
4976 188/300 Read block 4419 len 28672
4977 189/300 Write block 840 len 32768 data: 3 2 4 4 4 4 4 4
4978 190/300 Read block 1845 len 20480
4979 191/300 Write block 2867 len 20480 data: 3 3 3 3 3
4980 192/300 Write block 3988 len 32768 data: 2 2 2 2 2 2 2 2
4981 193/300 Write block 3936 len 8192 data: 2 2
4982 194/300 Flush
4983 195/300 Read block 2045 len 20480
4984 196/300 Write block 1259 len 28672 data: 2 2 2 2 2 2 2
4985 197/300 Read block 1640 len 8192
4986 198/300 Write block 200 len 40960 data: 3 3 3 3 3 4 4 2 2 3
4987 199/300 Read block 522 len 36864
4988 200/300 Flush
4989 201/300 Read block 4020 len 28672
4990 202/300 Write block 812 len 36864 data: 3 3 3 3 3 3 3 3 3
4991 203/300 Write block 2185 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4992 204/300 Flush
4993 205/300 Write block 2871 len 16384 data: 4 2 2 2
4994 206/300 Flush
4995 207/300 Write block 4461 len 16384 data: 2 2 2 2
4996 208/300 Write block 3555 len 20480 data: 2 2 2 2 2
4997 209/300 Write block 4847 len 12288 data: 3 3 3
4998 210/300 Read block 846 len 28672
4999 211/300 Write block 681 len 28672 data: 2 2 2 3 3 3 3
5000 212/300 Flush
5001 213/300 Read block 4485 len 32768
5002 214/300 Read block 418 len 4096
5003 215/300 Write block 1409 len 12288 data: 2 2 2
5004 216/300 Read block 18 len 36864
5005 217/300 Write block 2097 len 24576 data: 2 2 3 3 3 3
5006 218/300 Read block 220 len 8192
5007 219/300 Read block 392 len 40960
5008 220/300 Read block 4862 len 16384
5009 221/300 Read block 4539 len 4096
5010 222/300 Write block 2982 len 40960 data: 2 2 2 2 2 3 4 3 3 3
5011 223/300 Read block 3205 len 28672
5012 224/300 Write block 2657 len 4096 data: 2
5013 225/300 Write block 1304 len 32768 data: 2 2 2 2 2 2 2 2
5014 226/300 Write block 3149 len 36864 data: 3 2 2 3 3 3 3 3 3
5015 227/300 Write block 1894 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5016 228/300 Read block 3631 len 24576
5017 229/300 Write block 3600 len 8192 data: 2 2
5018 230/300 Write block 1770 len 36864 data: 2 2 2 2 2 3 3 3 3
5019 231/300 Read block 2103 len 20480
5020 232/300 Read block 4578 len 12288
5021 233/300 Read block 2267 len 4096
5022 234/300 Write block 1494 len 28672 data: 2 2 2 2 2 2 2
5023 235/300 Write block 1342 len 28672 data: 2 2 2 2 2 2 2
5024 236/300 Read block 1574 len 16384
5025 237/300 Write block 3995 len 36864 data: 3 2 2 2 2 2 2 2 2
5026 238/300 Read block 2435 len 20480
5027 239/300 Write block 3298 len 32768 data: 2 2 2 2 2 2 2 2
5028 240/300 Read block 1402 len 32768
5029 241/300 Read block 298 len 4096
5030 242/300 Write block 3266 len 12288 data: 3 3 2
5031 243/300 Write block 1095 len 36864 data: 2 2 2 2 2 2 2 2 2
5032 244/300 Write block 2531 len 16384 data: 3 3 2 3
5033 245/300 Write block 4822 len 32768 data: 3 3 3 3 2 2 2 2
5034 246/300 Read block 3623 len 28672
5035 247/300 Read block 161 len 28672
5036 248/300 Read block 1456 len 32768
5037 249/300 Read block 1978 len 24576
5038 250/300 Write block 2112 len 20480 data: 3 2 2 2 2
5039 251/300 Flush
5040 252/300 Read block 3269 len 8192
5041 253/300 Write block 4386 len 28672 data: 2 2 2 2 2 2 2
5042 254/300 Read block 3292 len 24576
5043 255/300 Read block 2628 len 32768
5044 256/300 Write block 854 len 32768 data: 2 2 2 2 2 2 2 2
5045 257/300 Read block 2336 len 12288
5046 258/300 Read block 1797 len 40960
5047 259/300 Read block 4005 len 4096
5048 260/300 Read block 3757 len 28672
5049 261/300 Flush
5050 262/300 Read block 3178 len 40960
5051 263/300 Read block 2798 len 40960
5052 264/300 Write block 4529 len 4096 data: 3
5053 265/300 Read block 2090 len 16384
5054 266/300 Flush
5055 267/300 Read block 2357 len 32768
5056 268/300 Write block 4357 len 16384 data: 2 2 2 2
5057 269/300 Read block 3417 len 16384
5058 270/300 Read block 2291 len 40960
5059 271/300 Write block 321 len 8192 data: 2 2
5060 272/300 Write block 3042 len 12288 data: 2 2 2
5061 273/300 Read block 2173 len 40960
5062 274/300 Write block 3154 len 4096 data: 4
5063 275/300 Write block 4564 len 36864 data: 2 2 2 2 2 2 2 2 2
5064 276/300 Write block 511 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5065 277/300 Write block 4630 len 12288 data: 2 2 2
5066 278/300 Read block 169 len 8192
5067 279/300 Read block 2848 len 40960
5068 280/300 Read block 1407 len 32768
5069 281/300 Flush
5070 282/300 Write block 1059 len 12288 data: 2 2 2
5071 283/300 Read block 3 len 8192
5072 284/300 Flush
5073 285/300 Read block 394 len 8192
5074 286/300 Read block 232 len 4096
5075 287/300 Flush
5076 288/300 Read block 4037 len 36864
5077 289/300 Read block 888 len 16384
5078 290/300 Write block 118 len 40960 data: 2 2 2 2 2 2 3 4 4 4
5079 291/300 Read block 480 len 28672
5080 292/300 Flush
5081 293/300 Write block 2229 len 28672 data: 2 2 2 2 2 2 3
5082 294/300 Write block 3813 len 8192 data: 3 2
5083 295/300 Write block 575 len 8192 data: 2 4
5084 296/300 Write block 573 len 40960 data: 2 2 3 5 4 4 5 5 4 4
5085 297/300 Read block 2030 len 20480
5086 298/300 Write block 1457 len 28672 data: 3 3 3 3 3 4 3
5087 299/300 Write block 1553 len 12288 data: 2 2 3
5088 300/300 Write block 1672 len 28672 data: 3 3 3 3 4 4 4
5089 301/300 Read block 4964 len 8192
5090 Main task has work to do, go find it
5091 got dsc Start(0)
5092 start 0
5093 Feb 25 20:47:49.874 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7d685724-a715-43d4-a8b6-c9313bfc62bd, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
5094 Replay: started 0, returned:Ok(())
5095 ----------------------------------------------------------------
5096 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5097 States: Offline Active Active
5098 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5099 2207 Acked Read 9 Sent Done Done false
5100 2208 Acked Write 7 Sent Done Done false
5101 2209 Acked Write 4 Sent Done Done false
5102 2210 Acked Read 7 Sent Done Done false
5103 2211 Acked Write 3 Sent Done Done false
5104 2212 Acked Read 10 Sent Done Done false
5105 2213 Acked Read 8 Sent Done Done false
5106 2214 Acked Write 1 Sent Done Done false
5107 2215 Acked Read 7 Sent Done Done false
5108 2216 Acked Read 1 Sent Done Done false
5109 2217 Acked Read 1 Sent Done Done false
5110 2218 Acked Read 3 Sent Done Done false
5111 2219 Acked Write 3 Sent Done Done false
5112 2220 Acked Read 2 Sent Done Done false
5113 2221 Acked Read 10 Sent Done Done false
5114 2222 Acked Read 9 Sent Done Done false
5115 2223 Acked Read 8 Sent Done Done false
5116 2224 Acked Write 6 Sent Done Done false
5117 2225 Acked Write 7 Sent Done Done false
5118 2226 Acked Read 5 Sent Done Done false
5119 2227 Acked Write 7 Sent Done Done false
5120 2228 Acked Write 5 Sent Done Done false
5121 2229 Acked Read 9 Sent Done Done false
5122 2230 Acked Flush 0 Sent Done Done false
5123 2231 Acked Write 10 Sent Done Done false
5124 2232 Acked Read 10 Sent Done Done false
5125 2233 Acked Flush 0 Sent Done Done false
5126 2234 Acked Read 9 Sent Done Done false
5127 2235 Acked Read 5 Sent Done Done false
5128 2236 Acked Flush 0 Sent Done Done false
5129 2237 Acked Read 8 Sent Done Done false
5130 2238 Acked Write 10 Sent Done Done false
5131 2239 Acked Write 9 Sent Done Done false
5132 2240 Acked Read 7 Sent Done Done false
5133 2241 Acked Write 5 Sent Done Done false
5134 2242 Acked Read 6 Sent Done Done false
5135 2243 Acked Write 5 Sent Done Done false
5136 2244 Acked Write 8 Sent Done Done false
5137 2245 Acked Read 4 Sent Done Done false
5138 2246 Acked Write 4 Sent Done Done false
5139 2247 Acked Read 10 Sent Done Done false
5140 2248 Acked Read 1 Sent Done Done false
5141 2249 Acked Flush 0 Sent Done Done false
5142 2250 Acked Read 9 Sent Done Done false
5143 2251 Acked Flush 0 Sent Done Done false
5144 2252 Acked Write 10 Sent Done Done false
5145 2253 Acked Flush 0 Sent Done Done false
5146 2254 Acked Read 8 Sent Done Done false
5147 2255 Acked Read 6 Sent Done Done false
5148 2256 Acked Read 3 Sent Done Done false
5149 2257 Acked Write 4 Sent Done Done false
5150 2258 Acked Read 6 Sent Done Done false
5151 2259 Acked Read 1 Sent Done Done false
5152 2260 Acked Read 5 Sent Done Done false
5153 2261 Acked Read 5 Sent Done Done false
5154 2262 Acked Read 1 Sent Done Done false
5155 2263 Acked Read 2 Sent Done Done false
5156 2264 Acked Read 5 Sent Done Done false
5157 2265 Acked Read 10 Sent Done Done false
5158 2266 Acked Write 5 Sent Done Done false
5159 2267 Acked Write 3 Sent Done Done false
5160 2268 Acked Read 1 Sent Done Done false
5161 2269 Acked Read 4 Sent Done Done false
5162 2270 Acked Read 5 Sent Done Done false
5163 2271 Acked Write 9 Sent Done Done false
5164 2272 Acked Flush 0 Sent Done Done false
5165 2273 Acked Flush 0 Sent Done Done false
5166 2274 Acked Write 5 Sent Done Done false
5167 2275 Acked Write 1 Sent Done Done false
5168 2276 Acked Write 10 Sent Done Done false
5169 2277 Acked Read 6 Sent Done Done false
5170 2278 Acked Read 10 Sent Done Done false
5171 2279 Acked Read 1 Sent Done Done false
5172 2280 Acked Read 5 Sent Done Done false
5173 2281 Acked Write 3 Sent Done Done false
5174 2282 Acked Flush 0 Sent Done Done false
5175 2283 Acked Read 2 Sent Done Done false
5176 2284 Acked Read 10 Sent Done Done false
5177 2285 Acked Flush 0 Sent Done Done false
5178 2286 Acked Read 3 Sent Done Done false
5179 2287 Acked Write 3 Sent Done Done false
5180 2288 Acked Write 9 Sent Done Done false
5181 2289 Acked Read 4 Sent Done Done false
5182 2290 Acked Read 7 Sent Done Done false
5183 2291 Acked Flush 0 Sent Done Done false
5184 2292 Acked Write 7 Sent Done Done false
5185 2293 Acked Read 3 Sent Done Done false
5186 2294 Acked Write 3 Sent Done Done false
5187 2295 Acked Flush 0 Sent Done Done false
5188 2296 Acked Read 3 Sent Done Done false
5189 2297 Acked Flush 0 Sent Done Done false
5190 2298 Acked Read 3 Sent Done Done false
5191 2299 Acked Read 7 Sent Done Done false
5192 2300 Acked Write 4 Sent Done Done false
5193 2301 Acked Write 3 Sent Done Done false
5194 2302 Acked Read 8 Sent Done Done false
5195 2303 Acked Write 1 Sent Done Done false
5196 2304 Acked Flush 0 Sent Done Done false
5197 2305 Acked Flush 0 Sent Done Done false
5198 2306 Acked Write 4 Sent Done Done false
5199 2307 Acked Write 7 Sent Done Done false
5200 2308 Acked Read 1 Sent Done Done false
5201 2309 Acked Read 9 Sent Done Done false
5202 2310 Acked Write 8 Sent Done Done false
5203 2311 Acked Flush 0 Sent Done Done false
5204 2312 Acked Write 7 Sent Done Done false
5205 2313 Acked Read 4 Sent Done Done false
5206 2314 Acked Read 2 Sent Done Done false
5207 2315 Acked Write 2 Sent Done Done false
5208 2316 Acked Write 2 Sent Done Done false
5209 2317 Acked Flush 0 Sent Done Done false
5210 2318 Acked Write 9 Sent Done Done false
5211 2319 Acked Write 6 Sent Done Done false
5212 2320 Acked Flush 0 Sent Done Done false
5213 2321 Acked Write 5 Sent Done Done false
5214 2322 Acked Write 1 Sent Done Done false
5215 2323 Acked Read 7 Sent Done Done false
5216 2324 Acked Write 10 Sent Done Done false
5217 2325 Acked Read 2 Sent Done Done false
5218 2326 Acked Read 6 Sent Done Done false
5219 2327 Acked Read 4 Sent Done Done false
5220 2328 Acked Write 3 Sent Done Done false
5221 2329 Acked Write 2 Sent Done Done false
5222 2330 Acked Read 4 Sent Done Done false
5223 2331 Acked Write 10 Sent Done Done false
5224 2332 Acked Write 4 Sent Done Done false
5225 2333 Acked Read 1 Sent Done Done false
5226 2334 Acked Write 6 Sent Done Done false
5227 2335 Acked Write 6 Sent Done Done false
5228 2336 Acked Read 9 Sent Done Done false
5229 2337 Acked Read 4 Sent Done Done false
5230 2338 Acked Write 4 Sent Done Done false
5231 2339 Acked Write 6 Sent Done Done false
5232 2340 Acked Write 2 Sent Done Done false
5233 2341 Acked Write 2 Sent Done Done false
5234 2342 Acked Read 6 Sent Done Done false
5235 2343 Acked Write 6 Sent Done Done false
5236 2344 Acked Write 2 Sent Done Done false
5237 2345 Acked Write 6 Sent Done Done false
5238 2346 Acked Read 7 Sent Done Done false
5239 2347 Acked Read 2 Sent Done Done false
5240 2348 Acked Read 5 Sent Done Done false
5241 2349 Acked Read 1 Sent Done Done false
5242 2350 Acked Write 6 Sent Done Done false
5243 2351 Acked Read 4 Sent Done Done false
5244 2352 Acked Read 10 Sent Done Done false
5245 2353 Acked Write 2 Sent Done Done false
5246 2354 Acked Read 10 Sent Done Done false
5247 2355 Acked Write 9 Sent Done Done false
5248 2356 Acked Read 9 Sent Done Done false
5249 2357 Acked Write 3 Sent Done Done false
5250 2358 Acked Write 10 Sent Done Done false
5251 2359 Acked Write 5 Sent Done Done false
5252 2360 Acked Read 10 Sent Done Done false
5253 2361 Acked Write 10 Sent Done Done false
5254 2362 Acked Read 10 Sent Done Done false
5255 2363 Acked Write 6 Sent Done Done false
5256 2364 Acked Flush 0 Sent Done Done false
5257 2365 Acked Read 2 Sent Done Done false
5258 2366 Acked Read 2 Sent Done Done false
5259 2367 Acked Read 9 Sent Done Done false
5260 2368 Acked Read 9 Sent Done Done false
5261 2369 Acked Write 4 Sent Done Done false
5262 2370 Acked Write 8 Sent Done Done false
5263 2371 Acked Read 8 Sent Done Done false
5264 2372 Acked Write 7 Sent Done Done false
5265 2373 Acked Read 2 Sent Done Done false
5266 2374 Acked Write 8 Sent Done Done false
5267 2375 Acked Read 10 Sent Done Done false
5268 2376 Acked Read 7 Sent Done Done false
5269 2377 Acked Write 5 Sent Done Done false
5270 2378 Acked Read 9 Sent Done Done false
5271 2379 Acked Flush 0 Sent Done Done false
5272 2380 Acked Flush 0 Sent Done Done false
5273 2381 Acked Read 3 Sent Done Done false
5274 2382 Acked Flush 0 Sent Done Done false
5275 2383 Acked Read 10 Sent Done Done false
5276 2384 Acked Flush 0 Sent Done Done false
5277 2385 Acked Write 9 Sent Done Done false
5278 2386 Acked Write 5 Sent Done Done false
5279 2387 Acked Read 2 Sent Done Done false
5280 2388 Acked Write 7 Sent Done Done false
5281 2389 Acked Flush 0 Sent Done Done false
5282 2390 Acked Read 6 Sent Done Done false
5283 2391 Acked Read 5 Sent Done Done false
5284 2392 Acked Read 10 Sent Done Done false
5285 2393 Acked Read 6 Sent Done Done false
5286 2394 Acked Read 7 Sent Done Done false
5287 2395 Acked Write 8 Sent Done Done false
5288 2396 Acked Read 5 Sent Done Done false
5289 2397 Acked Write 5 Sent Done Done false
5290 2398 Acked Write 8 Sent Done Done false
5291 2399 Acked Write 2 Sent Done Done false
5292 2400 Acked Flush 0 Sent Done Done false
5293 2401 Acked Read 5 Sent Done Done false
5294 2402 Acked Write 7 Sent Done Done false
5295 2403 Acked Read 2 Sent Done Done false
5296 2404 Acked Write 10 Sent Done Done false
5297 2405 Acked Read 9 Sent Done Done false
5298 2406 Acked Flush 0 Sent Done Done false
5299 2407 Acked Read 7 Sent Done Done false
5300 2408 Acked Write 9 Sent Done Done false
5301 2409 Acked Write 10 Sent Done Done false
5302 2410 Acked Flush 0 Sent Done Done false
5303 2411 Acked Write 4 Sent Done Done false
5304 2412 Acked Flush 0 Sent Done Done false
5305 2413 Acked Write 4 Sent Done Done false
5306 2414 Acked Write 5 Sent Done Done false
5307 2415 Acked Write 3 Sent Done Done false
5308 2416 Acked Read 7 Sent Done Done false
5309 2417 Acked Write 7 Sent Done Done false
5310 2418 Acked Flush 0 Sent Done Done false
5311 2419 Acked Read 8 Sent Done Done false
5312 2420 Acked Read 1 Sent Done Done false
5313 2421 Acked Write 3 Sent Done Done false
5314 2422 Acked Read 9 Sent Done Done false
5315 2423 Acked Write 6 Sent Done Done false
5316 2424 Acked Read 2 Sent Done Done false
5317 2425 Acked Read 10 Sent Done Done false
5318 2426 Acked Read 4 Sent Done Done false
5319 2427 Acked Read 1 Sent Done Done false
5320 2428 Acked Write 10 Sent Done Done false
5321 2429 Acked Read 7 Sent Done Done false
5322 2430 Acked Write 1 Sent Done Done false
5323 2431 Acked Write 8 Sent Done Done false
5324 2432 Acked Write 9 Sent Done Done false
5325 2433 Acked Write 10 Sent Done Done false
5326 2434 Acked Read 6 Sent Done Done false
5327 2435 Acked Write 2 Sent Done Done false
5328 2436 Acked Write 9 Sent Done Done false
5329 2437 Acked Read 5 Sent Done Done false
5330 2438 Acked Read 3 Sent Done Done false
5331 2439 Acked Read 1 Sent Done Done false
5332 2440 Acked Write 7 Sent Done Done false
5333 2441 Acked Write 7 Sent Done Done false
5334 2442 Acked Read 4 Sent Done Done false
5335 2443 Acked Write 9 Sent Done Done false
5336 2444 Acked Read 5 Sent Done Done false
5337 2445 Acked Write 8 Sent Done Done false
5338 2446 Acked Read 8 Sent Done Done false
5339 2447 Acked Read 1 Sent Done Done false
5340 2448 Acked Write 3 Sent Done Done false
5341 2449 Acked Write 9 Sent Done Done false
5342 2450 Acked Write 4 Sent Done Done false
5343 2451 Acked Write 8 Sent Done Done false
5344 2452 Acked Read 7 Sent Done Done false
5345 2453 Acked Read 7 Sent Done Done false
5346 2454 Acked Read 8 Sent Done Done false
5347 2455 Acked Read 6 Sent Done Done false
5348 2456 Acked Write 5 Sent Done Done false
5349 2457 Acked Flush 0 Sent Done Done false
5350 2458 Acked Read 2 Sent Done Done false
5351 2459 Acked Write 7 Sent Done Done false
5352 2460 Acked Read 6 Sent Done Done false
5353 2461 Acked Read 8 Sent Done Done false
5354 2462 Acked Write 8 Sent Done Done false
5355 2463 Acked Read 3 Sent Done Done false
5356 2464 Acked Read 10 Sent Done Done false
5357 2465 Acked Read 1 Sent Done Done false
5358 2466 Acked Read 7 Sent Done Done false
5359 2467 Acked Flush 0 Sent Done Done false
5360 2468 Acked Read 10 Sent Done Done false
5361 2469 Acked Read 10 Sent Done Done false
5362 2470 Acked Write 1 Sent Done Done false
5363 2471 Acked Read 4 Sent Done Done false
5364 2472 Acked Flush 0 Sent Done Done false
5365 2473 Acked Read 8 Sent Done Done false
5366 2474 Acked Write 4 Sent Done Done false
5367 2475 Acked Read 4 Sent Done Done false
5368 2476 Acked Read 10 Sent Done Done false
5369 2477 Acked Write 2 Sent Done Done false
5370 2478 Acked Write 3 Sent Done Done false
5371 2479 Acked Read 10 Sent Done Done false
5372 2480 Acked Write 1 Sent Done Done false
5373 2481 Acked Write 9 Sent Done Done false
5374 2482 Acked Write 10 Sent Done Done false
5375 2483 Acked Write 3 Sent Done Done false
5376 2484 Acked Read 2 Sent Done Done false
5377 2485 Acked Read 10 Sent Done Done false
5378 2486 Acked Read 8 Sent Done Done false
5379 2487 Acked Flush 0 Sent Done Done false
5380 2488 Acked Write 3 Sent Done Done false
5381 2489 Acked Read 2 Sent Done Done false
5382 2490 Acked Flush 0 Sent Done Done false
5383 2491 Acked Read 2 Sent Done Done false
5384 2492 Acked Read 1 Sent Done Done false
5385 2493 Acked Flush 0 Sent Done Done false
5386 2494 Acked Read 9 Sent Done Done false
5387 2495 Acked Read 4 Sent Done Done false
5388 2496 Acked Write 10 Sent Done Done false
5389 2497 Acked Read 7 Sent Done Done false
5390 2498 Acked Flush 0 Sent Done Done false
5391 2499 Acked Write 7 Sent Done Done false
5392 2500 Acked Write 2 Sent Done Done false
5393 2501 Acked Write 2 Sent Done Done false
5394 2502 Acked Write 10 Sent Done Done false
5395 2503 Acked Read 5 Sent Done Done false
5396 2504 Acked Write 7 Sent Done Done false
5397 2505 Acked Write 3 Sent Done Done false
5398 2506 Acked Write 7 Sent Done Done false
5399 2507 Acked Read 2 Sent Done Done false
5400 STATES DS:0 DS:1 DS:2 TOTAL
5401 Sent 301 0 0 301
5402 Done 0 301 301 602
5403 Skipped 0 0 0 0
5404 Error 0 0 0 0
5405 Last Flush: Some(JobId(2206)) Some(JobId(2498)) Some(JobId(2498))
5406 Downstairs last five completed: 2206 2205 2204 2203 2202
5407 Upstairs last five completed: 2507 2506 2505 2504 2503
5408 CLIENT: Up:0 ds:301 act:2
54092025-02-25T20:47:50.377ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
54102025-02-25T20:47:51.174ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
54112025-02-25T20:47:52.361ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
54122025-02-25T20:47:53.177ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
5413 [8810] got start action
5414 [8810] I am going to restart
5415 Starting downstairs at port 8810
5416 Make output file at "/tmp/dsc/downstairs-8810.txt"
5417 [8810][0] reports Starting
5418 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211)
5419 [8810][0] reports Running
5420 ----------------------------------------------------------------
5421 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5422 States: Offline Active Active
5423 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5424 2207 Acked Read 9 Sent Done Done false
5425 2208 Acked Write 7 Sent Done Done false
5426 2209 Acked Write 4 Sent Done Done false
5427 2210 Acked Read 7 Sent Done Done false
5428 2211 Acked Write 3 Sent Done Done false
5429 2212 Acked Read 10 Sent Done Done false
5430 2213 Acked Read 8 Sent Done Done false
5431 2214 Acked Write 1 Sent Done Done false
5432 2215 Acked Read 7 Sent Done Done false
5433 2216 Acked Read 1 Sent Done Done false
5434 2217 Acked Read 1 Sent Done Done false
5435 2218 Acked Read 3 Sent Done Done false
5436 2219 Acked Write 3 Sent Done Done false
5437 2220 Acked Read 2 Sent Done Done false
5438 2221 Acked Read 10 Sent Done Done false
5439 2222 Acked Read 9 Sent Done Done false
5440 2223 Acked Read 8 Sent Done Done false
5441 2224 Acked Write 6 Sent Done Done false
5442 2225 Acked Write 7 Sent Done Done false
5443 2226 Acked Read 5 Sent Done Done false
5444 2227 Acked Write 7 Sent Done Done false
5445 2228 Acked Write 5 Sent Done Done false
5446 2229 Acked Read 9 Sent Done Done false
5447 2230 Acked Flush 0 Sent Done Done false
5448 2231 Acked Write 10 Sent Done Done false
5449 2232 Acked Read 10 Sent Done Done false
5450 2233 Acked Flush 0 Sent Done Done false
5451 2234 Acked Read 9 Sent Done Done false
5452 2235 Acked Read 5 Sent Done Done false
5453 2236 Acked Flush 0 Sent Done Done false
5454 2237 Acked Read 8 Sent Done Done false
5455 2238 Acked Write 10 Sent Done Done false
5456 2239 Acked Write 9 Sent Done Done false
5457 2240 Acked Read 7 Sent Done Done false
5458 2241 Acked Write 5 Sent Done Done false
5459 2242 Acked Read 6 Sent Done Done false
5460 2243 Acked Write 5 Sent Done Done false
5461 2244 Acked Write 8 Sent Done Done false
5462 2245 Acked Read 4 Sent Done Done false
5463 2246 Acked Write 4 Sent Done Done false
5464 2247 Acked Read 10 Sent Done Done false
5465 2248 Acked Read 1 Sent Done Done false
5466 2249 Acked Flush 0 Sent Done Done false
5467 2250 Acked Read 9 Sent Done Done false
5468 2251 Acked Flush 0 Sent Done Done false
5469 2252 Acked Write 10 Sent Done Done false
5470 2253 Acked Flush 0 Sent Done Done false
5471 2254 Acked Read 8 Sent Done Done false
5472 2255 Acked Read 6 Sent Done Done false
5473 2256 Acked Read 3 Sent Done Done false
5474 2257 Acked Write 4 Sent Done Done false
5475 2258 Acked Read 6 Sent Done Done false
5476 2259 Acked Read 1 Sent Done Done false
5477 2260 Acked Read 5 Sent Done Done false
5478 2261 Acked Read 5 Sent Done Done false
5479 2262 Acked Read 1 Sent Done Done false
5480 2263 Acked Read 2 Sent Done Done false
5481 2264 Acked Read 5 Sent Done Done false
5482 2265 Acked Read 10 Sent Done Done false
5483 2266 Acked Write 5 Sent Done Done false
5484 2267 Acked Write 3 Sent Done Done false
5485 2268 Acked Read 1 Sent Done Done false
5486 2269 Acked Read 4 Sent Done Done false
5487 2270 Acked Read 5 Sent Done Done false
5488 2271 Acked Write 9 Sent Done Done false
5489 2272 Acked Flush 0 Sent Done Done false
5490 2273 Acked Flush 0 Sent Done Done false
5491 2274 Acked Write 5 Sent Done Done false
5492 2275 Acked Write 1 Sent Done Done false
5493 2276 Acked Write 10 Sent Done Done false
5494 2277 Acked Read 6 Sent Done Done false
5495 2278 Acked Read 10 Sent Done Done false
5496 2279 Acked Read 1 Sent Done Done false
5497 2280 Acked Read 5 Sent Done Done false
5498 2281 Acked Write 3 Sent Done Done false
5499 2282 Acked Flush 0 Sent Done Done false
5500 2283 Acked Read 2 Sent Done Done false
5501 2284 Acked Read 10 Sent Done Done false
5502 2285 Acked Flush 0 Sent Done Done false
5503 2286 Acked Read 3 Sent Done Done false
5504 2287 Acked Write 3 Sent Done Done false
5505 2288 Acked Write 9 Sent Done Done false
5506 2289 Acked Read 4 Sent Done Done false
5507 2290 Acked Read 7 Sent Done Done false
5508 2291 Acked Flush 0 Sent Done Done false
5509 2292 Acked Write 7 Sent Done Done false
5510 2293 Acked Read 3 Sent Done Done false
5511 2294 Acked Write 3 Sent Done Done false
5512 2295 Acked Flush 0 Sent Done Done false
5513 2296 Acked Read 3 Sent Done Done false
5514 2297 Acked Flush 0 Sent Done Done false
5515 2298 Acked Read 3 Sent Done Done false
5516 2299 Acked Read 7 Sent Done Done false
5517 2300 Acked Write 4 Sent Done Done false
5518 2301 Acked Write 3 Sent Done Done false
5519 2302 Acked Read 8 Sent Done Done false
5520 2303 Acked Write 1 Sent Done Done false
5521 2304 Acked Flush 0 Sent Done Done false
5522 2305 Acked Flush 0 Sent Done Done false
5523 2306 Acked Write 4 Sent Done Done false
5524 2307 Acked Write 7 Sent Done Done false
5525 2308 Acked Read 1 Sent Done Done false
5526 2309 Acked Read 9 Sent Done Done false
5527 2310 Acked Write 8 Sent Done Done false
5528 2311 Acked Flush 0 Sent Done Done false
5529 2312 Acked Write 7 Sent Done Done false
5530 2313 Acked Read 4 Sent Done Done false
5531 2314 Acked Read 2 Sent Done Done false
5532 2315 Acked Write 2 Sent Done Done false
5533 2316 Acked Write 2 Sent Done Done false
5534 2317 Acked Flush 0 Sent Done Done false
5535 2318 Acked Write 9 Sent Done Done false
5536 2319 Acked Write 6 Sent Done Done false
5537 2320 Acked Flush 0 Sent Done Done false
5538 2321 Acked Write 5 Sent Done Done false
5539 2322 Acked Write 1 Sent Done Done false
5540 2323 Acked Read 7 Sent Done Done false
5541 2324 Acked Write 10 Sent Done Done false
5542 2325 Acked Read 2 Sent Done Done false
5543 2326 Acked Read 6 Sent Done Done false
5544 2327 Acked Read 4 Sent Done Done false
5545 2328 Acked Write 3 Sent Done Done false
5546 2329 Acked Write 2 Sent Done Done false
5547 2330 Acked Read 4 Sent Done Done false
5548 2331 Acked Write 10 Sent Done Done false
5549 2332 Acked Write 4 Sent Done Done false
5550 2333 Acked Read 1 Sent Done Done false
5551 2334 Acked Write 6 Sent Done Done false
5552 2335 Acked Write 6 Sent Done Done false
5553 2336 Acked Read 9 Sent Done Done false
5554 2337 Acked Read 4 Sent Done Done false
5555 2338 Acked Write 4 Sent Done Done false
5556 2339 Acked Write 6 Sent Done Done false
5557 2340 Acked Write 2 Sent Done Done false
5558 2341 Acked Write 2 Sent Done Done false
5559 2342 Acked Read 6 Sent Done Done false
5560 2343 Acked Write 6 Sent Done Done false
5561 2344 Acked Write 2 Sent Done Done false
5562 2345 Acked Write 6 Sent Done Done false
5563 2346 Acked Read 7 Sent Done Done false
5564 2347 Acked Read 2 Sent Done Done false
5565 2348 Acked Read 5 Sent Done Done false
5566 2349 Acked Read 1 Sent Done Done false
5567 2350 Acked Write 6 Sent Done Done false
5568 2351 Acked Read 4 Sent Done Done false
5569 2352 Acked Read 10 Sent Done Done false
5570 2353 Acked Write 2 Sent Done Done false
5571 2354 Acked Read 10 Sent Done Done false
5572 2355 Acked Write 9 Sent Done Done false
5573 2356 Acked Read 9 Sent Done Done false
5574 2357 Acked Write 3 Sent Done Done false
5575 2358 Acked Write 10 Sent Done Done false
5576 2359 Acked Write 5 Sent Done Done false
5577 2360 Acked Read 10 Sent Done Done false
5578 2361 Acked Write 10 Sent Done Done false
5579 2362 Acked Read 10 Sent Done Done false
5580 2363 Acked Write 6 Sent Done Done false
5581 2364 Acked Flush 0 Sent Done Done false
5582 2365 Acked Read 2 Sent Done Done false
5583 2366 Acked Read 2 Sent Done Done false
5584 2367 Acked Read 9 Sent Done Done false
5585 2368 Acked Read 9 Sent Done Done false
5586 2369 Acked Write 4 Sent Done Done false
5587 2370 Acked Write 8 Sent Done Done false
5588 2371 Acked Read 8 Sent Done Done false
5589 2372 Acked Write 7 Sent Done Done false
5590 2373 Acked Read 2 Sent Done Done false
5591 2374 Acked Write 8 Sent Done Done false
5592 2375 Acked Read 10 Sent Done Done false
5593 2376 Acked Read 7 Sent Done Done false
5594 2377 Acked Write 5 Sent Done Done false
5595 2378 Acked Read 9 Sent Done Done false
5596 2379 Acked Flush 0 Sent Done Done false
5597 2380 Acked Flush 0 Sent Done Done false
5598 2381 Acked Read 3 Sent Done Done false
5599 2382 Acked Flush 0 Sent Done Done false
5600 2383 Acked Read 10 Sent Done Done false
5601 2384 Acked Flush 0 Sent Done Done false
5602 2385 Acked Write 9 Sent Done Done false
5603 2386 Acked Write 5 Sent Done Done false
5604 2387 Acked Read 2 Sent Done Done false
5605 2388 Acked Write 7 Sent Done Done false
5606 2389 Acked Flush 0 Sent Done Done false
5607 2390 Acked Read 6 Sent Done Done false
5608 2391 Acked Read 5 Sent Done Done false
5609 2392 Acked Read 10 Sent Done Done false
5610 2393 Acked Read 6 Sent Done Done false
5611 2394 Acked Read 7 Sent Done Done false
5612 2395 Acked Write 8 Sent Done Done false
5613 2396 Acked Read 5 Sent Done Done false
5614 2397 Acked Write 5 Sent Done Done false
5615 2398 Acked Write 8 Sent Done Done false
5616 2399 Acked Write 2 Sent Done Done false
5617 2400 Acked Flush 0 Sent Done Done false
5618 2401 Acked Read 5 Sent Done Done false
5619 2402 Acked Write 7 Sent Done Done false
5620 2403 Acked Read 2 Sent Done Done false
5621 2404 Acked Write 10 Sent Done Done false
5622 2405 Acked Read 9 Sent Done Done false
5623 2406 Acked Flush 0 Sent Done Done false
5624 2407 Acked Read 7 Sent Done Done false
5625 2408 Acked Write 9 Sent Done Done false
5626 2409 Acked Write 10 Sent Done Done false
5627 2410 Acked Flush 0 Sent Done Done false
5628 2411 Acked Write 4 Sent Done Done false
5629 2412 Acked Flush 0 Sent Done Done false
5630 2413 Acked Write 4 Sent Done Done false
5631 2414 Acked Write 5 Sent Done Done false
5632 2415 Acked Write 3 Sent Done Done false
5633 2416 Acked Read 7 Sent Done Done false
5634 2417 Acked Write 7 Sent Done Done false
5635 2418 Acked Flush 0 Sent Done Done false
5636 2419 Acked Read 8 Sent Done Done false
5637 2420 Acked Read 1 Sent Done Done false
5638 2421 Acked Write 3 Sent Done Done false
5639 2422 Acked Read 9 Sent Done Done false
5640 2423 Acked Write 6 Sent Done Done false
5641 2424 Acked Read 2 Sent Done Done false
5642 2425 Acked Read 10 Sent Done Done false
5643 2426 Acked Read 4 Sent Done Done false
5644 2427 Acked Read 1 Sent Done Done false
5645 2428 Acked Write 10 Sent Done Done false
5646 2429 Acked Read 7 Sent Done Done false
5647 2430 Acked Write 1 Sent Done Done false
5648 2431 Acked Write 8 Sent Done Done false
5649 2432 Acked Write 9 Sent Done Done false
5650 2433 Acked Write 10 Sent Done Done false
5651 2434 Acked Read 6 Sent Done Done false
5652 2435 Acked Write 2 Sent Done Done false
5653 2436 Acked Write 9 Sent Done Done false
5654 2437 Acked Read 5 Sent Done Done false
5655 2438 Acked Read 3 Sent Done Done false
5656 2439 Acked Read 1 Sent Done Done false
5657 2440 Acked Write 7 Sent Done Done false
5658 2441 Acked Write 7 Sent Done Done false
5659 2442 Acked Read 4 Sent Done Done false
5660 2443 Acked Write 9 Sent Done Done false
5661 2444 Acked Read 5 Sent Done Done false
5662 2445 Acked Write 8 Sent Done Done false
5663 2446 Acked Read 8 Sent Done Done false
5664 2447 Acked Read 1 Sent Done Done false
5665 2448 Acked Write 3 Sent Done Done false
5666 2449 Acked Write 9 Sent Done Done false
5667 2450 Acked Write 4 Sent Done Done false
5668 2451 Acked Write 8 Sent Done Done false
5669 2452 Acked Read 7 Sent Done Done false
5670 2453 Acked Read 7 Sent Done Done false
5671 2454 Acked Read 8 Sent Done Done false
5672 2455 Acked Read 6 Sent Done Done false
5673 2456 Acked Write 5 Sent Done Done false
5674 2457 Acked Flush 0 Sent Done Done false
5675 2458 Acked Read 2 Sent Done Done false
5676 2459 Acked Write 7 Sent Done Done false
5677 2460 Acked Read 6 Sent Done Done false
5678 2461 Acked Read 8 Sent Done Done false
5679 2462 Acked Write 8 Sent Done Done false
5680 2463 Acked Read 3 Sent Done Done false
5681 2464 Acked Read 10 Sent Done Done false
5682 2465 Acked Read 1 Sent Done Done false
5683 2466 Acked Read 7 Sent Done Done false
5684 2467 Acked Flush 0 Sent Done Done false
5685 2468 Acked Read 10 Sent Done Done false
5686 2469 Acked Read 10 Sent Done Done false
5687 2470 Acked Write 1 Sent Done Done false
5688 2471 Acked Read 4 Sent Done Done false
5689 2472 Acked Flush 0 Sent Done Done false
5690 2473 Acked Read 8 Sent Done Done false
5691 2474 Acked Write 4 Sent Done Done false
5692 2475 Acked Read 4 Sent Done Done false
5693 2476 Acked Read 10 Sent Done Done false
5694 2477 Acked Write 2 Sent Done Done false
5695 2478 Acked Write 3 Sent Done Done false
5696 2479 Acked Read 10 Sent Done Done false
5697 2480 Acked Write 1 Sent Done Done false
5698 2481 Acked Write 9 Sent Done Done false
5699 2482 Acked Write 10 Sent Done Done false
5700 2483 Acked Write 3 Sent Done Done false
5701 2484 Acked Read 2 Sent Done Done false
5702 2485 Acked Read 10 Sent Done Done false
5703 2486 Acked Read 8 Sent Done Done false
5704 2487 Acked Flush 0 Sent Done Done false
5705 2488 Acked Write 3 Sent Done Done false
5706 2489 Acked Read 2 Sent Done Done false
5707 2490 Acked Flush 0 Sent Done Done false
5708 2491 Acked Read 2 Sent Done Done false
5709 2492 Acked Read 1 Sent Done Done false
5710 2493 Acked Flush 0 Sent Done Done false
5711 2494 Acked Read 9 Sent Done Done false
5712 2495 Acked Read 4 Sent Done Done false
5713 2496 Acked Write 10 Sent Done Done false
5714 2497 Acked Read 7 Sent Done Done false
5715 2498 Acked Flush 0 Sent Done Done false
5716 2499 Acked Write 7 Sent Done Done false
5717 2500 Acked Write 2 Sent Done Done false
5718 2501 Acked Write 2 Sent Done Done false
5719 2502 Acked Write 10 Sent Done Done false
5720 2503 Acked Read 5 Sent Done Done false
5721 2504 Acked Write 7 Sent Done Done false
5722 2505 Acked Write 3 Sent Done Done false
5723 2506 Acked Write 7 Sent Done Done false
5724 2507 Acked Read 2 Sent Done Done false
5725 STATES DS:0 DS:1 DS:2 TOTAL
5726 Sent 301 0 0 301
5727 Done 0 301 301 602
5728 Skipped 0 0 0 0
5729 Error 0 0 0 0
5730 Last Flush: Some(JobId(2206)) Some(JobId(2498)) Some(JobId(2498))
5731 Downstairs last five completed: 2206 2205 2204 2203 2202
5732 Upstairs last five completed: 2507 2506 2505 2504 2503
5733 CLIENT: Up:0 ds:301 act:2
57342025-02-25T20:47:54.378ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57352025-02-25T20:47:55.179ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57362025-02-25T20:47:55.626ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57372025-02-25T20:47:55.626ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64482) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57382025-02-25T20:47:55.628ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57392025-02-25T20:47:55.628ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57402025-02-25T20:47:55.628ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57412025-02-25T20:47:55.628ZINFOcrucible: send last flush ID to this DS: Some(JobId(2206)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57422025-02-25T20:47:55.628ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57432025-02-25T20:47:55.628ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2206)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57442025-02-25T20:47:55.629ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2206)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
5745 ----------------------------------------------------------------
5746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5747 Downstairs last five completed: 2508 2507 2506 2505 2504
5748 Upstairs last five completed: 2508 2507 2506 2505 2504
5749 CLIENT: Up:0 ds:0 act:3
5750 Replay: All jobs finished
5751 Main task has work to do, go find it
5752 got dsc Stop(1)
5753 stop 1
5754 Feb 25 20:47:57.880 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/1[8820] Got stop action so:false kr:false
5755 , method: POST, req_id: b1918cf8-6215-41a2-8356-56685501e348, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
5756 Feb 25 20:47:57.881 INFO request completed, latency_us: 146, response_code: 200, uri: /state/cid/1, method: GET, req_id: ea201af1-f6cf-4186-b0f4-df10ede25f18, remote_addr: 127.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:47:57.881186649Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
5757 0.0.1:43930, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:47:57.881265934Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
5758 
57592025-02-25T20:47:57.881ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
57602025-02-25T20:47:57.881ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
5761 [8820] Exited with: ExitStatus(unix_wait_status(9))
5762 [8820][1] reports Exit
5763 Feb 25 20:48:01.883 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: ca748a1a-7822-411d-af44-5dfedf36bbef, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
5764 001/300 Write block 3883 len 40960 data: 4 4 4 4 2 2 2 2 2 2
5765 002/300 Read block 2992 len 20480
5766 003/300 Read block 108 len 32768
5767 004/300 Read block 428 len 28672
5768 005/300 Write block 3299 len 20480 data: 3 3 3 3 3
5769 006/300 Write block 2738 len 24576 data: 2 2 2 2 2 3
5770 007/300 Write block 194 len 20480 data: 2 2 2 2 2
5771 008/300 Write block 4141 len 24576 data: 3 4 3 3 3 2
5772 009/300 Write block 4908 len 8192 data: 2 3
5773 010/300 Read block 2108 len 28672
5774 011/300 Write block 1520 len 28672 data: 2 2 2 2 2 2 2
5775 012/300 Read block 1854 len 20480
5776 013/300 Write block 91 len 8192 data: 2 3
5777 014/300 Read block 3549 len 28672
5778 015/300 Read block 2176 len 40960
5779 016/300 Write block 2893 len 36864 data: 3 3 3 3 3 3 3 3 3
5780 017/300 Read block 1842 len 4096
5781 018/300 Read block 3766 len 32768
5782 019/300 Flush
5783 020/300 Write block 4480 len 8192 data: 2 2
5784 021/300 Write block 4694 len 4096 data: 3
5785 022/300 Read block 798 len 8192
5786 023/300 Read block 1552 len 12288
5787 024/300 Write block 2830 len 32768 data: 2 2 2 2 2 2 2 2
5788 025/300 Write block 3069 len 8192 data: 2 2
5789 026/300 Flush
5790 027/300 Read block 2962 len 36864
5791 028/300 Flush
5792 029/300 Read block 3202 len 20480
5793 030/300 Write block 4100 len 4096 data: 5
5794 031/300 Read block 3370 len 20480
5795 032/300 Read block 230 len 8192
5796 033/300 Write block 2622 len 20480 data: 2 3 3 4 4
5797 034/300 Read block 1526 len 8192
5798 035/300 Read block 93 len 16384
5799 036/300 Read block 1405 len 40960
5800 037/300 Read block 4401 len 32768
5801 038/300 Read block 4495 len 24576
5802 039/300 Flush
5803 040/300 Read block 1359 len 28672
5804 041/300 Write block 4886 len 8192 data: 3 3
5805 042/300 Read block 2501 len 28672
5806 043/300 Write block 2446 len 8192 data: 3 4
5807 044/300 Read block 4688 len 4096
5808 045/300 Read block 448 len 20480
5809 046/300 Read block 1113 len 16384
5810 047/300 Read block 3858 len 24576
5811 048/300 Read block 3913 len 28672
5812 049/300 Write block 1152 len 12288 data: 2 2 2
5813 050/300 Flush
5814 051/300 Write block 1121 len 20480 data: 3 3 3 3 3
5815 052/300 Read block 4676 len 16384
5816 053/300 Write block 3652 len 32768 data: 2 2 3 3 3 3 3 2
5817 054/300 Read block 1913 len 20480
5818 055/300 Write block 948 len 8192 data: 2 2
5819 056/300 Read block 3818 len 12288
5820 057/300 Read block 2881 len 36864
5821 058/300 Flush
5822 059/300 Flush
5823 060/300 Write block 2848 len 8192 data: 2 2
5824 061/300 Write block 4484 len 36864 data: 2 3 3 2 2 2 2 2 2
5825 062/300 Write block 3229 len 20480 data: 2 2 2 2 3
5826 063/300 Read block 1597 len 4096
5827 064/300 Read block 3277 len 16384
5828 065/300 Write block 3762 len 24576 data: 3 3 3 3 3 3
5829 066/300 Read block 2991 len 8192
5830 067/300 Write block 3769 len 8192 data: 3 3
5831 068/300 Write block 4917 len 24576 data: 3 3 3 3 3 2
5832 069/300 Read block 4703 len 40960
5833 070/300 Write block 3946 len 20480 data: 5 4 4 4 5
5834 071/300 Write block 495 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5835 072/300 Write block 4063 len 28672 data: 3 3 2 3 3 2 2
5836 073/300 Read block 2061 len 28672
5837 074/300 Read block 4381 len 4096
5838 075/300 Read block 1856 len 20480
5839 076/300 Read block 2550 len 36864
5840 077/300 Write block 2383 len 20480 data: 3 2 2 2 2
5841 078/300 Read block 1165 len 36864
5842 079/300 Write block 233 len 40960 data: 3 3 3 3 3 3 3 2 2 2
5843 080/300 Write block 4920 len 24576 data: 4 4 3 2 3 3
5844 081/300 Read block 1375 len 16384
5845 082/300 Read block 2478 len 36864
5846 083/300 Read block 3780 len 32768
5847 084/300 Write block 3278 len 40960 data: 4 3 3 4 3 3 3 2 2 2
5848 085/300 Write block 2208 len 12288 data: 5 4 3
5849 086/300 Read block 911 len 20480
5850 087/300 Read block 2478 len 4096
5851 088/300 Flush
5852 089/300 Read block 2197 len 28672
5853 090/300 Read block 3638 len 28672
5854 091/300 Write block 2860 len 16384 data: 2 2 2 2
5855 092/300 Read block 1258 len 4096
5856 093/300 Read block 4804 len 40960
5857 094/300 Read block 261 len 32768
5858 095/300 Read block 50 len 8192
5859 096/300 Flush
5860 097/300 Read block 498 len 4096
5861 098/300 Write block 3871 len 36864 data: 3 3 3 3 3 3 4 4 3
5862 099/300 Write block 1063 len 4096 data: 2
5863 100/300 Read block 1197 len 28672
5864 101/300 Write block 2386 len 8192 data: 3 3
5865 102/300 Write block 2838 len 32768 data: 2 2 2 2 2 2 2 2
5866 103/300 Write block 1391 len 40960 data: 2 2 3 3 3 2 2 2 2 2
5867 104/300 Write block 88 len 32768 data: 2 2 2 3 4 3 3 3
5868 105/300 Flush
5869 106/300 Read block 3448 len 8192
5870 107/300 Flush
5871 108/300 Read block 1150 len 40960
5872 109/300 Read block 1228 len 40960
5873 110/300 Read block 1568 len 4096
5874 111/300 Read block 3932 len 36864
5875 112/300 Read block 2953 len 16384
5876 113/300 Write block 2337 len 28672 data: 2 2 2 2 4 4 4
5877 114/300 Read block 827 len 36864
5878 115/300 Read block 3637 len 16384
5879 116/300 Flush
5880 117/300 Write block 3581 len 4096 data: 2
5881 118/300 Read block 3118 len 36864
5882 119/300 Write block 1202 len 20480 data: 2 2 3 3 3
5883 120/300 Read block 3576 len 24576
5884 121/300 Read block 4060 len 32768
5885 122/300 Write block 1862 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5886 123/300 Write block 1451 len 36864 data: 2 2 2 2 2 3 4 4 4
5887 124/300 Write block 3482 len 28672 data: 2 2 3 3 3 3 3
5888 125/300 Write block 2244 len 40960 data: 4 4 4 3 3 3 3 3 3 2
5889 126/300 Read block 1975 len 4096
5890 127/300 Read block 3336 len 16384
5891 128/300 Flush
5892 129/300 Read block 1212 len 20480
5893 130/300 Read block 702 len 12288
5894 131/300 Read block 397 len 32768
5895 132/300 Write block 2980 len 36864 data: 2 2 3 3 3 3 3 4 5
5896 133/300 Read block 2865 len 12288
5897 134/300 Write block 2830 len 4096 data: 3
5898 135/300 Write block 4211 len 32768 data: 2 2 2 2 2 2 3 3
5899 136/300 Write block 3704 len 20480 data: 2 3 3 4 4
5900 137/300 Flush
5901 138/300 Write block 2549 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5902 139/300 Flush
5903 140/300 Read block 4500 len 8192
5904 141/300 Write block 279 len 12288 data: 2 2 3
5905 142/300 Read block 3258 len 28672
5906 143/300 Write block 862 len 28672 data: 2 3 3 3 3 3 2
5907 144/300 Read block 4131 len 12288
5908 145/300 Write block 2538 len 40960 data: 3 3 3 3 3 3 3 3 3 2
5909 146/300 Read block 1977 len 4096
5910 147/300 Read block 1373 len 4096
5911 148/300 Write block 2007 len 32768 data: 2 2 4 4 2 2 2 2
5912 149/300 Read block 2207 len 40960
5913 150/300 Flush
5914 151/300 Read block 490 len 28672
5915 152/300 Write block 4939 len 12288 data: 2 2 2
5916 153/300 Flush
5917 154/300 Write block 600 len 24576 data: 2 2 2 2 2 2
5918 155/300 Write block 3417 len 24576 data: 3 3 3 3 3 4
5919 156/300 Flush
5920 157/300 Read block 2134 len 28672
5921 158/300 Read block 4265 len 40960
5922 159/300 Read block 3869 len 8192
5923 160/300 Read block 1752 len 8192
5924 161/300 Write block 3492 len 12288 data: 4 4 3
5925 162/300 Read block 938 len 16384
5926 163/300 Write block 412 len 24576 data: 2 2 2 2 2 3
5927 164/300 Read block 3496 len 28672
5928 165/300 Write block 282 len 4096 data: 3
5929 166/300 Read block 250 len 40960
5930 167/300 Write block 1534 len 12288 data: 5 5 4
5931 168/300 Write block 3334 len 20480 data: 2 2 2 2 2
5932 169/300 Flush
5933 170/300 Flush
5934 171/300 Read block 3445 len 12288
5935 172/300 Read block 3927 len 28672
5936 173/300 Read block 3576 len 20480
5937 174/300 Write block 3731 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5938 175/300 Read block 2240 len 12288
5939 176/300 Write block 1260 len 12288 data: 3 3 3
5940 177/300 Write block 3047 len 36864 data: 2 2 3 3 3 3 2 3 3
5941 178/300 Read block 4730 len 4096
5942 179/300 Write block 4704 len 8192 data: 2 2
5943 180/300 Read block 2099 len 12288
5944 181/300 Write block 3391 len 20480 data: 2 2 2 2 2
5945 182/300 Read block 1950 len 24576
5946 183/300 Write block 3942 len 16384 data: 2 3 3 4
5947 184/300 Write block 3298 len 36864 data: 3 4 4 4 4 4 3 3 2
5948 185/300 Read block 1863 len 12288
5949 186/300 Read block 247 len 40960
5950 187/300 Read block 3321 len 32768
5951 188/300 Flush
5952 189/300 Flush
5953 190/300 Write block 2745 len 16384 data: 3 3 3 3
5954 191/300 Read block 2355 len 36864
5955 192/300 Write block 3257 len 8192 data: 2 2
5956 193/300 Write block 656 len 40960 data: 2 2 2 2 3 3 3 3 3 3
5957 194/300 Read block 3806 len 32768
5958 195/300 Flush
5959 196/300 Write block 2737 len 28672 data: 2 3 3 3 3 3 4
5960 197/300 Read block 2101 len 40960
5961 198/300 Read block 1641 len 20480
5962 199/300 Read block 1264 len 20480
5963 200/300 Read block 2537 len 32768
5964 201/300 Write block 1669 len 20480 data: 3 3 3 4 4
5965 202/300 Write block 2229 len 16384 data: 3 3 3 3
5966 203/300 Flush
5967 204/300 Read block 1423 len 8192
5968 205/300 Read block 39 len 4096
5969 206/300 Read block 4795 len 40960
5970 207/300 Read block 506 len 24576
5971 208/300 Read block 2668 len 12288
5972 209/300 Flush
5973 210/300 Read block 1467 len 36864
5974 211/300 Read block 4030 len 20480
5975 212/300 Write block 3283 len 40960 data: 4 4 3 3 3 2 2 2 2 2
5976 213/300 Read block 961 len 8192
5977 214/300 Write block 1415 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5978 215/300 Write block 2889 len 12288 data: 4 5 2
5979 216/300 Read block 2570 len 8192
5980 217/300 Write block 1947 len 8192 data: 3 4
5981 218/300 Read block 4693 len 28672
5982 219/300 Write block 4876 len 8192 data: 3 3
5983 220/300 Read block 1357 len 32768
5984 221/300 Read block 3342 len 16384
5985 222/300 Write block 926 len 4096 data: 2
5986 223/300 Read block 409 len 40960
5987 224/300 Write block 127 len 28672 data: 5 3 3 3 2 2 2
5988 225/300 Write block 4715 len 40960 data: 2 3 2 2 2 3 3 3 3 3
5989 226/300 Flush
5990 227/300 Write block 795 len 28672 data: 2 2 2 2 2 2 2
5991 228/300 Read block 3272 len 32768
5992 229/300 Read block 4578 len 32768
5993 230/300 Read block 656 len 20480
5994 231/300 Read block 4041 len 8192
5995 232/300 Write block 1775 len 20480 data: 4 4 4 4 4
5996 233/300 Flush
5997 234/300 Write block 3442 len 36864 data: 3 3 3 2 2 2 2 2 2
5998 235/300 Write block 3798 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5999 236/300 Read block 2367 len 12288
6000 237/300 Flush
6001 238/300 Read block 4805 len 12288
6002 239/300 Write block 1323 len 28672 data: 4 4 5 4 3 3 3
6003 240/300 Read block 2291 len 36864
6004 241/300 Write block 2238 len 28672 data: 2 2 2 2 2 3 5
6005 242/300 Read block 4158 len 28672
6006 243/300 Read block 1526 len 8192
6007 244/300 Read block 1305 len 8192
6008 245/300 Read block 4700 len 28672
6009 246/300 Read block 3426 len 32768
6010 247/300 Flush
6011 248/300 Read block 1573 len 36864
6012 249/300 Flush
6013 250/300 Write block 916 len 40960 data: 2 2 2 3 3 3 3 3 3 2
6014 251/300 Read block 58 len 40960
6015 252/300 Read block 3216 len 12288
6016 253/300 Write block 2754 len 32768 data: 2 2 3 3 3 3 3 3
6017 254/300 Write block 1595 len 28672 data: 2 2 2 2 2 3 2
6018 255/300 Write block 738 len 8192 data: 2 2
6019 256/300 Read block 180 len 36864
6020 257/300 Write block 3062 len 12288 data: 2 2 3
6021 258/300 Read block 1974 len 16384
6022 259/300 Read block 4459 len 8192
6023 260/300 Write block 2988 len 20480 data: 6 4 4 4 4
6024 261/300 Write block 3333 len 32768 data: 2 3 3 3 3 3 2 2
6025 262/300 Read block 3651 len 16384
6026 263/300 Write block 445 len 24576 data: 2 2 2 2 2 3
6027 264/300 Flush
6028 265/300 Read block 1643 len 4096
6029 266/300 Flush
6030 267/300 Read block 3692 len 8192
6031 268/300 Write block 4940 len 24576 data: 3 3 2 2 2 2
6032 269/300 Read block 1870 len 28672
6033 270/300 Read block 3027 len 32768
6034 271/300 Write block 3242 len 12288 data: 4 3 3
6035 272/300 Write block 1788 len 24576 data: 2 2 2 2 3 3
6036 273/300 Write block 2845 len 12288 data: 3 2 2
6037 274/300 Flush
6038 275/300 Read block 894 len 4096
6039 276/300 Read block 2498 len 20480
6040 277/300 Write block 4968 len 20480 data: 4 3 3 3 2
6041 278/300 Write block 280 len 16384 data: 3 4 4 3
6042 279/300 Read block 4123 len 4096
6043 280/300 Read block 4419 len 4096
6044 281/300 Write block 2832 len 40960 data: 3 3 3 3 3 3 3 3 3 3
6045 282/300 Read block 3866 len 16384
6046 283/300 Read block 933 len 32768
6047 284/300 Read block 3794 len 12288
6048 285/300 Write block 2319 len 28672 data: 2 2 2 3 2 2 2
6049 286/300 Write block 3424 len 4096 data: 3
6050 287/300 Write block 4391 len 40960 data: 3 3 2 2 2 2 2 3 3 3
6051 288/300 Read block 2094 len 8192
6052 289/300 Read block 3292 len 28672
6053 290/300 Read block 2479 len 24576
6054 291/300 Read block 878 len 28672
6055 292/300 Read block 3134 len 12288
6056 293/300 Read block 1870 len 32768
6057 294/300 Read block 783 len 36864
6058 295/300 Read block 1142 len 4096
6059 296/300 Write block 3820 len 4096 data: 3
6060 297/300 Write block 3136 len 20480 data: 3 3 2 2 2
6061 298/300 Read block 627 len 12288
6062 299/300 Read block 2427 len 40960
6063 300/300 Read block 4862 len 8192
6064 301/300 Read block 2308 len 32768
6065 Main task has work to do, go find it
6066 got dsc Start(1)
6067 start 1
6068 Feb 25 20:48:02.129 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/1, method: POST, req_id: 39e04df5-c1b8-4650-afad-a6497b87b7a1, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
6069 Replay: started 1, returned:Ok(())
6070 ----------------------------------------------------------------
6071 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6072 States: Active Offline Active
6073 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6074 2509 Acked Write 10 Done Sent Done false
6075 2510 Acked Read 5 Done Sent Done false
6076 2511 Acked Read 8 Done Sent Done false
6077 2512 Acked Read 7 Done Sent Done false
6078 2513 Acked Write 5 Done Sent Done false
6079 2514 Acked Write 6 Done Sent Done false
6080 2515 Acked Write 5 Done Sent Done false
6081 2516 Acked Write 6 Done Sent Done false
6082 2517 Acked Write 2 Done Sent Done false
6083 2518 Acked Read 7 Done Sent Done false
6084 2519 Acked Write 7 Done Sent Done false
6085 2520 Acked Read 5 Done Sent Done false
6086 2521 Acked Write 2 Done Sent Done false
6087 2522 Acked Read 7 Done Sent Done false
6088 2523 Acked Read 10 Done Sent Done false
6089 2524 Acked Write 9 Done Sent Done false
6090 2525 Acked Read 1 Done Sent Done false
6091 2526 Acked Read 8 Done Sent Done false
6092 2527 Acked Flush 0 Done Sent Done false
6093 2528 Acked Write 2 Done Sent Done false
6094 2529 Acked Write 1 Done Sent Done false
6095 2530 Acked Read 2 Done Sent Done false
6096 2531 Acked Read 3 Done Sent Done false
6097 2532 Acked Write 8 Done Sent Done false
6098 2533 Acked Write 2 Done Sent Done false
6099 2534 Acked Flush 0 Done Sent Done false
6100 2535 Acked Read 9 Done Sent Done false
6101 2536 Acked Flush 0 Done Sent Done false
6102 2537 Acked Read 5 Done Sent Done false
6103 2538 Acked Write 1 Done Sent Done false
6104 2539 Acked Read 5 Done Sent Done false
6105 2540 Acked Read 2 Done Sent Done false
6106 2541 Acked Write 5 Done Sent Done false
6107 2542 Acked Read 2 Done Sent Done false
6108 2543 Acked Read 4 Done Sent Done false
6109 2544 Acked Read 10 Done Sent Done false
6110 2545 Acked Read 8 Done Sent Done false
6111 2546 Acked Read 6 Done Sent Done false
6112 2547 Acked Flush 0 Done Sent Done false
6113 2548 Acked Read 7 Done Sent Done false
6114 2549 Acked Write 2 Done Sent Done false
6115 2550 Acked Read 7 Done Sent Done false
6116 2551 Acked Write 2 Done Sent Done false
6117 2552 Acked Read 1 Done Sent Done false
6118 2553 Acked Read 5 Done Sent Done false
6119 2554 Acked Read 4 Done Sent Done false
6120 2555 Acked Read 6 Done Sent Done false
6121 2556 Acked Read 7 Done Sent Done false
6122 2557 Acked Write 3 Done Sent Done false
6123 2558 Acked Flush 0 Done Sent Done false
6124 2559 Acked Write 5 Done Sent Done false
6125 2560 Acked Read 4 Done Sent Done false
6126 2561 Acked Write 8 Done Sent Done false
6127 2562 Acked Read 5 Done Sent Done false
6128 2563 Acked Write 2 Done Sent Done false
6129 2564 Acked Read 3 Done Sent Done false
6130 2565 Acked Read 9 Done Sent Done false
6131 2566 Acked Flush 0 Done Sent Done false
6132 2567 Acked Flush 0 Done Sent Done false
6133 2568 Acked Write 2 Done Sent Done false
6134 2569 Acked Write 9 Done Sent Done false
6135 2570 Acked Write 5 Done Sent Done false
6136 2571 Acked Read 1 Done Sent Done false
6137 2572 Acked Read 4 Done Sent Done false
6138 2573 Acked Write 6 Done Sent Done false
6139 2574 Acked Read 2 Done Sent Done false
6140 2575 Acked Write 2 Done Sent Done false
6141 2576 Acked Write 6 Done Sent Done false
6142 2577 Acked Read 10 Done Sent Done false
6143 2578 Acked Write 5 Done Sent Done false
6144 2579 Acked Write 10 Done Sent Done false
6145 2580 Acked Write 7 Done Sent Done false
6146 2581 Acked Read 7 Done Sent Done false
6147 2582 Acked Read 1 Done Sent Done false
6148 2583 Acked Read 5 Done Sent Done false
6149 2584 Acked Read 9 Done Sent Done false
6150 2585 Acked Write 5 Done Sent Done false
6151 2586 Acked Read 9 Done Sent Done false
6152 2587 Acked Write 10 Done Sent Done false
6153 2588 Acked Write 6 Done Sent Done false
6154 2589 Acked Read 4 Done Sent Done false
6155 2590 Acked Read 9 Done Sent Done false
6156 2591 Acked Read 8 Done Sent Done false
6157 2592 Acked Write 10 Done Sent Done false
6158 2593 Acked Write 3 Done Sent Done false
6159 2594 Acked Read 5 Done Sent Done false
6160 2595 Acked Read 1 Done Sent Done false
6161 2596 Acked Flush 0 Done Sent Done false
6162 2597 Acked Read 7 Done Sent Done false
6163 2598 Acked Read 7 Done Sent Done false
6164 2599 Acked Write 4 Done Sent Done false
6165 2600 Acked Read 1 Done Sent Done false
6166 2601 Acked Read 10 Done Sent Done false
6167 2602 Acked Read 8 Done Sent Done false
6168 2603 Acked Read 2 Done Sent Done false
6169 2604 Acked Flush 0 Done Sent Done false
6170 2605 Acked Read 1 Done Sent Done false
6171 2606 Acked Write 9 Done Sent Done false
6172 2607 Acked Write 1 Done Sent Done false
6173 2608 Acked Read 7 Done Sent Done false
6174 2609 Acked Write 2 Done Sent Done false
6175 2610 Acked Write 8 Done Sent Done false
6176 2611 Acked Write 10 Done Sent Done false
6177 2612 Acked Write 8 Done Sent Done false
6178 2613 Acked Flush 0 Done Sent Done false
6179 2614 Acked Read 2 Done Sent Done false
6180 2615 Acked Flush 0 Done Sent Done false
6181 2616 Acked Read 10 Done Sent Done false
6182 2617 Acked Read 10 Done Sent Done false
6183 2618 Acked Read 1 Done Sent Done false
6184 2619 Acked Read 9 Done Sent Done false
6185 2620 Acked Read 4 Done Sent Done false
6186 2621 Acked Write 7 Done Sent Done false
6187 2622 Acked Read 9 Done Sent Done false
6188 2623 Acked Read 4 Done Sent Done false
6189 2624 Acked Flush 0 Done Sent Done false
6190 2625 Acked Write 1 Done Sent Done false
6191 2626 Acked Read 9 Done Sent Done false
6192 2627 Acked Write 5 Done Sent Done false
6193 2628 Acked Read 6 Done Sent Done false
6194 2629 Acked Read 8 Done Sent Done false
6195 2630 Acked Write 10 Done Sent Done false
6196 2631 Acked Write 9 Done Sent Done false
6197 2632 Acked Write 7 Done Sent Done false
6198 2633 Acked Write 10 Done Sent Done false
6199 2634 Acked Read 1 Done Sent Done false
6200 2635 Acked Read 4 Done Sent Done false
6201 2636 Acked Flush 0 Done Sent Done false
6202 2637 Acked Read 5 Done Sent Done false
6203 2638 Acked Read 3 Done Sent Done false
6204 2639 Acked Read 8 Done Sent Done false
6205 2640 Acked Write 9 Done Sent Done false
6206 2641 Acked Read 3 Done Sent Done false
6207 2642 Acked Write 1 Done Sent Done false
6208 2643 Acked Write 8 Done Sent Done false
6209 2644 Acked Write 5 Done Sent Done false
6210 2645 Acked Flush 0 Done Sent Done false
6211 2646 Acked Write 10 Done Sent Done false
6212 2647 Acked Flush 0 Done Sent Done false
6213 2648 Acked Read 2 Done Sent Done false
6214 2649 Acked Write 3 Done Sent Done false
6215 2650 Acked Read 7 Done Sent Done false
6216 2651 Acked Write 7 Done Sent Done false
6217 2652 Acked Read 3 Done Sent Done false
6218 2653 Acked Write 10 Done Sent Done false
6219 2654 Acked Read 1 Done Sent Done false
6220 2655 Acked Read 1 Done Sent Done false
6221 2656 Acked Write 8 Done Sent Done false
6222 2657 Acked Read 10 Done Sent Done false
6223 2658 Acked Flush 0 Done Sent Done false
6224 2659 Acked Read 7 Done Sent Done false
6225 2660 Acked Write 3 Done Sent Done false
6226 2661 Acked Flush 0 Done Sent Done false
6227 2662 Acked Write 6 Done Sent Done false
6228 2663 Acked Write 6 Done Sent Done false
6229 2664 Acked Flush 0 Done Sent Done false
6230 2665 Acked Read 7 Done Sent Done false
6231 2666 Acked Read 10 Done Sent Done false
6232 2667 Acked Read 2 Done Sent Done false
6233 2668 Acked Read 2 Done Sent Done false
6234 2669 Acked Write 3 Done Sent Done false
6235 2670 Acked Read 4 Done Sent Done false
6236 2671 Acked Write 6 Done Sent Done false
6237 2672 Acked Read 7 Done Sent Done false
6238 2673 Acked Write 1 Done Sent Done false
6239 2674 Acked Read 10 Done Sent Done false
6240 2675 Acked Write 3 Done Sent Done false
6241 2676 Acked Write 5 Done Sent Done false
6242 2677 Acked Flush 0 Done Sent Done false
6243 2678 Acked Flush 0 Done Sent Done false
6244 2679 Acked Read 3 Done Sent Done false
6245 2680 Acked Read 7 Done Sent Done false
6246 2681 Acked Read 5 Done Sent Done false
6247 2682 Acked Write 10 Done Sent Done false
6248 2683 Acked Read 3 Done Sent Done false
6249 2684 Acked Write 3 Done Sent Done false
6250 2685 Acked Write 9 Done Sent Done false
6251 2686 Acked Read 1 Done Sent Done false
6252 2687 Acked Write 2 Done Sent Done false
6253 2688 Acked Read 3 Done Sent Done false
6254 2689 Acked Write 5 Done Sent Done false
6255 2690 Acked Read 6 Done Sent Done false
6256 2691 Acked Write 4 Done Sent Done false
6257 2692 Acked Write 9 Done Sent Done false
6258 2693 Acked Read 3 Done Sent Done false
6259 2694 Acked Read 10 Done Sent Done false
6260 2695 Acked Read 8 Done Sent Done false
6261 2696 Acked Flush 0 Done Sent Done false
6262 2697 Acked Flush 0 Done Sent Done false
6263 2698 Acked Write 4 Done Sent Done false
6264 2699 Acked Read 9 Done Sent Done false
6265 2700 Acked Write 2 Done Sent Done false
6266 2701 Acked Write 10 Done Sent Done false
6267 2702 Acked Read 8 Done Sent Done false
6268 2703 Acked Flush 0 Done Sent Done false
6269 2704 Acked Write 7 Done Sent Done false
6270 2705 Acked Read 10 Done Sent Done false
6271 2706 Acked Read 5 Done Sent Done false
6272 2707 Acked Read 5 Done Sent Done false
6273 2708 Acked Read 8 Done Sent Done false
6274 2709 Acked Write 5 Done Sent Done false
6275 2710 Acked Write 4 Done Sent Done false
6276 2711 Acked Flush 0 Done Sent Done false
6277 2712 Acked Read 2 Done Sent Done false
6278 2713 Acked Read 1 Done Sent Done false
6279 2714 Acked Read 10 Done Sent Done false
6280 2715 Acked Read 6 Done Sent Done false
6281 2716 Acked Read 3 Done Sent Done false
6282 2717 Acked Flush 0 Done Sent Done false
6283 2718 Acked Read 9 Done Sent Done false
6284 2719 Acked Read 5 Done Sent Done false
6285 2720 Acked Write 10 Done Sent Done false
6286 2721 Acked Read 2 Done Sent Done false
6287 2722 Acked Write 10 Done Sent Done false
6288 2723 Acked Write 3 Done Sent Done false
6289 2724 Acked Read 2 Done Sent Done false
6290 2725 Acked Write 2 Done Sent Done false
6291 2726 Acked Read 7 Done Sent Done false
6292 2727 Acked Write 2 Done Sent Done false
6293 2728 Acked Read 8 Done Sent Done false
6294 2729 Acked Read 4 Done Sent Done false
6295 2730 Acked Write 1 Done Sent Done false
6296 2731 Acked Read 10 Done Sent Done false
6297 2732 Acked Write 7 Done Sent Done false
6298 2733 Acked Write 10 Done Sent Done false
6299 2734 Acked Flush 0 Done Sent Done false
6300 2735 Acked Write 7 Done Sent Done false
6301 2736 Acked Read 8 Done Sent Done false
6302 2737 Acked Read 8 Done Sent Done false
6303 2738 Acked Read 5 Done Sent Done false
6304 2739 Acked Read 2 Done Sent Done false
6305 2740 Acked Write 5 Done Sent Done false
6306 2741 Acked Flush 0 Done Sent Done false
6307 2742 Acked Write 9 Done Sent Done false
6308 2743 Acked Write 10 Done Sent Done false
6309 2744 Acked Read 3 Done Sent Done false
6310 2745 Acked Flush 0 Done Sent Done false
6311 2746 Acked Read 3 Done Sent Done false
6312 2747 Acked Write 7 Done Sent Done false
6313 2748 Acked Read 9 Done Sent Done false
6314 2749 Acked Write 7 Done Sent Done false
6315 2750 Acked Read 7 Done Sent Done false
6316 2751 Acked Read 2 Done Sent Done false
6317 2752 Acked Read 2 Done Sent Done false
6318 2753 Acked Read 7 Done Sent Done false
6319 2754 Acked Read 8 Done Sent Done false
6320 2755 Acked Flush 0 Done Sent Done false
6321 2756 Acked Read 9 Done Sent Done false
6322 2757 Acked Flush 0 Done Sent Done false
6323 2758 Acked Write 10 Done Sent Done false
6324 2759 Acked Read 10 Done Sent Done false
6325 2760 Acked Read 3 Done Sent Done false
6326 2761 Acked Write 8 Done Sent Done false
6327 2762 Acked Write 7 Done Sent Done false
6328 2763 Acked Write 2 Done Sent Done false
6329 2764 Acked Read 9 Done Sent Done false
6330 2765 Acked Write 3 Done Sent Done false
6331 2766 Acked Read 4 Done Sent Done false
6332 2767 Acked Read 2 Done Sent Done false
6333 2768 Acked Write 5 Done Sent Done false
6334 2769 Acked Write 8 Done Sent Done false
6335 2770 Acked Read 4 Done Sent Done false
6336 2771 Acked Write 6 Done Sent Done false
6337 2772 Acked Flush 0 Done Sent Done false
6338 2773 Acked Read 1 Done Sent Done false
6339 2774 Acked Flush 0 Done Sent Done false
6340 2775 Acked Read 2 Done Sent Done false
6341 2776 Acked Write 6 Done Sent Done false
6342 2777 Acked Read 7 Done Sent Done false
6343 2778 Acked Read 8 Done Sent Done false
6344 2779 Acked Write 3 Done Sent Done false
6345 2780 Acked Write 6 Done Sent Done false
6346 2781 Acked Write 3 Done Sent Done false
6347 2782 Acked Flush 0 Done Sent Done false
6348 2783 Acked Read 1 Done Sent Done false
6349 2784 Acked Read 5 Done Sent Done false
6350 2785 Acked Write 5 Done Sent Done false
6351 2786 Acked Write 4 Done Sent Done false
6352 2787 Acked Read 1 Done Sent Done false
6353 2788 Acked Read 1 Done Sent Done false
6354 2789 Acked Write 10 Done Sent Done false
6355 2790 Acked Read 4 Done Sent Done false
6356 2791 Acked Read 8 Done Sent Done false
6357 2792 Acked Read 3 Done Sent Done false
6358 2793 Acked Write 7 Done Sent Done false
6359 2794 Acked Write 1 Done Sent Done false
6360 2795 Acked Write 10 Done Sent Done false
6361 2796 Acked Read 2 Done Sent Done false
6362 2797 Acked Read 7 Done Sent Done false
6363 2798 Acked Read 6 Done Sent Done false
6364 2799 Acked Read 7 Done Sent Done false
6365 2800 Acked Read 3 Done Sent Done false
6366 2801 Acked Read 8 Done Sent Done false
6367 2802 Acked Read 9 Done Sent Done false
6368 2803 Acked Read 1 Done Sent Done false
6369 2804 Acked Write 1 Done Sent Done false
6370 2805 Acked Write 5 Done Sent Done false
6371 2806 Acked Read 3 Done Sent Done false
6372 2807 Acked Read 10 Done Sent Done false
6373 2808 Acked Read 2 Done Sent Done false
6374 2809 Acked Read 8 Done Sent Done false
6375 STATES DS:0 DS:1 DS:2 TOTAL
6376 Sent 0 301 0 301
6377 Done 301 0 301 602
6378 Skipped 0 0 0 0
6379 Error 0 0 0 0
6380 Last Flush: Some(JobId(2782)) Some(JobId(2508)) Some(JobId(2782))
6381 Downstairs last five completed: 2508 2507 2506 2505 2504
6382 Upstairs last five completed: 2809 2808 2807 2806 2805
6383 CLIENT: Up:0 ds:301 act:2
63842025-02-25T20:48:02.631ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
63852025-02-25T20:48:03.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
63862025-02-25T20:48:04.195ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
63872025-02-25T20:48:05.197ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
6388 [8820] got start action
6389 [8820] I am going to restart
6390 Starting downstairs at port 8820
6391 Make output file at "/tmp/dsc/downstairs-8820.txt"
6392 [8820][1] reports Starting
6393 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
6394 [8820][1] reports Running
6395 ----------------------------------------------------------------
6396 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6397 States: Active Offline Active
6398 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6399 2509 Acked Write 10 Done Sent Done false
6400 2510 Acked Read 5 Done Sent Done false
6401 2511 Acked Read 8 Done Sent Done false
6402 2512 Acked Read 7 Done Sent Done false
6403 2513 Acked Write 5 Done Sent Done false
6404 2514 Acked Write 6 Done Sent Done false
6405 2515 Acked Write 5 Done Sent Done false
6406 2516 Acked Write 6 Done Sent Done false
6407 2517 Acked Write 2 Done Sent Done false
6408 2518 Acked Read 7 Done Sent Done false
6409 2519 Acked Write 7 Done Sent Done false
6410 2520 Acked Read 5 Done Sent Done false
6411 2521 Acked Write 2 Done Sent Done false
6412 2522 Acked Read 7 Done Sent Done false
6413 2523 Acked Read 10 Done Sent Done false
6414 2524 Acked Write 9 Done Sent Done false
6415 2525 Acked Read 1 Done Sent Done false
6416 2526 Acked Read 8 Done Sent Done false
6417 2527 Acked Flush 0 Done Sent Done false
6418 2528 Acked Write 2 Done Sent Done false
6419 2529 Acked Write 1 Done Sent Done false
6420 2530 Acked Read 2 Done Sent Done false
6421 2531 Acked Read 3 Done Sent Done false
6422 2532 Acked Write 8 Done Sent Done false
6423 2533 Acked Write 2 Done Sent Done false
6424 2534 Acked Flush 0 Done Sent Done false
6425 2535 Acked Read 9 Done Sent Done false
6426 2536 Acked Flush 0 Done Sent Done false
6427 2537 Acked Read 5 Done Sent Done false
6428 2538 Acked Write 1 Done Sent Done false
6429 2539 Acked Read 5 Done Sent Done false
6430 2540 Acked Read 2 Done Sent Done false
6431 2541 Acked Write 5 Done Sent Done false
6432 2542 Acked Read 2 Done Sent Done false
6433 2543 Acked Read 4 Done Sent Done false
6434 2544 Acked Read 10 Done Sent Done false
6435 2545 Acked Read 8 Done Sent Done false
6436 2546 Acked Read 6 Done Sent Done false
6437 2547 Acked Flush 0 Done Sent Done false
6438 2548 Acked Read 7 Done Sent Done false
6439 2549 Acked Write 2 Done Sent Done false
6440 2550 Acked Read 7 Done Sent Done false
6441 2551 Acked Write 2 Done Sent Done false
6442 2552 Acked Read 1 Done Sent Done false
6443 2553 Acked Read 5 Done Sent Done false
6444 2554 Acked Read 4 Done Sent Done false
6445 2555 Acked Read 6 Done Sent Done false
6446 2556 Acked Read 7 Done Sent Done false
6447 2557 Acked Write 3 Done Sent Done false
6448 2558 Acked Flush 0 Done Sent Done false
6449 2559 Acked Write 5 Done Sent Done false
6450 2560 Acked Read 4 Done Sent Done false
6451 2561 Acked Write 8 Done Sent Done false
6452 2562 Acked Read 5 Done Sent Done false
6453 2563 Acked Write 2 Done Sent Done false
6454 2564 Acked Read 3 Done Sent Done false
6455 2565 Acked Read 9 Done Sent Done false
6456 2566 Acked Flush 0 Done Sent Done false
6457 2567 Acked Flush 0 Done Sent Done false
6458 2568 Acked Write 2 Done Sent Done false
6459 2569 Acked Write 9 Done Sent Done false
6460 2570 Acked Write 5 Done Sent Done false
6461 2571 Acked Read 1 Done Sent Done false
6462 2572 Acked Read 4 Done Sent Done false
6463 2573 Acked Write 6 Done Sent Done false
6464 2574 Acked Read 2 Done Sent Done false
6465 2575 Acked Write 2 Done Sent Done false
6466 2576 Acked Write 6 Done Sent Done false
6467 2577 Acked Read 10 Done Sent Done false
6468 2578 Acked Write 5 Done Sent Done false
6469 2579 Acked Write 10 Done Sent Done false
6470 2580 Acked Write 7 Done Sent Done false
6471 2581 Acked Read 7 Done Sent Done false
6472 2582 Acked Read 1 Done Sent Done false
6473 2583 Acked Read 5 Done Sent Done false
6474 2584 Acked Read 9 Done Sent Done false
6475 2585 Acked Write 5 Done Sent Done false
6476 2586 Acked Read 9 Done Sent Done false
6477 2587 Acked Write 10 Done Sent Done false
6478 2588 Acked Write 6 Done Sent Done false
6479 2589 Acked Read 4 Done Sent Done false
6480 2590 Acked Read 9 Done Sent Done false
6481 2591 Acked Read 8 Done Sent Done false
6482 2592 Acked Write 10 Done Sent Done false
6483 2593 Acked Write 3 Done Sent Done false
6484 2594 Acked Read 5 Done Sent Done false
6485 2595 Acked Read 1 Done Sent Done false
6486 2596 Acked Flush 0 Done Sent Done false
6487 2597 Acked Read 7 Done Sent Done false
6488 2598 Acked Read 7 Done Sent Done false
6489 2599 Acked Write 4 Done Sent Done false
6490 2600 Acked Read 1 Done Sent Done false
6491 2601 Acked Read 10 Done Sent Done false
6492 2602 Acked Read 8 Done Sent Done false
6493 2603 Acked Read 2 Done Sent Done false
6494 2604 Acked Flush 0 Done Sent Done false
6495 2605 Acked Read 1 Done Sent Done false
6496 2606 Acked Write 9 Done Sent Done false
6497 2607 Acked Write 1 Done Sent Done false
6498 2608 Acked Read 7 Done Sent Done false
6499 2609 Acked Write 2 Done Sent Done false
6500 2610 Acked Write 8 Done Sent Done false
6501 2611 Acked Write 10 Done Sent Done false
6502 2612 Acked Write 8 Done Sent Done false
6503 2613 Acked Flush 0 Done Sent Done false
6504 2614 Acked Read 2 Done Sent Done false
6505 2615 Acked Flush 0 Done Sent Done false
6506 2616 Acked Read 10 Done Sent Done false
6507 2617 Acked Read 10 Done Sent Done false
6508 2618 Acked Read 1 Done Sent Done false
6509 2619 Acked Read 9 Done Sent Done false
6510 2620 Acked Read 4 Done Sent Done false
6511 2621 Acked Write 7 Done Sent Done false
6512 2622 Acked Read 9 Done Sent Done false
6513 2623 Acked Read 4 Done Sent Done false
6514 2624 Acked Flush 0 Done Sent Done false
6515 2625 Acked Write 1 Done Sent Done false
6516 2626 Acked Read 9 Done Sent Done false
6517 2627 Acked Write 5 Done Sent Done false
6518 2628 Acked Read 6 Done Sent Done false
6519 2629 Acked Read 8 Done Sent Done false
6520 2630 Acked Write 10 Done Sent Done false
6521 2631 Acked Write 9 Done Sent Done false
6522 2632 Acked Write 7 Done Sent Done false
6523 2633 Acked Write 10 Done Sent Done false
6524 2634 Acked Read 1 Done Sent Done false
6525 2635 Acked Read 4 Done Sent Done false
6526 2636 Acked Flush 0 Done Sent Done false
6527 2637 Acked Read 5 Done Sent Done false
6528 2638 Acked Read 3 Done Sent Done false
6529 2639 Acked Read 8 Done Sent Done false
6530 2640 Acked Write 9 Done Sent Done false
6531 2641 Acked Read 3 Done Sent Done false
6532 2642 Acked Write 1 Done Sent Done false
6533 2643 Acked Write 8 Done Sent Done false
6534 2644 Acked Write 5 Done Sent Done false
6535 2645 Acked Flush 0 Done Sent Done false
6536 2646 Acked Write 10 Done Sent Done false
6537 2647 Acked Flush 0 Done Sent Done false
6538 2648 Acked Read 2 Done Sent Done false
6539 2649 Acked Write 3 Done Sent Done false
6540 2650 Acked Read 7 Done Sent Done false
6541 2651 Acked Write 7 Done Sent Done false
6542 2652 Acked Read 3 Done Sent Done false
6543 2653 Acked Write 10 Done Sent Done false
6544 2654 Acked Read 1 Done Sent Done false
6545 2655 Acked Read 1 Done Sent Done false
6546 2656 Acked Write 8 Done Sent Done false
6547 2657 Acked Read 10 Done Sent Done false
6548 2658 Acked Flush 0 Done Sent Done false
6549 2659 Acked Read 7 Done Sent Done false
6550 2660 Acked Write 3 Done Sent Done false
6551 2661 Acked Flush 0 Done Sent Done false
6552 2662 Acked Write 6 Done Sent Done false
6553 2663 Acked Write 6 Done Sent Done false
6554 2664 Acked Flush 0 Done Sent Done false
6555 2665 Acked Read 7 Done Sent Done false
6556 2666 Acked Read 10 Done Sent Done false
6557 2667 Acked Read 2 Done Sent Done false
6558 2668 Acked Read 2 Done Sent Done false
6559 2669 Acked Write 3 Done Sent Done false
6560 2670 Acked Read 4 Done Sent Done false
6561 2671 Acked Write 6 Done Sent Done false
6562 2672 Acked Read 7 Done Sent Done false
6563 2673 Acked Write 1 Done Sent Done false
6564 2674 Acked Read 10 Done Sent Done false
6565 2675 Acked Write 3 Done Sent Done false
6566 2676 Acked Write 5 Done Sent Done false
6567 2677 Acked Flush 0 Done Sent Done false
6568 2678 Acked Flush 0 Done Sent Done false
6569 2679 Acked Read 3 Done Sent Done false
6570 2680 Acked Read 7 Done Sent Done false
6571 2681 Acked Read 5 Done Sent Done false
6572 2682 Acked Write 10 Done Sent Done false
6573 2683 Acked Read 3 Done Sent Done false
6574 2684 Acked Write 3 Done Sent Done false
6575 2685 Acked Write 9 Done Sent Done false
6576 2686 Acked Read 1 Done Sent Done false
6577 2687 Acked Write 2 Done Sent Done false
6578 2688 Acked Read 3 Done Sent Done false
6579 2689 Acked Write 5 Done Sent Done false
6580 2690 Acked Read 6 Done Sent Done false
6581 2691 Acked Write 4 Done Sent Done false
6582 2692 Acked Write 9 Done Sent Done false
6583 2693 Acked Read 3 Done Sent Done false
6584 2694 Acked Read 10 Done Sent Done false
6585 2695 Acked Read 8 Done Sent Done false
6586 2696 Acked Flush 0 Done Sent Done false
6587 2697 Acked Flush 0 Done Sent Done false
6588 2698 Acked Write 4 Done Sent Done false
6589 2699 Acked Read 9 Done Sent Done false
6590 2700 Acked Write 2 Done Sent Done false
6591 2701 Acked Write 10 Done Sent Done false
6592 2702 Acked Read 8 Done Sent Done false
6593 2703 Acked Flush 0 Done Sent Done false
6594 2704 Acked Write 7 Done Sent Done false
6595 2705 Acked Read 10 Done Sent Done false
6596 2706 Acked Read 5 Done Sent Done false
6597 2707 Acked Read 5 Done Sent Done false
6598 2708 Acked Read 8 Done Sent Done false
6599 2709 Acked Write 5 Done Sent Done false
6600 2710 Acked Write 4 Done Sent Done false
6601 2711 Acked Flush 0 Done Sent Done false
6602 2712 Acked Read 2 Done Sent Done false
6603 2713 Acked Read 1 Done Sent Done false
6604 2714 Acked Read 10 Done Sent Done false
6605 2715 Acked Read 6 Done Sent Done false
6606 2716 Acked Read 3 Done Sent Done false
6607 2717 Acked Flush 0 Done Sent Done false
6608 2718 Acked Read 9 Done Sent Done false
6609 2719 Acked Read 5 Done Sent Done false
6610 2720 Acked Write 10 Done Sent Done false
6611 2721 Acked Read 2 Done Sent Done false
6612 2722 Acked Write 10 Done Sent Done false
6613 2723 Acked Write 3 Done Sent Done false
6614 2724 Acked Read 2 Done Sent Done false
6615 2725 Acked Write 2 Done Sent Done false
6616 2726 Acked Read 7 Done Sent Done false
6617 2727 Acked Write 2 Done Sent Done false
6618 2728 Acked Read 8 Done Sent Done false
6619 2729 Acked Read 4 Done Sent Done false
6620 2730 Acked Write 1 Done Sent Done false
6621 2731 Acked Read 10 Done Sent Done false
6622 2732 Acked Write 7 Done Sent Done false
6623 2733 Acked Write 10 Done Sent Done false
6624 2734 Acked Flush 0 Done Sent Done false
6625 2735 Acked Write 7 Done Sent Done false
6626 2736 Acked Read 8 Done Sent Done false
6627 2737 Acked Read 8 Done Sent Done false
6628 2738 Acked Read 5 Done Sent Done false
6629 2739 Acked Read 2 Done Sent Done false
6630 2740 Acked Write 5 Done Sent Done false
6631 2741 Acked Flush 0 Done Sent Done false
6632 2742 Acked Write 9 Done Sent Done false
6633 2743 Acked Write 10 Done Sent Done false
6634 2744 Acked Read 3 Done Sent Done false
6635 2745 Acked Flush 0 Done Sent Done false
6636 2746 Acked Read 3 Done Sent Done false
6637 2747 Acked Write 7 Done Sent Done false
6638 2748 Acked Read 9 Done Sent Done false
6639 2749 Acked Write 7 Done Sent Done false
6640 2750 Acked Read 7 Done Sent Done false
6641 2751 Acked Read 2 Done Sent Done false
6642 2752 Acked Read 2 Done Sent Done false
6643 2753 Acked Read 7 Done Sent Done false
6644 2754 Acked Read 8 Done Sent Done false
6645 2755 Acked Flush 0 Done Sent Done false
6646 2756 Acked Read 9 Done Sent Done false
6647 2757 Acked Flush 0 Done Sent Done false
6648 2758 Acked Write 10 Done Sent Done false
6649 2759 Acked Read 10 Done Sent Done false
6650 2760 Acked Read 3 Done Sent Done false
6651 2761 Acked Write 8 Done Sent Done false
6652 2762 Acked Write 7 Done Sent Done false
6653 2763 Acked Write 2 Done Sent Done false
6654 2764 Acked Read 9 Done Sent Done false
6655 2765 Acked Write 3 Done Sent Done false
6656 2766 Acked Read 4 Done Sent Done false
6657 2767 Acked Read 2 Done Sent Done false
6658 2768 Acked Write 5 Done Sent Done false
6659 2769 Acked Write 8 Done Sent Done false
6660 2770 Acked Read 4 Done Sent Done false
6661 2771 Acked Write 6 Done Sent Done false
6662 2772 Acked Flush 0 Done Sent Done false
6663 2773 Acked Read 1 Done Sent Done false
6664 2774 Acked Flush 0 Done Sent Done false
6665 2775 Acked Read 2 Done Sent Done false
6666 2776 Acked Write 6 Done Sent Done false
6667 2777 Acked Read 7 Done Sent Done false
6668 2778 Acked Read 8 Done Sent Done false
6669 2779 Acked Write 3 Done Sent Done false
6670 2780 Acked Write 6 Done Sent Done false
6671 2781 Acked Write 3 Done Sent Done false
6672 2782 Acked Flush 0 Done Sent Done false
6673 2783 Acked Read 1 Done Sent Done false
6674 2784 Acked Read 5 Done Sent Done false
6675 2785 Acked Write 5 Done Sent Done false
6676 2786 Acked Write 4 Done Sent Done false
6677 2787 Acked Read 1 Done Sent Done false
6678 2788 Acked Read 1 Done Sent Done false
6679 2789 Acked Write 10 Done Sent Done false
6680 2790 Acked Read 4 Done Sent Done false
6681 2791 Acked Read 8 Done Sent Done false
6682 2792 Acked Read 3 Done Sent Done false
6683 2793 Acked Write 7 Done Sent Done false
6684 2794 Acked Write 1 Done Sent Done false
6685 2795 Acked Write 10 Done Sent Done false
6686 2796 Acked Read 2 Done Sent Done false
6687 2797 Acked Read 7 Done Sent Done false
6688 2798 Acked Read 6 Done Sent Done false
6689 2799 Acked Read 7 Done Sent Done false
6690 2800 Acked Read 3 Done Sent Done false
6691 2801 Acked Read 8 Done Sent Done false
6692 2802 Acked Read 9 Done Sent Done false
6693 2803 Acked Read 1 Done Sent Done false
6694 2804 Acked Write 1 Done Sent Done false
6695 2805 Acked Write 5 Done Sent Done false
6696 2806 Acked Read 3 Done Sent Done false
6697 2807 Acked Read 10 Done Sent Done false
6698 2808 Acked Read 2 Done Sent Done false
6699 2809 Acked Read 8 Done Sent Done false
6700 STATES DS:0 DS:1 DS:2 TOTAL
6701 Sent 0 301 0 301
6702 Done 301 0 301 602
6703 Skipped 0 0 0 0
6704 Error 0 0 0 0
6705 Last Flush: Some(JobId(2782)) Some(JobId(2508)) Some(JobId(2782))
6706 Downstairs last five completed: 2508 2507 2506 2505 2504
6707 Upstairs last five completed: 2809 2808 2807 2806 2805
6708 CLIENT: Up:0 ds:301 act:2
67092025-02-25T20:48:06.634ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67102025-02-25T20:48:07.201ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67112025-02-25T20:48:07.882ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67122025-02-25T20:48:07.882ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48634) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67132025-02-25T20:48:07.884ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67142025-02-25T20:48:07.884ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67152025-02-25T20:48:07.884ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67162025-02-25T20:48:07.884ZINFOcrucible: send last flush ID to this DS: Some(JobId(2508)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67172025-02-25T20:48:07.884ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67182025-02-25T20:48:07.884ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2508)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67192025-02-25T20:48:07.884ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2508)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
6720 ----------------------------------------------------------------
6721 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6722 Downstairs last five completed: 2810 2809 2808 2807 2806
6723 Upstairs last five completed: 2810 2809 2808 2807 2806
6724 CLIENT: Up:0 ds:0 act:3
6725 Replay: All jobs finished
6726 Main task has work to do, go find it
6727 got dsc Stop(1)
6728 stop 1
6729 Feb 25 20:48:10.135 INFO request completed, latency_us: 378, response_code: 204, uri: /stop/cid/1, [8820] Got stop action so:false kr:false
6730 method: POST, req_id: c1df7505-99df-4548-a3a2-82fa4ee0d283, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
67312025-02-25T20:48:10.136ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67322025-02-25T20:48:10.136ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
67332025-02-25T20:48:10.136ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
6734 Feb 25 20:48:10.136 INFO request completed, latency_us: 129, response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:48:10.136398769Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
6735 , uri: /state/cid/1, method: GET, req_id: ea1eaf93-6451-4ed3-a682-c8b95951152c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
6736 [8820] Exited with: ExitStatus(unix_wait_status(9))
6737 [8820][1] reports Exit
6738 Feb 25 20:48:14.138 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/1, method: GET, req_id: f0a8f77c-c5d6-4d14-acce-b909422facd4, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
6739 001/300 Flush
6740 002/300 Read block 3496 len 36864
6741 003/300 Read block 1564 len 40960
6742 004/300 Read block 3480 len 8192
6743 005/300 Write block 1285 len 8192 data: 2 2
6744 006/300 Write block 4838 len 8192 data: 2 2
6745 007/300 Read block 1823 len 40960
6746 008/300 Write block 471 len 12288 data: 2 2 2
6747 009/300 Read block 2139 len 12288
6748 010/300 Write block 4094 len 24576 data: 3 2 2 2 2 4
6749 011/300 Read block 602 len 8192
6750 012/300 Read block 446 len 20480
6751 013/300 Read block 610 len 12288
6752 014/300 Flush
6753 015/300 Write block 1812 len 16384 data: 3 4 4 4
6754 016/300 Read block 1209 len 32768
6755 017/300 Flush
6756 018/300 Read block 4674 len 8192
6757 019/300 Read block 1859 len 16384
6758 020/300 Flush
6759 021/300 Flush
6760 022/300 Read block 3456 len 8192
6761 023/300 Write block 2803 len 28672 data: 3 3 3 2 2 2 2
6762 024/300 Read block 2216 len 24576
6763 025/300 Write block 2333 len 16384 data: 3 2 2 2
6764 026/300 Flush
6765 027/300 Flush
6766 028/300 Write block 2187 len 40960 data: 3 3 3 3 3 3 3 3 2 2
6767 029/300 Read block 1295 len 12288
6768 030/300 Flush
6769 031/300 Read block 2204 len 40960
6770 032/300 Write block 2310 len 20480 data: 2 2 2 2 2
6771 033/300 Flush
6772 034/300 Flush
6773 035/300 Flush
6774 036/300 Read block 2856 len 36864
6775 037/300 Write block 2197 len 4096 data: 2
6776 038/300 Write block 2968 len 28672 data: 2 2 2 2 2 2 2
6777 039/300 Read block 1331 len 28672
6778 040/300 Read block 3127 len 20480
6779 041/300 Read block 3370 len 16384
6780 042/300 Read block 1343 len 8192
6781 043/300 Read block 2225 len 28672
6782 044/300 Write block 2106 len 12288 data: 3 4 4
6783 045/300 Read block 4607 len 16384
6784 046/300 Write block 2326 len 40960 data: 2 2 2 3 3 3 3 4 3 3
6785 047/300 Read block 233 len 8192
6786 048/300 Read block 1910 len 24576
6787 049/300 Write block 2848 len 24576 data: 3 3 2 2 2 3
6788 050/300 Read block 3928 len 8192
6789 051/300 Read block 1626 len 40960
6790 052/300 Write block 3999 len 20480 data: 3 3 3 3 3
6791 053/300 Write block 2438 len 8192 data: 2 3
6792 054/300 Read block 662 len 24576
6793 055/300 Read block 1316 len 24576
6794 056/300 Read block 2596 len 12288
6795 057/300 Read block 4599 len 32768
6796 058/300 Read block 1302 len 36864
6797 059/300 Flush
6798 060/300 Read block 3778 len 28672
6799 061/300 Read block 986 len 36864
6800 062/300 Read block 1605 len 8192
6801 063/300 Write block 226 len 20480 data: 2 2 2 2 2
6802 064/300 Write block 4874 len 40960 data: 4 3 4 4 3 3 3 4 3 4
6803 065/300 Flush
6804 066/300 Write block 2719 len 8192 data: 4 4
6805 067/300 Write block 3640 len 20480 data: 4 4 5 5 5
6806 068/300 Read block 3271 len 16384
6807 069/300 Write block 1823 len 28672 data: 2 2 3 3 3 3 3
6808 070/300 Write block 4361 len 32768 data: 2 2 2 2 2 2 2 3
6809 071/300 Flush
6810 072/300 Flush
6811 073/300 Write block 4507 len 40960 data: 3 3 3 3 3 3 3 3 2 2
6812 074/300 Read block 232 len 36864
6813 075/300 Read block 3385 len 4096
6814 076/300 Flush
6815 077/300 Read block 551 len 24576
6816 078/300 Read block 2823 len 28672
6817 079/300 Read block 4126 len 40960
6818 080/300 Write block 320 len 20480 data: 2 3 3 2 2
6819 081/300 Write block 3101 len 28672 data: 3 3 3 3 4 3 3
6820 082/300 Write block 3978 len 20480 data: 2 2 2 2 3
6821 083/300 Flush
6822 084/300 Read block 815 len 24576
6823 085/300 Flush
6824 086/300 Flush
6825 087/300 Read block 1660 len 32768
6826 088/300 Read block 3321 len 8192
6827 089/300 Write block 1855 len 12288 data: 2 2 2
6828 090/300 Read block 4230 len 36864
6829 091/300 Write block 2907 len 8192 data: 4 4
6830 092/300 Write block 2243 len 32768 data: 4 6 5 5 4 4 4 4
6831 093/300 Read block 1935 len 36864
6832 094/300 Write block 3960 len 8192 data: 2 2
6833 095/300 Read block 2429 len 8192
6834 096/300 Read block 4 len 36864
6835 097/300 Flush
6836 098/300 Write block 2584 len 12288 data: 3 3 3
6837 099/300 Flush
6838 100/300 Write block 3303 len 12288 data: 5 4 4
6839 101/300 Write block 176 len 28672 data: 2 2 2 2 2 2 3
6840 102/300 Read block 2436 len 24576
6841 103/300 Flush
6842 104/300 Read block 1290 len 28672
6843 105/300 Write block 226 len 12288 data: 3 3 3
6844 106/300 Write block 2792 len 20480 data: 2 2 2 2 2
6845 107/300 Write block 2165 len 28672 data: 2 2 2 2 2 2 2
6846 108/300 Read block 4517 len 40960
6847 109/300 Read block 1616 len 24576
6848 110/300 Read block 3426 len 32768
6849 111/300 Write block 4420 len 40960 data: 3 4 5 3 4 4 5 5 5 5
6850 112/300 Write block 691 len 12288 data: 3 3 3
6851 113/300 Write block 357 len 24576 data: 3 2 2 2 2 2
6852 114/300 Read block 1407 len 28672
6853 115/300 Write block 4226 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6854 116/300 Read block 1619 len 12288
6855 117/300 Write block 4065 len 4096 data: 3
6856 118/300 Read block 2699 len 40960
6857 119/300 Write block 3181 len 24576 data: 4 4 4 3 2 2
6858 120/300 Read block 4198 len 28672
6859 121/300 Write block 2590 len 28672 data: 2 2 2 2 2 2 2
6860 122/300 Read block 4879 len 24576
6861 123/300 Read block 2031 len 8192
6862 124/300 Write block 2112 len 36864 data: 4 3 3 3 3 3 3 3 3
6863 125/300 Write block 3659 len 20480 data: 3 3 3 3 3
6864 126/300 Read block 3497 len 16384
6865 127/300 Read block 4435 len 8192
6866 128/300 Read block 4695 len 8192
6867 129/300 Flush
6868 130/300 Read block 3645 len 36864
6869 131/300 Flush
6870 132/300 Flush
6871 133/300 Read block 3808 len 8192
6872 134/300 Write block 3185 len 12288 data: 3 3 2
6873 135/300 Read block 4159 len 12288
6874 136/300 Read block 249 len 36864
6875 137/300 Flush
6876 138/300 Read block 3818 len 12288
6877 139/300 Write block 4755 len 32768 data: 3 3 3 3 3 2 2 2
6878 140/300 Read block 3604 len 8192
6879 141/300 Write block 2470 len 32768 data: 3 4 3 3 2 2 2 2
6880 142/300 Write block 3774 len 32768 data: 4 4 4 3 3 3 3 3
6881 143/300 Read block 3284 len 24576
6882 144/300 Flush
6883 145/300 Read block 63 len 4096
6884 146/300 Write block 1041 len 24576 data: 2 3 3 3 3 3
6885 147/300 Read block 1624 len 16384
6886 148/300 Read block 3081 len 32768
6887 149/300 Write block 1171 len 4096 data: 2
6888 150/300 Write block 1991 len 16384 data: 4 4 4 4
6889 151/300 Read block 23 len 36864
6890 152/300 Read block 4623 len 12288
6891 153/300 Read block 3332 len 40960
6892 154/300 Write block 4015 len 4096 data: 2
6893 155/300 Flush
6894 156/300 Read block 226 len 36864
6895 157/300 Write block 47 len 12288 data: 3 3 3
6896 158/300 Read block 3719 len 36864
6897 159/300 Read block 2144 len 24576
6898 160/300 Write block 2836 len 20480 data: 4 4 4 4 4
6899 161/300 Read block 4905 len 16384
6900 162/300 Write block 2548 len 24576 data: 2 3 3 3 3 3
6901 163/300 Write block 352 len 40960 data: 3 3 3 3 3 4 3 3 3 3
6902 164/300 Read block 3547 len 28672
6903 165/300 Write block 2132 len 32768 data: 2 2 2 2 2 2 2 2
6904 166/300 Read block 3431 len 16384
6905 167/300 Write block 3725 len 32768 data: 2 2 2 2 2 2 3 3
6906 168/300 Flush
6907 169/300 Read block 4429 len 16384
6908 170/300 Read block 416 len 16384
6909 171/300 Write block 4794 len 24576 data: 3 4 4 4 4 4
6910 172/300 Flush
6911 173/300 Write block 2872 len 8192 data: 3 3
6912 174/300 Read block 4718 len 24576
6913 175/300 Read block 1335 len 36864
6914 176/300 Read block 2803 len 32768
6915 177/300 Write block 3620 len 16384 data: 3 3 3 3
6916 178/300 Read block 3373 len 4096
6917 179/300 Read block 4704 len 4096
6918 180/300 Read block 1543 len 36864
6919 181/300 Flush
6920 182/300 Read block 4844 len 40960
6921 183/300 Flush
6922 184/300 Read block 3327 len 24576
6923 185/300 Write block 2377 len 12288 data: 3 3 3
6924 186/300 Write block 2892 len 32768 data: 2 4 4 4 4 4 4 4
6925 187/300 Read block 1389 len 40960
6926 188/300 Write block 1270 len 24576 data: 2 2 2 2 2 2
6927 189/300 Read block 3079 len 4096
6928 190/300 Read block 4476 len 32768
6929 191/300 Read block 3661 len 24576
6930 192/300 Read block 3921 len 12288
6931 193/300 Flush
6932 194/300 Read block 2212 len 36864
6933 195/300 Read block 3237 len 12288
6934 196/300 Write block 1505 len 20480 data: 3 3 2 2 2
6935 197/300 Read block 3788 len 32768
6936 198/300 Write block 3880 len 40960 data: 3 3 4 5 5 5 5 3 3 3
6937 199/300 Write block 4046 len 28672 data: 2 2 2 2 2 2 2
6938 200/300 Write block 3617 len 4096 data: 4
6939 201/300 Write block 1000 len 28672 data: 2 2 2 3 3 3 2
6940 202/300 Flush
6941 203/300 Read block 4286 len 20480
6942 204/300 Write block 3156 len 40960 data: 4 4 3 3 3 2 2 2 3 4
6943 205/300 Read block 560 len 4096
6944 206/300 Flush
6945 207/300 Write block 1843 len 12288 data: 2 2 2
6946 208/300 Read block 1384 len 16384
6947 209/300 Write block 1974 len 4096 data: 3
6948 210/300 Read block 504 len 20480
6949 211/300 Read block 1402 len 24576
6950 212/300 Read block 4448 len 24576
6951 213/300 Read block 998 len 32768
6952 214/300 Read block 4328 len 28672
6953 215/300 Read block 584 len 40960
6954 216/300 Write block 854 len 4096 data: 3
6955 217/300 Read block 4175 len 4096
6956 218/300 Flush
6957 219/300 Flush
6958 220/300 Read block 39 len 8192
6959 221/300 Read block 1706 len 36864
6960 222/300 Flush
6961 223/300 Write block 2999 len 20480 data: 3 3 2 2 3
6962 224/300 Write block 3384 len 12288 data: 4 4 4
6963 225/300 Read block 1701 len 36864
6964 226/300 Read block 4423 len 36864
6965 227/300 Write block 841 len 20480 data: 3 5 5 5 5
6966 228/300 Read block 2146 len 4096
6967 229/300 Read block 2915 len 36864
6968 230/300 Write block 4069 len 4096 data: 3
6969 231/300 Write block 1768 len 40960 data: 2 2 3 3 3 3 3 5 5 5
6970 232/300 Write block 1195 len 32768 data: 2 2 2 2 2 2 2 3
6971 233/300 Write block 3376 len 24576 data: 3 3 3 3 4 5
6972 234/300 Write block 4796 len 16384 data: 5 5 5 5
6973 235/300 Read block 3181 len 28672
6974 236/300 Read block 520 len 28672
6975 237/300 Flush
6976 238/300 Flush
6977 239/300 Flush
6978 240/300 Flush
6979 241/300 Write block 1007 len 12288 data: 2 2 2
6980 242/300 Read block 104 len 24576
6981 243/300 Flush
6982 244/300 Read block 1853 len 36864
6983 245/300 Flush
6984 246/300 Flush
6985 247/300 Write block 4347 len 28672 data: 2 3 3 3 2 2 2
6986 248/300 Read block 308 len 28672
6987 249/300 Read block 3619 len 40960
6988 250/300 Read block 3439 len 32768
6989 251/300 Read block 1347 len 24576
6990 252/300 Flush
6991 253/300 Write block 3757 len 40960 data: 2 2 3 3 3 4 4 4 4 4
6992 254/300 Write block 4598 len 4096 data: 2
6993 255/300 Read block 2546 len 28672
6994 256/300 Write block 2350 len 40960 data: 4 4 3 3 3 3 3 3 2 2
6995 257/300 Read block 3979 len 4096
6996 258/300 Write block 3198 len 8192 data: 3 3
6997 259/300 Write block 4344 len 4096 data: 3
6998 260/300 Write block 2708 len 4096 data: 5
6999 261/300 Write block 3899 len 28672 data: 2 2 3 3 3 3 3
7000 262/300 Read block 1666 len 24576
7001 263/300 Write block 4935 len 20480 data: 2 2 2 2 3
7002 264/300 Read block 3055 len 8192
7003 265/300 Flush
7004 266/300 Read block 2467 len 4096
7005 267/300 Write block 4161 len 24576 data: 3 3 4 4 5 6
7006 268/300 Write block 1454 len 36864 data: 3 3 4 5 5 5 4 4 5
7007 269/300 Write block 3011 len 24576 data: 3 3 2 2 2 2
7008 270/300 Write block 2337 len 40960 data: 3 3 3 3 5 5 5 3 3 2
7009 271/300 Write block 256 len 32768 data: 3 3 2 2 2 2 3 3
7010 272/300 Read block 4930 len 4096
7011 273/300 Write block 3421 len 28672 data: 4 5 4 4 3 3 3
7012 274/300 Read block 3951 len 24576
7013 275/300 Write block 624 len 8192 data: 3 3
7014 276/300 Read block 1448 len 20480
7015 277/300 Read block 2046 len 28672
7016 278/300 Write block 2650 len 20480 data: 3 3 2 2 2
7017 279/300 Flush
7018 280/300 Read block 372 len 36864
7019 281/300 Write block 856 len 4096 data: 3
7020 282/300 Read block 2677 len 12288
7021 283/300 Write block 4391 len 4096 data: 4
7022 284/300 Read block 865 len 8192
7023 285/300 Read block 2535 len 20480
7024 286/300 Write block 3126 len 40960 data: 2 2 2 2 2 2 2 2 3 3
7025 287/300 Read block 2776 len 28672
7026 288/300 Read block 4453 len 36864
7027 289/300 Flush
7028 290/300 Read block 2077 len 40960
7029 291/300 Write block 3008 len 24576 data: 4 5 4 4 4 3
7030 292/300 Read block 1896 len 16384
7031 293/300 Write block 1857 len 36864 data: 3 2 2 3 3 4 4 4 3
7032 294/300 Write block 397 len 4096 data: 3
7033 295/300 Read block 2759 len 24576
7034 296/300 Flush
7035 297/300 Write block 211 len 4096 data: 3
7036 298/300 Read block 1373 len 4096
7037 299/300 Flush
7038 300/300 Read block 2976 len 16384
7039 301/300 Write block 524 len 12288 data: 2 2 2
7040 Main task has work to do, go find it
7041 got dsc Start(1)
7042 start 1
7043 Feb 25 20:48:14.381 INFO request completed, latency_us: 240, response_code: 204, uri: /start/cid/1, method: POST, req_id: d5850745-68eb-4fb7-9a1e-3c3685e3fff4, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
7044 Replay: started 1, returned:Ok(())
7045 ----------------------------------------------------------------
7046 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7047 States: Active Offline Active
7048 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7049 2811 Acked Flush 0 Done Sent Done false
7050 2812 Acked Read 9 Done Sent Done false
7051 2813 Acked Read 10 Done Sent Done false
7052 2814 Acked Read 2 Done Sent Done false
7053 2815 Acked Write 2 Done Sent Done false
7054 2816 Acked Write 2 Done Sent Done false
7055 2817 Acked Read 10 Done Sent Done false
7056 2818 Acked Write 3 Done Sent Done false
7057 2819 Acked Read 3 Done Sent Done false
7058 2820 Acked Write 6 Done Sent Done false
7059 2821 Acked Read 2 Done Sent Done false
7060 2822 Acked Read 5 Done Sent Done false
7061 2823 Acked Read 3 Done Sent Done false
7062 2824 Acked Flush 0 Done Sent Done false
7063 2825 Acked Write 4 Done Sent Done false
7064 2826 Acked Read 8 Done Sent Done false
7065 2827 Acked Flush 0 Done Sent Done false
7066 2828 Acked Read 2 Done Sent Done false
7067 2829 Acked Read 4 Done Sent Done false
7068 2830 Acked Flush 0 Done Sent Done false
7069 2831 Acked Flush 0 Done Sent Done false
7070 2832 Acked Read 2 Done Sent Done false
7071 2833 Acked Write 7 Done Sent Done false
7072 2834 Acked Read 6 Done Sent Done false
7073 2835 Acked Write 4 Done Sent Done false
7074 2836 Acked Flush 0 Done Sent Done false
7075 2837 Acked Flush 0 Done Sent Done false
7076 2838 Acked Write 10 Done Sent Done false
7077 2839 Acked Read 3 Done Sent Done false
7078 2840 Acked Flush 0 Done Sent Done false
7079 2841 Acked Read 10 Done Sent Done false
7080 2842 Acked Write 5 Done Sent Done false
7081 2843 Acked Flush 0 Done Sent Done false
7082 2844 Acked Flush 0 Done Sent Done false
7083 2845 Acked Flush 0 Done Sent Done false
7084 2846 Acked Read 9 Done Sent Done false
7085 2847 Acked Write 1 Done Sent Done false
7086 2848 Acked Write 7 Done Sent Done false
7087 2849 Acked Read 7 Done Sent Done false
7088 2850 Acked Read 5 Done Sent Done false
7089 2851 Acked Read 4 Done Sent Done false
7090 2852 Acked Read 2 Done Sent Done false
7091 2853 Acked Read 7 Done Sent Done false
7092 2854 Acked Write 3 Done Sent Done false
7093 2855 Acked Read 4 Done Sent Done false
7094 2856 Acked Write 10 Done Sent Done false
7095 2857 Acked Read 2 Done Sent Done false
7096 2858 Acked Read 6 Done Sent Done false
7097 2859 Acked Write 6 Done Sent Done false
7098 2860 Acked Read 2 Done Sent Done false
7099 2861 Acked Read 10 Done Sent Done false
7100 2862 Acked Write 5 Done Sent Done false
7101 2863 Acked Write 2 Done Sent Done false
7102 2864 Acked Read 6 Done Sent Done false
7103 2865 Acked Read 6 Done Sent Done false
7104 2866 Acked Read 3 Done Sent Done false
7105 2867 Acked Read 8 Done Sent Done false
7106 2868 Acked Read 9 Done Sent Done false
7107 2869 Acked Flush 0 Done Sent Done false
7108 2870 Acked Read 7 Done Sent Done false
7109 2871 Acked Read 9 Done Sent Done false
7110 2872 Acked Read 2 Done Sent Done false
7111 2873 Acked Write 5 Done Sent Done false
7112 2874 Acked Write 10 Done Sent Done false
7113 2875 Acked Flush 0 Done Sent Done false
7114 2876 Acked Write 2 Done Sent Done false
7115 2877 Acked Write 5 Done Sent Done false
7116 2878 Acked Read 4 Done Sent Done false
7117 2879 Acked Write 7 Done Sent Done false
7118 2880 Acked Write 8 Done Sent Done false
7119 2881 Acked Flush 0 Done Sent Done false
7120 2882 Acked Flush 0 Done Sent Done false
7121 2883 Acked Write 10 Done Sent Done false
7122 2884 Acked Read 9 Done Sent Done false
7123 2885 Acked Read 1 Done Sent Done false
7124 2886 Acked Flush 0 Done Sent Done false
7125 2887 Acked Read 6 Done Sent Done false
7126 2888 Acked Read 7 Done Sent Done false
7127 2889 Acked Read 10 Done Sent Done false
7128 2890 Acked Write 5 Done Sent Done false
7129 2891 Acked Write 7 Done Sent Done false
7130 2892 Acked Write 5 Done Sent Done false
7131 2893 Acked Flush 0 Done Sent Done false
7132 2894 Acked Read 6 Done Sent Done false
7133 2895 Acked Flush 0 Done Sent Done false
7134 2896 Acked Flush 0 Done Sent Done false
7135 2897 Acked Read 8 Done Sent Done false
7136 2898 Acked Read 2 Done Sent Done false
7137 2899 Acked Write 3 Done Sent Done false
7138 2900 Acked Read 9 Done Sent Done false
7139 2901 Acked Write 2 Done Sent Done false
7140 2902 Acked Write 8 Done Sent Done false
7141 2903 Acked Read 9 Done Sent Done false
7142 2904 Acked Write 2 Done Sent Done false
7143 2905 Acked Read 2 Done Sent Done false
7144 2906 Acked Read 9 Done Sent Done false
7145 2907 Acked Flush 0 Done Sent Done false
7146 2908 Acked Write 3 Done Sent Done false
7147 2909 Acked Flush 0 Done Sent Done false
7148 2910 Acked Write 3 Done Sent Done false
7149 2911 Acked Write 7 Done Sent Done false
7150 2912 Acked Read 6 Done Sent Done false
7151 2913 Acked Flush 0 Done Sent Done false
7152 2914 Acked Read 7 Done Sent Done false
7153 2915 Acked Write 3 Done Sent Done false
7154 2916 Acked Write 5 Done Sent Done false
7155 2917 Acked Write 7 Done Sent Done false
7156 2918 Acked Read 10 Done Sent Done false
7157 2919 Acked Read 6 Done Sent Done false
7158 2920 Acked Read 8 Done Sent Done false
7159 2921 Acked Write 10 Done Sent Done false
7160 2922 Acked Write 3 Done Sent Done false
7161 2923 Acked Write 6 Done Sent Done false
7162 2924 Acked Read 7 Done Sent Done false
7163 2925 Acked Write 10 Done Sent Done false
7164 2926 Acked Read 3 Done Sent Done false
7165 2927 Acked Write 1 Done Sent Done false
7166 2928 Acked Read 10 Done Sent Done false
7167 2929 Acked Write 6 Done Sent Done false
7168 2930 Acked Read 7 Done Sent Done false
7169 2931 Acked Write 7 Done Sent Done false
7170 2932 Acked Read 6 Done Sent Done false
7171 2933 Acked Read 2 Done Sent Done false
7172 2934 Acked Write 9 Done Sent Done false
7173 2935 Acked Write 5 Done Sent Done false
7174 2936 Acked Read 4 Done Sent Done false
7175 2937 Acked Read 2 Done Sent Done false
7176 2938 Acked Read 2 Done Sent Done false
7177 2939 Acked Flush 0 Done Sent Done false
7178 2940 Acked Read 9 Done Sent Done false
7179 2941 Acked Flush 0 Done Sent Done false
7180 2942 Acked Flush 0 Done Sent Done false
7181 2943 Acked Read 2 Done Sent Done false
7182 2944 Acked Write 3 Done Sent Done false
7183 2945 Acked Read 3 Done Sent Done false
7184 2946 Acked Read 9 Done Sent Done false
7185 2947 Acked Flush 0 Done Sent Done false
7186 2948 Acked Read 3 Done Sent Done false
7187 2949 Acked Write 8 Done Sent Done false
7188 2950 Acked Read 2 Done Sent Done false
7189 2951 Acked Write 8 Done Sent Done false
7190 2952 Acked Write 8 Done Sent Done false
7191 2953 Acked Read 6 Done Sent Done false
7192 2954 Acked Flush 0 Done Sent Done false
7193 2955 Acked Read 1 Done Sent Done false
7194 2956 Acked Write 6 Done Sent Done false
7195 2957 Acked Read 4 Done Sent Done false
7196 2958 Acked Read 8 Done Sent Done false
7197 2959 Acked Write 1 Done Sent Done false
7198 2960 Acked Write 4 Done Sent Done false
7199 2961 Acked Read 9 Done Sent Done false
7200 2962 Acked Read 3 Done Sent Done false
7201 2963 Acked Read 10 Done Sent Done false
7202 2964 Acked Write 1 Done Sent Done false
7203 2965 Acked Flush 0 Done Sent Done false
7204 2966 Acked Read 9 Done Sent Done false
7205 2967 Acked Write 3 Done Sent Done false
7206 2968 Acked Read 9 Done Sent Done false
7207 2969 Acked Read 6 Done Sent Done false
7208 2970 Acked Write 5 Done Sent Done false
7209 2971 Acked Read 4 Done Sent Done false
7210 2972 Acked Write 6 Done Sent Done false
7211 2973 Acked Write 10 Done Sent Done false
7212 2974 Acked Read 7 Done Sent Done false
7213 2975 Acked Write 8 Done Sent Done false
7214 2976 Acked Read 4 Done Sent Done false
7215 2977 Acked Write 8 Done Sent Done false
7216 2978 Acked Flush 0 Done Sent Done false
7217 2979 Acked Read 4 Done Sent Done false
7218 2980 Acked Read 4 Done Sent Done false
7219 2981 Acked Write 6 Done Sent Done false
7220 2982 Acked Flush 0 Done Sent Done false
7221 2983 Acked Write 2 Done Sent Done false
7222 2984 Acked Read 6 Done Sent Done false
7223 2985 Acked Read 9 Done Sent Done false
7224 2986 Acked Read 8 Done Sent Done false
7225 2987 Acked Write 4 Done Sent Done false
7226 2988 Acked Read 1 Done Sent Done false
7227 2989 Acked Read 1 Done Sent Done false
7228 2990 Acked Read 9 Done Sent Done false
7229 2991 Acked Flush 0 Done Sent Done false
7230 2992 Acked Read 10 Done Sent Done false
7231 2993 Acked Flush 0 Done Sent Done false
7232 2994 Acked Read 6 Done Sent Done false
7233 2995 Acked Write 3 Done Sent Done false
7234 2996 Acked Write 8 Done Sent Done false
7235 2997 Acked Read 10 Done Sent Done false
7236 2998 Acked Write 6 Done Sent Done false
7237 2999 Acked Read 1 Done Sent Done false
7238 3000 Acked Read 8 Done Sent Done false
7239 3001 Acked Read 6 Done Sent Done false
7240 3002 Acked Read 3 Done Sent Done false
7241 3003 Acked Flush 0 Done Sent Done false
7242 3004 Acked Read 9 Done Sent Done false
7243 3005 Acked Read 3 Done Sent Done false
7244 3006 Acked Write 5 Done Sent Done false
7245 3007 Acked Read 8 Done Sent Done false
7246 3008 Acked Write 10 Done Sent Done false
7247 3009 Acked Write 7 Done Sent Done false
7248 3010 Acked Write 1 Done Sent Done false
7249 3011 Acked Write 7 Done Sent Done false
7250 3012 Acked Flush 0 Done Sent Done false
7251 3013 Acked Read 5 Done Sent Done false
7252 3014 Acked Write 10 Done Sent Done false
7253 3015 Acked Read 1 Done Sent Done false
7254 3016 Acked Flush 0 Done Sent Done false
7255 3017 Acked Write 3 Done Sent Done false
7256 3018 Acked Read 4 Done Sent Done false
7257 3019 Acked Write 1 Done Sent Done false
7258 3020 Acked Read 5 Done Sent Done false
7259 3021 Acked Read 6 Done Sent Done false
7260 3022 Acked Read 6 Done Sent Done false
7261 3023 Acked Read 8 Done Sent Done false
7262 3024 Acked Read 7 Done Sent Done false
7263 3025 Acked Read 10 Done Sent Done false
7264 3026 Acked Write 1 Done Sent Done false
7265 3027 Acked Read 1 Done Sent Done false
7266 3028 Acked Flush 0 Done Sent Done false
7267 3029 Acked Flush 0 Done Sent Done false
7268 3030 Acked Read 2 Done Sent Done false
7269 3031 Acked Read 9 Done Sent Done false
7270 3032 Acked Flush 0 Done Sent Done false
7271 3033 Acked Write 5 Done Sent Done false
7272 3034 Acked Write 3 Done Sent Done false
7273 3035 Acked Read 9 Done Sent Done false
7274 3036 Acked Read 9 Done Sent Done false
7275 3037 Acked Write 5 Done Sent Done false
7276 3038 Acked Read 1 Done Sent Done false
7277 3039 Acked Read 9 Done Sent Done false
7278 3040 Acked Write 1 Done Sent Done false
7279 3041 Acked Write 10 Done Sent Done false
7280 3042 Acked Write 8 Done Sent Done false
7281 3043 Acked Write 6 Done Sent Done false
7282 3044 Acked Write 4 Done Sent Done false
7283 3045 Acked Read 7 Done Sent Done false
7284 3046 Acked Read 7 Done Sent Done false
7285 3047 Acked Flush 0 Done Sent Done false
7286 3048 Acked Flush 0 Done Sent Done false
7287 3049 Acked Flush 0 Done Sent Done false
7288 3050 Acked Flush 0 Done Sent Done false
7289 3051 Acked Write 3 Done Sent Done false
7290 3052 Acked Read 6 Done Sent Done false
7291 3053 Acked Flush 0 Done Sent Done false
7292 3054 Acked Read 9 Done Sent Done false
7293 3055 Acked Flush 0 Done Sent Done false
7294 3056 Acked Flush 0 Done Sent Done false
7295 3057 Acked Write 7 Done Sent Done false
7296 3058 Acked Read 7 Done Sent Done false
7297 3059 Acked Read 10 Done Sent Done false
7298 3060 Acked Read 8 Done Sent Done false
7299 3061 Acked Read 6 Done Sent Done false
7300 3062 Acked Flush 0 Done Sent Done false
7301 3063 Acked Write 10 Done Sent Done false
7302 3064 Acked Write 1 Done Sent Done false
7303 3065 Acked Read 7 Done Sent Done false
7304 3066 Acked Write 10 Done Sent Done false
7305 3067 Acked Read 1 Done Sent Done false
7306 3068 Acked Write 2 Done Sent Done false
7307 3069 Acked Write 1 Done Sent Done false
7308 3070 Acked Write 1 Done Sent Done false
7309 3071 Acked Write 7 Done Sent Done false
7310 3072 Acked Read 6 Done Sent Done false
7311 3073 Acked Write 5 Done Sent Done false
7312 3074 Acked Read 2 Done Sent Done false
7313 3075 Acked Flush 0 Done Sent Done false
7314 3076 Acked Read 1 Done Sent Done false
7315 3077 Acked Write 6 Done Sent Done false
7316 3078 Acked Write 9 Done Sent Done false
7317 3079 Acked Write 6 Done Sent Done false
7318 3080 Acked Write 10 Done Sent Done false
7319 3081 Acked Write 8 Done Sent Done false
7320 3082 Acked Read 1 Done Sent Done false
7321 3083 Acked Write 7 Done Sent Done false
7322 3084 Acked Read 6 Done Sent Done false
7323 3085 Acked Write 2 Done Sent Done false
7324 3086 Acked Read 5 Done Sent Done false
7325 3087 Acked Read 7 Done Sent Done false
7326 3088 Acked Write 5 Done Sent Done false
7327 3089 Acked Flush 0 Done Sent Done false
7328 3090 Acked Read 9 Done Sent Done false
7329 3091 Acked Write 1 Done Sent Done false
7330 3092 Acked Read 3 Done Sent Done false
7331 3093 Acked Write 1 Done Sent Done false
7332 3094 Acked Read 2 Done Sent Done false
7333 3095 Acked Read 5 Done Sent Done false
7334 3096 Acked Write 10 Done Sent Done false
7335 3097 Acked Read 7 Done Sent Done false
7336 3098 Acked Read 9 Done Sent Done false
7337 3099 Acked Flush 0 Done Sent Done false
7338 3100 Acked Read 10 Done Sent Done false
7339 3101 Acked Write 6 Done Sent Done false
7340 3102 Acked Read 4 Done Sent Done false
7341 3103 Acked Write 9 Done Sent Done false
7342 3104 Acked Write 1 Done Sent Done false
7343 3105 Acked Read 6 Done Sent Done false
7344 3106 Acked Flush 0 Done Sent Done false
7345 3107 Acked Write 1 Done Sent Done false
7346 3108 Acked Read 1 Done Sent Done false
7347 3109 Acked Flush 0 Done Sent Done false
7348 3110 Acked Read 4 Done Sent Done false
7349 3111 Acked Write 3 Done Sent Done false
7350 STATES DS:0 DS:1 DS:2 TOTAL
7351 Sent 0 301 0 301
7352 Done 301 0 301 602
7353 Skipped 0 0 0 0
7354 Error 0 0 0 0
7355 Last Flush: Some(JobId(3109)) Some(JobId(2810)) Some(JobId(3109))
7356 Downstairs last five completed: 2810 2809 2808 2807 2806
7357 Upstairs last five completed: 3111 3110 3109 3108 3107
7358 CLIENT: Up:0 ds:301 act:2
73592025-02-25T20:48:15.215ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
7360 [8820] got start action
7361 [8820] I am going to restart
7362 Starting downstairs at port 8820
7363 Make output file at "/tmp/dsc/downstairs-8820.txt"
7364 [8820][1] reports Starting
7365 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
7366 [8820][1] reports Running
73672025-02-25T20:48:16.216ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
73682025-02-25T20:48:17.218ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
73692025-02-25T20:48:18.221ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
7370 ----------------------------------------------------------------
7371 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7372 States: Active Offline Active
7373 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7374 2811 Acked Flush 0 Done Sent Done false
7375 2812 Acked Read 9 Done Sent Done false
7376 2813 Acked Read 10 Done Sent Done false
7377 2814 Acked Read 2 Done Sent Done false
7378 2815 Acked Write 2 Done Sent Done false
7379 2816 Acked Write 2 Done Sent Done false
7380 2817 Acked Read 10 Done Sent Done false
7381 2818 Acked Write 3 Done Sent Done false
7382 2819 Acked Read 3 Done Sent Done false
7383 2820 Acked Write 6 Done Sent Done false
7384 2821 Acked Read 2 Done Sent Done false
7385 2822 Acked Read 5 Done Sent Done false
7386 2823 Acked Read 3 Done Sent Done false
7387 2824 Acked Flush 0 Done Sent Done false
7388 2825 Acked Write 4 Done Sent Done false
7389 2826 Acked Read 8 Done Sent Done false
7390 2827 Acked Flush 0 Done Sent Done false
7391 2828 Acked Read 2 Done Sent Done false
7392 2829 Acked Read 4 Done Sent Done false
7393 2830 Acked Flush 0 Done Sent Done false
7394 2831 Acked Flush 0 Done Sent Done false
7395 2832 Acked Read 2 Done Sent Done false
7396 2833 Acked Write 7 Done Sent Done false
7397 2834 Acked Read 6 Done Sent Done false
7398 2835 Acked Write 4 Done Sent Done false
7399 2836 Acked Flush 0 Done Sent Done false
7400 2837 Acked Flush 0 Done Sent Done false
7401 2838 Acked Write 10 Done Sent Done false
7402 2839 Acked Read 3 Done Sent Done false
7403 2840 Acked Flush 0 Done Sent Done false
7404 2841 Acked Read 10 Done Sent Done false
7405 2842 Acked Write 5 Done Sent Done false
7406 2843 Acked Flush 0 Done Sent Done false
7407 2844 Acked Flush 0 Done Sent Done false
7408 2845 Acked Flush 0 Done Sent Done false
7409 2846 Acked Read 9 Done Sent Done false
7410 2847 Acked Write 1 Done Sent Done false
7411 2848 Acked Write 7 Done Sent Done false
7412 2849 Acked Read 7 Done Sent Done false
7413 2850 Acked Read 5 Done Sent Done false
7414 2851 Acked Read 4 Done Sent Done false
7415 2852 Acked Read 2 Done Sent Done false
7416 2853 Acked Read 7 Done Sent Done false
7417 2854 Acked Write 3 Done Sent Done false
7418 2855 Acked Read 4 Done Sent Done false
7419 2856 Acked Write 10 Done Sent Done false
7420 2857 Acked Read 2 Done Sent Done false
7421 2858 Acked Read 6 Done Sent Done false
7422 2859 Acked Write 6 Done Sent Done false
7423 2860 Acked Read 2 Done Sent Done false
7424 2861 Acked Read 10 Done Sent Done false
7425 2862 Acked Write 5 Done Sent Done false
7426 2863 Acked Write 2 Done Sent Done false
7427 2864 Acked Read 6 Done Sent Done false
7428 2865 Acked Read 6 Done Sent Done false
7429 2866 Acked Read 3 Done Sent Done false
7430 2867 Acked Read 8 Done Sent Done false
7431 2868 Acked Read 9 Done Sent Done false
7432 2869 Acked Flush 0 Done Sent Done false
7433 2870 Acked Read 7 Done Sent Done false
7434 2871 Acked Read 9 Done Sent Done false
7435 2872 Acked Read 2 Done Sent Done false
7436 2873 Acked Write 5 Done Sent Done false
7437 2874 Acked Write 10 Done Sent Done false
7438 2875 Acked Flush 0 Done Sent Done false
7439 2876 Acked Write 2 Done Sent Done false
7440 2877 Acked Write 5 Done Sent Done false
7441 2878 Acked Read 4 Done Sent Done false
7442 2879 Acked Write 7 Done Sent Done false
7443 2880 Acked Write 8 Done Sent Done false
7444 2881 Acked Flush 0 Done Sent Done false
7445 2882 Acked Flush 0 Done Sent Done false
7446 2883 Acked Write 10 Done Sent Done false
7447 2884 Acked Read 9 Done Sent Done false
7448 2885 Acked Read 1 Done Sent Done false
7449 2886 Acked Flush 0 Done Sent Done false
7450 2887 Acked Read 6 Done Sent Done false
7451 2888 Acked Read 7 Done Sent Done false
7452 2889 Acked Read 10 Done Sent Done false
7453 2890 Acked Write 5 Done Sent Done false
7454 2891 Acked Write 7 Done Sent Done false
7455 2892 Acked Write 5 Done Sent Done false
7456 2893 Acked Flush 0 Done Sent Done false
7457 2894 Acked Read 6 Done Sent Done false
7458 2895 Acked Flush 0 Done Sent Done false
7459 2896 Acked Flush 0 Done Sent Done false
7460 2897 Acked Read 8 Done Sent Done false
7461 2898 Acked Read 2 Done Sent Done false
7462 2899 Acked Write 3 Done Sent Done false
7463 2900 Acked Read 9 Done Sent Done false
7464 2901 Acked Write 2 Done Sent Done false
7465 2902 Acked Write 8 Done Sent Done false
7466 2903 Acked Read 9 Done Sent Done false
7467 2904 Acked Write 2 Done Sent Done false
7468 2905 Acked Read 2 Done Sent Done false
7469 2906 Acked Read 9 Done Sent Done false
7470 2907 Acked Flush 0 Done Sent Done false
7471 2908 Acked Write 3 Done Sent Done false
7472 2909 Acked Flush 0 Done Sent Done false
7473 2910 Acked Write 3 Done Sent Done false
7474 2911 Acked Write 7 Done Sent Done false
7475 2912 Acked Read 6 Done Sent Done false
7476 2913 Acked Flush 0 Done Sent Done false
7477 2914 Acked Read 7 Done Sent Done false
7478 2915 Acked Write 3 Done Sent Done false
7479 2916 Acked Write 5 Done Sent Done false
7480 2917 Acked Write 7 Done Sent Done false
7481 2918 Acked Read 10 Done Sent Done false
7482 2919 Acked Read 6 Done Sent Done false
7483 2920 Acked Read 8 Done Sent Done false
7484 2921 Acked Write 10 Done Sent Done false
7485 2922 Acked Write 3 Done Sent Done false
7486 2923 Acked Write 6 Done Sent Done false
7487 2924 Acked Read 7 Done Sent Done false
7488 2925 Acked Write 10 Done Sent Done false
7489 2926 Acked Read 3 Done Sent Done false
7490 2927 Acked Write 1 Done Sent Done false
7491 2928 Acked Read 10 Done Sent Done false
7492 2929 Acked Write 6 Done Sent Done false
7493 2930 Acked Read 7 Done Sent Done false
7494 2931 Acked Write 7 Done Sent Done false
7495 2932 Acked Read 6 Done Sent Done false
7496 2933 Acked Read 2 Done Sent Done false
7497 2934 Acked Write 9 Done Sent Done false
7498 2935 Acked Write 5 Done Sent Done false
7499 2936 Acked Read 4 Done Sent Done false
7500 2937 Acked Read 2 Done Sent Done false
7501 2938 Acked Read 2 Done Sent Done false
7502 2939 Acked Flush 0 Done Sent Done false
7503 2940 Acked Read 9 Done Sent Done false
7504 2941 Acked Flush 0 Done Sent Done false
7505 2942 Acked Flush 0 Done Sent Done false
7506 2943 Acked Read 2 Done Sent Done false
7507 2944 Acked Write 3 Done Sent Done false
7508 2945 Acked Read 3 Done Sent Done false
7509 2946 Acked Read 9 Done Sent Done false
7510 2947 Acked Flush 0 Done Sent Done false
7511 2948 Acked Read 3 Done Sent Done false
7512 2949 Acked Write 8 Done Sent Done false
7513 2950 Acked Read 2 Done Sent Done false
7514 2951 Acked Write 8 Done Sent Done false
7515 2952 Acked Write 8 Done Sent Done false
7516 2953 Acked Read 6 Done Sent Done false
7517 2954 Acked Flush 0 Done Sent Done false
7518 2955 Acked Read 1 Done Sent Done false
7519 2956 Acked Write 6 Done Sent Done false
7520 2957 Acked Read 4 Done Sent Done false
7521 2958 Acked Read 8 Done Sent Done false
7522 2959 Acked Write 1 Done Sent Done false
7523 2960 Acked Write 4 Done Sent Done false
7524 2961 Acked Read 9 Done Sent Done false
7525 2962 Acked Read 3 Done Sent Done false
7526 2963 Acked Read 10 Done Sent Done false
7527 2964 Acked Write 1 Done Sent Done false
7528 2965 Acked Flush 0 Done Sent Done false
7529 2966 Acked Read 9 Done Sent Done false
7530 2967 Acked Write 3 Done Sent Done false
7531 2968 Acked Read 9 Done Sent Done false
7532 2969 Acked Read 6 Done Sent Done false
7533 2970 Acked Write 5 Done Sent Done false
7534 2971 Acked Read 4 Done Sent Done false
7535 2972 Acked Write 6 Done Sent Done false
7536 2973 Acked Write 10 Done Sent Done false
7537 2974 Acked Read 7 Done Sent Done false
7538 2975 Acked Write 8 Done Sent Done false
7539 2976 Acked Read 4 Done Sent Done false
7540 2977 Acked Write 8 Done Sent Done false
7541 2978 Acked Flush 0 Done Sent Done false
7542 2979 Acked Read 4 Done Sent Done false
7543 2980 Acked Read 4 Done Sent Done false
7544 2981 Acked Write 6 Done Sent Done false
7545 2982 Acked Flush 0 Done Sent Done false
7546 2983 Acked Write 2 Done Sent Done false
7547 2984 Acked Read 6 Done Sent Done false
7548 2985 Acked Read 9 Done Sent Done false
7549 2986 Acked Read 8 Done Sent Done false
7550 2987 Acked Write 4 Done Sent Done false
7551 2988 Acked Read 1 Done Sent Done false
7552 2989 Acked Read 1 Done Sent Done false
7553 2990 Acked Read 9 Done Sent Done false
7554 2991 Acked Flush 0 Done Sent Done false
7555 2992 Acked Read 10 Done Sent Done false
7556 2993 Acked Flush 0 Done Sent Done false
7557 2994 Acked Read 6 Done Sent Done false
7558 2995 Acked Write 3 Done Sent Done false
7559 2996 Acked Write 8 Done Sent Done false
7560 2997 Acked Read 10 Done Sent Done false
7561 2998 Acked Write 6 Done Sent Done false
7562 2999 Acked Read 1 Done Sent Done false
7563 3000 Acked Read 8 Done Sent Done false
7564 3001 Acked Read 6 Done Sent Done false
7565 3002 Acked Read 3 Done Sent Done false
7566 3003 Acked Flush 0 Done Sent Done false
7567 3004 Acked Read 9 Done Sent Done false
7568 3005 Acked Read 3 Done Sent Done false
7569 3006 Acked Write 5 Done Sent Done false
7570 3007 Acked Read 8 Done Sent Done false
7571 3008 Acked Write 10 Done Sent Done false
7572 3009 Acked Write 7 Done Sent Done false
7573 3010 Acked Write 1 Done Sent Done false
7574 3011 Acked Write 7 Done Sent Done false
7575 3012 Acked Flush 0 Done Sent Done false
7576 3013 Acked Read 5 Done Sent Done false
7577 3014 Acked Write 10 Done Sent Done false
7578 3015 Acked Read 1 Done Sent Done false
7579 3016 Acked Flush 0 Done Sent Done false
7580 3017 Acked Write 3 Done Sent Done false
7581 3018 Acked Read 4 Done Sent Done false
7582 3019 Acked Write 1 Done Sent Done false
7583 3020 Acked Read 5 Done Sent Done false
7584 3021 Acked Read 6 Done Sent Done false
7585 3022 Acked Read 6 Done Sent Done false
7586 3023 Acked Read 8 Done Sent Done false
7587 3024 Acked Read 7 Done Sent Done false
7588 3025 Acked Read 10 Done Sent Done false
7589 3026 Acked Write 1 Done Sent Done false
7590 3027 Acked Read 1 Done Sent Done false
7591 3028 Acked Flush 0 Done Sent Done false
7592 3029 Acked Flush 0 Done Sent Done false
7593 3030 Acked Read 2 Done Sent Done false
7594 3031 Acked Read 9 Done Sent Done false
7595 3032 Acked Flush 0 Done Sent Done false
7596 3033 Acked Write 5 Done Sent Done false
7597 3034 Acked Write 3 Done Sent Done false
7598 3035 Acked Read 9 Done Sent Done false
7599 3036 Acked Read 9 Done Sent Done false
7600 3037 Acked Write 5 Done Sent Done false
7601 3038 Acked Read 1 Done Sent Done false
7602 3039 Acked Read 9 Done Sent Done false
7603 3040 Acked Write 1 Done Sent Done false
7604 3041 Acked Write 10 Done Sent Done false
7605 3042 Acked Write 8 Done Sent Done false
7606 3043 Acked Write 6 Done Sent Done false
7607 3044 Acked Write 4 Done Sent Done false
7608 3045 Acked Read 7 Done Sent Done false
7609 3046 Acked Read 7 Done Sent Done false
7610 3047 Acked Flush 0 Done Sent Done false
7611 3048 Acked Flush 0 Done Sent Done false
7612 3049 Acked Flush 0 Done Sent Done false
7613 3050 Acked Flush 0 Done Sent Done false
7614 3051 Acked Write 3 Done Sent Done false
7615 3052 Acked Read 6 Done Sent Done false
7616 3053 Acked Flush 0 Done Sent Done false
7617 3054 Acked Read 9 Done Sent Done false
7618 3055 Acked Flush 0 Done Sent Done false
7619 3056 Acked Flush 0 Done Sent Done false
7620 3057 Acked Write 7 Done Sent Done false
7621 3058 Acked Read 7 Done Sent Done false
7622 3059 Acked Read 10 Done Sent Done false
7623 3060 Acked Read 8 Done Sent Done false
7624 3061 Acked Read 6 Done Sent Done false
7625 3062 Acked Flush 0 Done Sent Done false
7626 3063 Acked Write 10 Done Sent Done false
7627 3064 Acked Write 1 Done Sent Done false
7628 3065 Acked Read 7 Done Sent Done false
7629 3066 Acked Write 10 Done Sent Done false
7630 3067 Acked Read 1 Done Sent Done false
7631 3068 Acked Write 2 Done Sent Done false
7632 3069 Acked Write 1 Done Sent Done false
7633 3070 Acked Write 1 Done Sent Done false
7634 3071 Acked Write 7 Done Sent Done false
7635 3072 Acked Read 6 Done Sent Done false
7636 3073 Acked Write 5 Done Sent Done false
7637 3074 Acked Read 2 Done Sent Done false
7638 3075 Acked Flush 0 Done Sent Done false
7639 3076 Acked Read 1 Done Sent Done false
7640 3077 Acked Write 6 Done Sent Done false
7641 3078 Acked Write 9 Done Sent Done false
7642 3079 Acked Write 6 Done Sent Done false
7643 3080 Acked Write 10 Done Sent Done false
7644 3081 Acked Write 8 Done Sent Done false
7645 3082 Acked Read 1 Done Sent Done false
7646 3083 Acked Write 7 Done Sent Done false
7647 3084 Acked Read 6 Done Sent Done false
7648 3085 Acked Write 2 Done Sent Done false
7649 3086 Acked Read 5 Done Sent Done false
7650 3087 Acked Read 7 Done Sent Done false
7651 3088 Acked Write 5 Done Sent Done false
7652 3089 Acked Flush 0 Done Sent Done false
7653 3090 Acked Read 9 Done Sent Done false
7654 3091 Acked Write 1 Done Sent Done false
7655 3092 Acked Read 3 Done Sent Done false
7656 3093 Acked Write 1 Done Sent Done false
7657 3094 Acked Read 2 Done Sent Done false
7658 3095 Acked Read 5 Done Sent Done false
7659 3096 Acked Write 10 Done Sent Done false
7660 3097 Acked Read 7 Done Sent Done false
7661 3098 Acked Read 9 Done Sent Done false
7662 3099 Acked Flush 0 Done Sent Done false
7663 3100 Acked Read 10 Done Sent Done false
7664 3101 Acked Write 6 Done Sent Done false
7665 3102 Acked Read 4 Done Sent Done false
7666 3103 Acked Write 9 Done Sent Done false
7667 3104 Acked Write 1 Done Sent Done false
7668 3105 Acked Read 6 Done Sent Done false
7669 3106 Acked Flush 0 Done Sent Done false
7670 3107 Acked Write 1 Done Sent Done false
7671 3108 Acked Read 1 Done Sent Done false
7672 3109 Acked Flush 0 Done Sent Done false
7673 3110 Acked Read 4 Done Sent Done false
7674 3111 Acked Write 3 Done Sent Done false
7675 STATES DS:0 DS:1 DS:2 TOTAL
7676 Sent 0 301 0 301
7677 Done 301 0 301 602
7678 Skipped 0 0 0 0
7679 Error 0 0 0 0
7680 Last Flush: Some(JobId(3109)) Some(JobId(2810)) Some(JobId(3109))
7681 Downstairs last five completed: 2810 2809 2808 2807 2806
7682 Upstairs last five completed: 3111 3110 3109 3108 3107
7683 CLIENT: Up:0 ds:301 act:2
76842025-02-25T20:48:19.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76852025-02-25T20:48:20.138ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76862025-02-25T20:48:20.138ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64612) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76872025-02-25T20:48:20.139ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76882025-02-25T20:48:20.140ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76892025-02-25T20:48:20.140ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76902025-02-25T20:48:20.140ZINFOcrucible: send last flush ID to this DS: Some(JobId(2810)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76912025-02-25T20:48:20.140ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76922025-02-25T20:48:20.140ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2810)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
76932025-02-25T20:48:20.140ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2810)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
7694 ----------------------------------------------------------------
7695 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7696 Downstairs last five completed: 3112 3111 3110 3109 3108
7697 Upstairs last five completed: 3112 3111 3110 3109 3108
7698 CLIENT: Up:0 ds:0 act:3
7699 Replay: All jobs finished
7700 Main task has work to do, go find it
7701 got dsc Stop(2)
7702 stop 2
7703 Feb 25 20:48:22.386 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
7704 383, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7db639d8-6b33-46e1-80d0-5ba3855db310, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
77052025-02-25T20:48:22.387ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
77062025-02-25T20:48:22.387ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
77072025-02-25T20:48:22.387ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
7708 Feb 25 20:48:22.387 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0d6d9c65-6397-4107-b76e-aed0994cef72, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
77092025-02-25T20:48:22.387ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
7710 [8830] Exited with: ExitStatus(unix_wait_status(9))
7711 [8830][2] reports Exit
7712 Feb 25 20:48:26.389 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: f88376b9-94ea-4609-aff2-79dde788d56b, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
7713 001/300 Write block 693 len 16384 data: 4 3 3 2
7714 002/300 Write block 3958 len 20480 data: 3 2 3 3 3
7715 003/300 Write block 2519 len 32768 data: 3 3 3 4 4 3 3 3
7716 004/300 Write block 1577 len 28672 data: 3 3 3 4 3 4 5
7717 005/300 Write block 3882 len 4096 data: 5
7718 006/300 Read block 1853 len 28672
7719 007/300 Write block 3436 len 24576 data: 3 2 2 2 2 2
7720 008/300 Flush
7721 009/300 Flush
7722 010/300 Read block 3340 len 40960
7723 011/300 Read block 2542 len 20480
7724 012/300 Read block 4682 len 12288
7725 013/300 Write block 2533 len 12288 data: 3 4 2
7726 014/300 Write block 4149 len 16384 data: 3 3 3 3
7727 015/300 Write block 1305 len 8192 data: 3 3
7728 016/300 Read block 1848 len 12288
7729 017/300 Write block 3968 len 32768 data: 3 3 3 3 3 2 2 2
7730 018/300 Write block 3641 len 24576 data: 5 6 6 6 5 6
7731 019/300 Read block 4622 len 28672
7732 020/300 Read block 2438 len 32768
7733 021/300 Write block 186 len 36864 data: 3 3 3 3 3 3 2 2 3
7734 022/300 Flush
7735 023/300 Read block 2911 len 8192
7736 024/300 Read block 2100 len 40960
7737 025/300 Write block 1230 len 16384 data: 2 2 2 2
7738 026/300 Write block 4436 len 32768 data: 2 2 2 2 3 3 3 3
7739 027/300 Read block 42 len 8192
7740 028/300 Write block 3479 len 4096 data: 3
7741 029/300 Read block 3643 len 36864
7742 030/300 Read block 2736 len 28672
7743 031/300 Flush
7744 032/300 Write block 1104 len 4096 data: 2
7745 033/300 Write block 1263 len 20480 data: 3 3 3 3 3
7746 034/300 Flush
7747 035/300 Read block 2204 len 24576
7748 036/300 Write block 2528 len 12288 data: 2 2 3
7749 037/300 Write block 2512 len 28672 data: 3 3 4 3 3 3 3
7750 038/300 Flush
7751 039/300 Write block 1110 len 36864 data: 2 2 3 2 2 2 2 2 2
7752 040/300 Write block 1409 len 36864 data: 3 3 3 2 2 2 3 3 3
7753 041/300 Write block 3019 len 20480 data: 2 2 2 3 3
7754 042/300 Read block 4047 len 32768
7755 043/300 Read block 3790 len 24576
7756 044/300 Write block 2800 len 8192 data: 3 3
7757 045/300 Write block 2271 len 36864 data: 2 3 2 2 2 2 2 2 2
7758 046/300 Write block 2974 len 4096 data: 3
7759 047/300 Flush
7760 048/300 Write block 780 len 36864 data: 2 2 2 2 2 2 2 2 2
7761 049/300 Write block 4160 len 12288 data: 2 4 4
7762 050/300 Write block 3310 len 24576 data: 2 2 2 2 2 3
7763 051/300 Read block 2381 len 24576
7764 052/300 Read block 4623 len 40960
7765 053/300 Read block 963 len 40960
7766 054/300 Write block 3292 len 24576 data: 3 2 2 2 2 2
7767 055/300 Read block 2133 len 4096
7768 056/300 Read block 3243 len 4096
7769 057/300 Write block 3364 len 36864 data: 2 2 2 2 3 3 4 4 4
7770 058/300 Read block 4650 len 20480
7771 059/300 Read block 259 len 36864
7772 060/300 Flush
7773 061/300 Write block 4454 len 28672 data: 3 3 2 2 2 3 2
7774 062/300 Write block 4159 len 32768 data: 2 3 5 5 5 5 6 7
7775 063/300 Read block 4008 len 16384
7776 064/300 Read block 1856 len 8192
7777 065/300 Read block 336 len 28672
7778 066/300 Write block 1617 len 20480 data: 3 2 2 2 3
7779 067/300 Read block 1372 len 32768
7780 068/300 Write block 87 len 32768 data: 3 3 3 3 4 5 4 4
7781 069/300 Read block 1768 len 28672
7782 070/300 Flush
7783 071/300 Read block 3027 len 20480
7784 072/300 Flush
7785 073/300 Write block 2982 len 36864 data: 4 4 4 4 4 5 7 5 5
7786 074/300 Write block 2974 len 16384 data: 4 2 2 2
7787 075/300 Read block 2693 len 28672
7788 076/300 Read block 3806 len 8192
7789 077/300 Write block 20 len 8192 data: 2 2
7790 078/300 Read block 3174 len 12288
7791 079/300 Read block 755 len 12288
7792 080/300 Write block 2445 len 16384 data: 4 4 5 4
7793 081/300 Flush
7794 082/300 Read block 4074 len 20480
7795 083/300 Write block 2000 len 24576 data: 3 2 2 2 2 2
7796 084/300 Flush
7797 085/300 Read block 3910 len 36864
7798 086/300 Read block 4024 len 4096
7799 087/300 Write block 2337 len 28672 data: 4 4 4 4 6 6 6
7800 088/300 Write block 3088 len 12288 data: 3 3 2
7801 089/300 Flush
7802 090/300 Flush
7803 091/300 Write block 619 len 20480 data: 2 2 2 2 3
7804 092/300 Write block 1252 len 12288 data: 2 2 2
7805 093/300 Read block 3331 len 20480
7806 094/300 Write block 322 len 20480 data: 4 3 3 2 2
7807 095/300 Read block 1590 len 40960
7808 096/300 Read block 916 len 40960
7809 097/300 Read block 902 len 20480
7810 098/300 Read block 112 len 40960
7811 099/300 Read block 2023 len 4096
7812 100/300 Read block 909 len 36864
7813 101/300 Read block 1807 len 28672
7814 102/300 Read block 1139 len 8192
7815 103/300 Flush
7816 104/300 Write block 1689 len 4096 data: 2
7817 105/300 Write block 4676 len 36864 data: 2 2 2 2 2 2 2 3 3
7818 106/300 Read block 3396 len 12288
7819 107/300 Write block 383 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7820 108/300 Write block 4625 len 28672 data: 2 2 2 2 2 3 3
7821 109/300 Write block 4107 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7822 110/300 Read block 901 len 32768
7823 111/300 Flush
7824 112/300 Read block 286 len 4096
7825 113/300 Read block 460 len 20480
7826 114/300 Read block 2831 len 36864
7827 115/300 Read block 4186 len 4096
7828 116/300 Flush
7829 117/300 Read block 1110 len 24576
7830 118/300 Write block 1319 len 40960 data: 3 3 3 3 5 5 6 5 4 4
7831 119/300 Read block 882 len 4096
7832 120/300 Read block 3255 len 8192
7833 121/300 Write block 4526 len 28672 data: 4 4 4 4 3 3 3
7834 122/300 Write block 2413 len 12288 data: 3 3 3
7835 123/300 Read block 2708 len 24576
7836 124/300 Write block 4831 len 40960 data: 2 2 2 2 2 2 2 3 3 2
7837 125/300 Write block 4534 len 4096 data: 2
7838 126/300 Write block 1789 len 40960 data: 3 3 3 4 4 3 3 3 3 3
7839 127/300 Write block 4415 len 32768 data: 4 3 3 4 3 4 5 6
7840 128/300 Read block 4095 len 20480
7841 129/300 Write block 1546 len 40960 data: 3 2 2 2 2 2 2 3 3 4
7842 130/300 Read block 3604 len 4096
7843 131/300 Flush
7844 132/300 Read block 2382 len 24576
7845 133/300 Write block 605 len 28672 data: 3 2 2 2 2 2 2
7846 134/300 Read block 569 len 8192
7847 135/300 Write block 3747 len 20480 data: 3 2 2 3 3
7848 136/300 Write block 994 len 24576 data: 4 4 4 2 2 2
7849 137/300 Write block 1386 len 32768 data: 2 2 2 2 2 3 3 4
7850 138/300 Write block 3166 len 4096 data: 5
7851 139/300 Read block 613 len 32768
7852 140/300 Flush
7853 141/300 Read block 1631 len 4096
7854 142/300 Read block 551 len 16384
7855 143/300 Read block 2678 len 40960
7856 144/300 Write block 4478 len 20480 data: 2 2 3 3 2
7857 145/300 Read block 2813 len 36864
7858 146/300 Read block 4301 len 8192
7859 147/300 Write block 383 len 36864 data: 3 3 3 3 3 3 3 3 3
7860 148/300 Flush
7861 149/300 Read block 4381 len 12288
7862 150/300 Read block 1738 len 32768
7863 151/300 Write block 4700 len 4096 data: 3
7864 152/300 Write block 3389 len 32768 data: 2 2 3 3 3 3 3 2
7865 153/300 Flush
7866 154/300 Read block 4069 len 12288
7867 155/300 Write block 1455 len 4096 data: 4
7868 156/300 Read block 778 len 8192
7869 157/300 Write block 4609 len 20480 data: 2 2 2 2 2
7870 158/300 Write block 2647 len 24576 data: 4 4 4 4 4 3
7871 159/300 Read block 2579 len 36864
7872 160/300 Read block 3904 len 36864
7873 161/300 Read block 2774 len 4096
7874 162/300 Write block 196 len 16384 data: 3 3 3 3
7875 163/300 Read block 1502 len 20480
7876 164/300 Write block 3578 len 12288 data: 2 2 2
7877 165/300 Read block 484 len 20480
7878 166/300 Write block 4740 len 4096 data: 3
7879 167/300 Read block 1372 len 4096
7880 168/300 Read block 3408 len 8192
7881 169/300 Write block 4583 len 28672 data: 2 2 2 3 2 2 2
7882 170/300 Read block 4860 len 24576
7883 171/300 Read block 3228 len 40960
7884 172/300 Read block 1152 len 28672
7885 173/300 Read block 329 len 32768
7886 174/300 Read block 4489 len 8192
7887 175/300 Write block 3350 len 12288 data: 4 3 3
7888 176/300 Read block 4231 len 20480
7889 177/300 Write block 4514 len 4096 data: 4
7890 178/300 Read block 3739 len 4096
7891 179/300 Write block 808 len 8192 data: 2 2
7892 180/300 Write block 3701 len 16384 data: 2 2 2 3
7893 181/300 Read block 4104 len 12288
7894 182/300 Write block 3676 len 24576 data: 2 2 2 3 4 4
7895 183/300 Write block 1852 len 28672 data: 3 3 2 3 3 4 3
7896 184/300 Write block 414 len 12288 data: 3 3 3
7897 185/300 Read block 4660 len 24576
7898 186/300 Write block 4528 len 16384 data: 5 5 4 4
7899 187/300 Read block 2598 len 8192
7900 188/300 Read block 3016 len 12288
7901 189/300 Read block 1217 len 40960
7902 190/300 Read block 688 len 8192
7903 191/300 Write block 4355 len 8192 data: 2 2
7904 192/300 Read block 2219 len 24576
7905 193/300 Read block 3851 len 28672
7906 194/300 Read block 808 len 8192
7907 195/300 Flush
7908 196/300 Read block 4533 len 40960
7909 197/300 Read block 489 len 32768
7910 198/300 Read block 4275 len 4096
7911 199/300 Write block 366 len 24576 data: 2 2 2 2 2 2
7912 200/300 Write block 4817 len 40960 data: 2 2 2 2 3 4 4 4 4 3
7913 201/300 Read block 634 len 24576
7914 202/300 Read block 4871 len 20480
7915 203/300 Read block 2895 len 24576
7916 204/300 Write block 123 len 40960 data: 3 4 5 5 6 4 4 4 3 3
7917 205/300 Read block 1697 len 24576
7918 206/300 Write block 1012 len 40960 data: 3 3 3 3 3 3 3 4 3 3
7919 207/300 Read block 2423 len 24576
7920 208/300 Flush
7921 209/300 Read block 4598 len 24576
7922 210/300 Read block 3555 len 20480
7923 211/300 Read block 1203 len 4096
7924 212/300 Read block 2371 len 20480
7925 213/300 Write block 1032 len 24576 data: 3 3 3 3 4 4
7926 214/300 Read block 3419 len 8192
7927 215/300 Write block 2913 len 40960 data: 2 2 2 2 2 2 3 3 3 3
7928 216/300 Read block 2190 len 28672
7929 217/300 Flush
7930 218/300 Write block 777 len 36864 data: 2 2 2 3 3 3 3 3 3
7931 219/300 Read block 861 len 32768
7932 220/300 Flush
7933 221/300 Write block 1782 len 36864 data: 3 3 3 3 3 3 3 4 4
7934 222/300 Write block 1489 len 4096 data: 2
7935 223/300 Read block 2041 len 32768
7936 224/300 Write block 1582 len 24576 data: 5 6 5 4 3 3
7937 225/300 Write block 957 len 20480 data: 3 3 3 3 3
7938 226/300 Write block 2532 len 20480 data: 4 4 5 3 2
7939 227/300 Read block 2593 len 28672
7940 228/300 Write block 2619 len 20480 data: 2 2 2 3 4
7941 229/300 Read block 4084 len 28672
7942 230/300 Read block 2950 len 12288
7943 231/300 Write block 2761 len 8192 data: 4 3
7944 232/300 Read block 891 len 4096
7945 233/300 Read block 777 len 32768
7946 234/300 Write block 250 len 20480 data: 4 4 4 3 3
7947 235/300 Write block 4159 len 36864 data: 3 4 6 6 6 6 7 8 6
7948 236/300 Read block 4612 len 12288
7949 237/300 Write block 4123 len 36864 data: 3 3 3 3 2 2 2 2 2
7950 238/300 Write block 713 len 36864 data: 4 3 3 3 2 2 2 2 3
7951 239/300 Read block 1033 len 36864
7952 240/300 Read block 3534 len 20480
7953 241/300 Read block 2035 len 12288
7954 242/300 Flush
7955 243/300 Write block 4685 len 28672 data: 3 3 3 3 3 4 4
7956 244/300 Flush
7957 245/300 Write block 383 len 20480 data: 4 4 4 4 4
7958 246/300 Write block 2124 len 24576 data: 3 3 3 3 3 3
7959 247/300 Flush
7960 248/300 Write block 3717 len 32768 data: 2 2 2 3 3 3 3 3
7961 249/300 Write block 3439 len 16384 data: 3 3 3 4
7962 250/300 Read block 2475 len 4096
7963 251/300 Write block 1211 len 28672 data: 3 3 2 2 2 2 4
7964 252/300 Write block 425 len 28672 data: 4 4 3 2 2 2 2
7965 253/300 Write block 4652 len 36864 data: 2 2 2 2 2 3 3 3 3
7966 254/300 Write block 2749 len 36864 data: 3 3 3 2 2 3 3 4 4
7967 255/300 Flush
7968 256/300 Read block 4288 len 16384
7969 257/300 Write block 4967 len 36864 data: 4 5 4 4 4 3 3 3 3
7970 258/300 Write block 2353 len 28672 data: 4 4 4 4 4 3 3
7971 259/300 Write block 2256 len 24576 data: 4 3 4 4 3 3
7972 260/300 Read block 3139 len 20480
7973 261/300 Read block 1427 len 24576
7974 262/300 Read block 2115 len 12288
7975 263/300 Read block 83 len 24576
7976 264/300 Read block 4866 len 16384
7977 265/300 Read block 2005 len 12288
7978 266/300 Write block 3144 len 8192 data: 3 3
7979 267/300 Write block 497 len 28672 data: 3 3 3 3 3 3 3
7980 268/300 Read block 4511 len 24576
7981 269/300 Write block 1174 len 24576 data: 2 2 2 2 2 2
7982 270/300 Write block 2626 len 24576 data: 5 4 4 3 3 2
7983 271/300 Write block 4021 len 16384 data: 2 3 4 3
7984 272/300 Write block 591 len 32768 data: 2 2 2 2 2 2 2 2
7985 273/300 Read block 4141 len 16384
7986 274/300 Write block 2483 len 8192 data: 2 2
7987 275/300 Write block 4180 len 24576 data: 2 2 2 2 2 3
7988 276/300 Read block 4608 len 28672
7989 277/300 Write block 3933 len 8192 data: 3 3
7990 278/300 Read block 43 len 40960
7991 279/300 Read block 2266 len 8192
7992 280/300 Write block 514 len 16384 data: 3 3 3 3
7993 281/300 Read block 2519 len 8192
7994 282/300 Write block 3619 len 4096 data: 4
7995 283/300 Read block 768 len 36864
7996 284/300 Write block 4454 len 36864 data: 4 4 3 3 3 4 3 3 3
7997 285/300 Read block 4456 len 16384
7998 286/300 Read block 4187 len 36864
7999 287/300 Write block 2315 len 16384 data: 2 2 2 2
8000 288/300 Read block 2642 len 32768
8001 289/300 Write block 3216 len 24576 data: 2 2 2 2 2 2
8002 290/300 Read block 2544 len 32768
8003 291/300 Read block 2155 len 12288
8004 292/300 Write block 4887 len 36864 data: 4 3 3 3 3 3 3 3 3
8005 293/300 Write block 979 len 36864 data: 2 2 2 2 3 4 4 4 4
8006 294/300 Read block 4323 len 28672
8007 295/300 Write block 25 len 24576 data: 3 3 3 2 2 2
8008 296/300 Read block 1486 len 24576
8009 297/300 Read block 2168 len 8192
8010 298/300 Write block 3299 len 32768 data: 5 5 5 5 6 5 5 3
8011 299/300 Read block 3115 len 8192
8012 300/300 Read block 1037 len 28672
8013 301/300 Write block 1610 len 36864 data: 2 2 3 3 3 3 3 4 3
8014 Main task has work to do, go find it
8015 got dsc Start(2)
8016 start 2
8017 Feb 25 20:48:26.630 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5cfeb8cd-d795-4a20-a023-ae3afb912dc2, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
8018 Replay: started 2, returned:Ok(())
8019 ----------------------------------------------------------------
8020 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8021 States: Active Active Offline
8022 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8023 3113 Acked Write 4 Done Done Sent false
8024 3114 Acked Write 5 Done Done Sent false
8025 3115 Acked Write 8 Done Done Sent false
8026 3116 Acked Write 7 Done Done Sent false
8027 3117 Acked Write 1 Done Done Sent false
8028 3118 Acked Read 7 Done Done Sent false
8029 3119 Acked Write 6 Done Done Sent false
8030 3120 Acked Flush 0 Done Done Sent false
8031 3121 Acked Flush 0 Done Done Sent false
8032 3122 Acked Read 10 Done Done Sent false
8033 3123 Acked Read 5 Done Done Sent false
8034 3124 Acked Read 3 Done Done Sent false
8035 3125 Acked Write 3 Done Done Sent false
8036 3126 Acked Write 4 Done Done Sent false
8037 3127 Acked Write 2 Done Done Sent false
8038 3128 Acked Read 3 Done Done Sent false
8039 3129 Acked Write 8 Done Done Sent false
8040 3130 Acked Write 6 Done Done Sent false
8041 3131 Acked Read 7 Done Done Sent false
8042 3132 Acked Read 8 Done Done Sent false
8043 3133 Acked Write 9 Done Done Sent false
8044 3134 Acked Flush 0 Done Done Sent false
8045 3135 Acked Read 2 Done Done Sent false
8046 3136 Acked Read 10 Done Done Sent false
8047 3137 Acked Write 4 Done Done Sent false
8048 3138 Acked Write 8 Done Done Sent false
8049 3139 Acked Read 2 Done Done Sent false
8050 3140 Acked Write 1 Done Done Sent false
8051 3141 Acked Read 9 Done Done Sent false
8052 3142 Acked Read 7 Done Done Sent false
8053 3143 Acked Flush 0 Done Done Sent false
8054 3144 Acked Write 1 Done Done Sent false
8055 3145 Acked Write 5 Done Done Sent false
8056 3146 Acked Flush 0 Done Done Sent false
8057 3147 Acked Read 6 Done Done Sent false
8058 3148 Acked Write 3 Done Done Sent false
8059 3149 Acked Write 7 Done Done Sent false
8060 3150 Acked Flush 0 Done Done Sent false
8061 3151 Acked Write 9 Done Done Sent false
8062 3152 Acked Write 9 Done Done Sent false
8063 3153 Acked Write 5 Done Done Sent false
8064 3154 Acked Read 8 Done Done Sent false
8065 3155 Acked Read 6 Done Done Sent false
8066 3156 Acked Write 2 Done Done Sent false
8067 3157 Acked Write 9 Done Done Sent false
8068 3158 Acked Write 1 Done Done Sent false
8069 3159 Acked Flush 0 Done Done Sent false
8070 3160 Acked Write 9 Done Done Sent false
8071 3161 Acked Write 3 Done Done Sent false
8072 3162 Acked Write 6 Done Done Sent false
8073 3163 Acked Read 6 Done Done Sent false
8074 3164 Acked Read 10 Done Done Sent false
8075 3165 Acked Read 10 Done Done Sent false
8076 3166 Acked Write 6 Done Done Sent false
8077 3167 Acked Read 1 Done Done Sent false
8078 3168 Acked Read 1 Done Done Sent false
8079 3169 Acked Write 9 Done Done Sent false
8080 3170 Acked Read 5 Done Done Sent false
8081 3171 Acked Read 9 Done Done Sent false
8082 3172 Acked Flush 0 Done Done Sent false
8083 3173 Acked Write 7 Done Done Sent false
8084 3174 Acked Write 8 Done Done Sent false
8085 3175 Acked Read 4 Done Done Sent false
8086 3176 Acked Read 2 Done Done Sent false
8087 3177 Acked Read 7 Done Done Sent false
8088 3178 Acked Write 5 Done Done Sent false
8089 3179 Acked Read 8 Done Done Sent false
8090 3180 Acked Write 8 Done Done Sent false
8091 3181 Acked Read 7 Done Done Sent false
8092 3182 Acked Flush 0 Done Done Sent false
8093 3183 Acked Read 5 Done Done Sent false
8094 3184 Acked Flush 0 Done Done Sent false
8095 3185 Acked Write 9 Done Done Sent false
8096 3186 Acked Write 4 Done Done Sent false
8097 3187 Acked Read 7 Done Done Sent false
8098 3188 Acked Read 2 Done Done Sent false
8099 3189 Acked Write 2 Done Done Sent false
8100 3190 Acked Read 3 Done Done Sent false
8101 3191 Acked Read 3 Done Done Sent false
8102 3192 Acked Write 4 Done Done Sent false
8103 3193 Acked Flush 0 Done Done Sent false
8104 3194 Acked Read 5 Done Done Sent false
8105 3195 Acked Write 6 Done Done Sent false
8106 3196 Acked Flush 0 Done Done Sent false
8107 3197 Acked Read 9 Done Done Sent false
8108 3198 Acked Read 1 Done Done Sent false
8109 3199 Acked Write 7 Done Done Sent false
8110 3200 Acked Write 3 Done Done Sent false
8111 3201 Acked Flush 0 Done Done Sent false
8112 3202 Acked Flush 0 Done Done Sent false
8113 3203 Acked Write 5 Done Done Sent false
8114 3204 Acked Write 3 Done Done Sent false
8115 3205 Acked Read 5 Done Done Sent false
8116 3206 Acked Write 5 Done Done Sent false
8117 3207 Acked Read 10 Done Done Sent false
8118 3208 Acked Read 10 Done Done Sent false
8119 3209 Acked Read 5 Done Done Sent false
8120 3210 Acked Read 10 Done Done Sent false
8121 3211 Acked Read 1 Done Done Sent false
8122 3212 Acked Read 9 Done Done Sent false
8123 3213 Acked Read 7 Done Done Sent false
8124 3214 Acked Read 2 Done Done Sent false
8125 3215 Acked Flush 0 Done Done Sent false
8126 3216 Acked Write 1 Done Done Sent false
8127 3217 Acked Write 9 Done Done Sent false
8128 3218 Acked Read 3 Done Done Sent false
8129 3219 Acked Write 10 Done Done Sent false
8130 3220 Acked Write 7 Done Done Sent false
8131 3221 Acked Write 10 Done Done Sent false
8132 3222 Acked Read 8 Done Done Sent false
8133 3223 Acked Flush 0 Done Done Sent false
8134 3224 Acked Read 1 Done Done Sent false
8135 3225 Acked Read 5 Done Done Sent false
8136 3226 Acked Read 9 Done Done Sent false
8137 3227 Acked Read 1 Done Done Sent false
8138 3228 Acked Flush 0 Done Done Sent false
8139 3229 Acked Read 6 Done Done Sent false
8140 3230 Acked Write 10 Done Done Sent false
8141 3231 Acked Read 1 Done Done Sent false
8142 3232 Acked Read 2 Done Done Sent false
8143 3233 Acked Write 7 Done Done Sent false
8144 3234 Acked Write 3 Done Done Sent false
8145 3235 Acked Read 6 Done Done Sent false
8146 3236 Acked Write 10 Done Done Sent false
8147 3237 Acked Write 1 Done Done Sent false
8148 3238 Acked Write 10 Done Done Sent false
8149 3239 Acked Write 8 Done Done Sent false
8150 3240 Acked Read 5 Done Done Sent false
8151 3241 Acked Write 10 Done Done Sent false
8152 3242 Acked Read 1 Done Done Sent false
8153 3243 Acked Flush 0 Done Done Sent false
8154 3244 Acked Read 6 Done Done Sent false
8155 3245 Acked Write 7 Done Done Sent false
8156 3246 Acked Read 2 Done Done Sent false
8157 3247 Acked Write 5 Done Done Sent false
8158 3248 Acked Write 6 Done Done Sent false
8159 3249 Acked Write 8 Done Done Sent false
8160 3250 Acked Write 1 Done Done Sent false
8161 3251 Acked Read 8 Done Done Sent false
8162 3252 Acked Flush 0 Done Done Sent false
8163 3253 Acked Read 1 Done Done Sent false
8164 3254 Acked Read 4 Done Done Sent false
8165 3255 Acked Read 10 Done Done Sent false
8166 3256 Acked Write 5 Done Done Sent false
8167 3257 Acked Read 9 Done Done Sent false
8168 3258 Acked Read 2 Done Done Sent false
8169 3259 Acked Write 9 Done Done Sent false
8170 3260 Acked Flush 0 Done Done Sent false
8171 3261 Acked Read 3 Done Done Sent false
8172 3262 Acked Read 8 Done Done Sent false
8173 3263 Acked Write 1 Done Done Sent false
8174 3264 Acked Write 8 Done Done Sent false
8175 3265 Acked Flush 0 Done Done Sent false
8176 3266 Acked Read 3 Done Done Sent false
8177 3267 Acked Write 1 Done Done Sent false
8178 3268 Acked Read 2 Done Done Sent false
8179 3269 Acked Write 5 Done Done Sent false
8180 3270 Acked Write 6 Done Done Sent false
8181 3271 Acked Read 9 Done Done Sent false
8182 3272 Acked Read 9 Done Done Sent false
8183 3273 Acked Read 1 Done Done Sent false
8184 3274 Acked Write 4 Done Done Sent false
8185 3275 Acked Read 5 Done Done Sent false
8186 3276 Acked Write 3 Done Done Sent false
8187 3277 Acked Read 5 Done Done Sent false
8188 3278 Acked Write 1 Done Done Sent false
8189 3279 Acked Read 1 Done Done Sent false
8190 3280 Acked Read 2 Done Done Sent false
8191 3281 Acked Write 7 Done Done Sent false
8192 3282 Acked Read 6 Done Done Sent false
8193 3283 Acked Read 10 Done Done Sent false
8194 3284 Acked Read 7 Done Done Sent false
8195 3285 Acked Read 8 Done Done Sent false
8196 3286 Acked Read 2 Done Done Sent false
8197 3287 Acked Write 3 Done Done Sent false
8198 3288 Acked Read 5 Done Done Sent false
8199 3289 Acked Write 1 Done Done Sent false
8200 3290 Acked Read 1 Done Done Sent false
8201 3291 Acked Write 2 Done Done Sent false
8202 3292 Acked Write 4 Done Done Sent false
8203 3293 Acked Read 3 Done Done Sent false
8204 3294 Acked Write 6 Done Done Sent false
8205 3295 Acked Write 7 Done Done Sent false
8206 3296 Acked Write 3 Done Done Sent false
8207 3297 Acked Read 6 Done Done Sent false
8208 3298 Acked Write 4 Done Done Sent false
8209 3299 Acked Read 2 Done Done Sent false
8210 3300 Acked Read 3 Done Done Sent false
8211 3301 Acked Read 10 Done Done Sent false
8212 3302 Acked Read 2 Done Done Sent false
8213 3303 Acked Write 2 Done Done Sent false
8214 3304 Acked Read 6 Done Done Sent false
8215 3305 Acked Read 7 Done Done Sent false
8216 3306 Acked Read 2 Done Done Sent false
8217 3307 Acked Flush 0 Done Done Sent false
8218 3308 Acked Read 10 Done Done Sent false
8219 3309 Acked Read 8 Done Done Sent false
8220 3310 Acked Read 1 Done Done Sent false
8221 3311 Acked Write 6 Done Done Sent false
8222 3312 Acked Write 10 Done Done Sent false
8223 3313 Acked Read 6 Done Done Sent false
8224 3314 Acked Read 5 Done Done Sent false
8225 3315 Acked Read 6 Done Done Sent false
8226 3316 Acked Write 10 Done Done Sent false
8227 3317 Acked Read 6 Done Done Sent false
8228 3318 Acked Write 10 Done Done Sent false
8229 3319 Acked Read 6 Done Done Sent false
8230 3320 Acked Flush 0 Done Done Sent false
8231 3321 Acked Read 6 Done Done Sent false
8232 3322 Acked Read 5 Done Done Sent false
8233 3323 Acked Read 1 Done Done Sent false
8234 3324 Acked Read 5 Done Done Sent false
8235 3325 Acked Write 6 Done Done Sent false
8236 3326 Acked Read 2 Done Done Sent false
8237 3327 Acked Write 10 Done Done Sent false
8238 3328 Acked Read 7 Done Done Sent false
8239 3329 Acked Flush 0 Done Done Sent false
8240 3330 Acked Write 9 Done Done Sent false
8241 3331 Acked Read 8 Done Done Sent false
8242 3332 Acked Flush 0 Done Done Sent false
8243 3333 Acked Write 9 Done Done Sent false
8244 3334 Acked Write 1 Done Done Sent false
8245 3335 Acked Read 8 Done Done Sent false
8246 3336 Acked Write 6 Done Done Sent false
8247 3337 Acked Write 5 Done Done Sent false
8248 3338 Acked Write 5 Done Done Sent false
8249 3339 Acked Read 7 Done Done Sent false
8250 3340 Acked Write 5 Done Done Sent false
8251 3341 Acked Read 7 Done Done Sent false
8252 3342 Acked Read 3 Done Done Sent false
8253 3343 Acked Write 2 Done Done Sent false
8254 3344 Acked Read 1 Done Done Sent false
8255 3345 Acked Read 8 Done Done Sent false
8256 3346 Acked Write 5 Done Done Sent false
8257 3347 Acked Write 9 Done Done Sent false
8258 3348 Acked Read 3 Done Done Sent false
8259 3349 Acked Write 9 Done Done Sent false
8260 3350 Acked Write 9 Done Done Sent false
8261 3351 Acked Read 9 Done Done Sent false
8262 3352 Acked Read 5 Done Done Sent false
8263 3353 Acked Read 3 Done Done Sent false
8264 3354 Acked Flush 0 Done Done Sent false
8265 3355 Acked Write 7 Done Done Sent false
8266 3356 Acked Flush 0 Done Done Sent false
8267 3357 Acked Write 5 Done Done Sent false
8268 3358 Acked Write 6 Done Done Sent false
8269 3359 Acked Flush 0 Done Done Sent false
8270 3360 Acked Write 8 Done Done Sent false
8271 3361 Acked Write 4 Done Done Sent false
8272 3362 Acked Read 1 Done Done Sent false
8273 3363 Acked Write 7 Done Done Sent false
8274 3364 Acked Write 7 Done Done Sent false
8275 3365 Acked Write 9 Done Done Sent false
8276 3366 Acked Write 9 Done Done Sent false
8277 3367 Acked Flush 0 Done Done Sent false
8278 3368 Acked Read 4 Done Done Sent false
8279 3369 Acked Write 9 Done Done Sent false
8280 3370 Acked Write 7 Done Done Sent false
8281 3371 Acked Write 6 Done Done Sent false
8282 3372 Acked Read 5 Done Done Sent false
8283 3373 Acked Read 6 Done Done Sent false
8284 3374 Acked Read 3 Done Done Sent false
8285 3375 Acked Read 6 Done Done Sent false
8286 3376 Acked Read 4 Done Done Sent false
8287 3377 Acked Read 3 Done Done Sent false
8288 3378 Acked Write 2 Done Done Sent false
8289 3379 Acked Write 7 Done Done Sent false
8290 3380 Acked Read 6 Done Done Sent false
8291 3381 Acked Write 6 Done Done Sent false
8292 3382 Acked Write 6 Done Done Sent false
8293 3383 Acked Write 4 Done Done Sent false
8294 3384 Acked Write 8 Done Done Sent false
8295 3385 Acked Read 4 Done Done Sent false
8296 3386 Acked Write 2 Done Done Sent false
8297 3387 Acked Write 6 Done Done Sent false
8298 3388 Acked Read 7 Done Done Sent false
8299 3389 Acked Write 2 Done Done Sent false
8300 3390 Acked Read 10 Done Done Sent false
8301 3391 Acked Read 2 Done Done Sent false
8302 3392 Acked Write 4 Done Done Sent false
8303 3393 Acked Read 2 Done Done Sent false
8304 3394 Acked Write 1 Done Done Sent false
8305 3395 Acked Read 9 Done Done Sent false
8306 3396 Acked Write 9 Done Done Sent false
8307 3397 Acked Read 4 Done Done Sent false
8308 3398 Acked Read 9 Done Done Sent false
8309 3399 Acked Write 4 Done Done Sent false
8310 3400 Acked Read 8 Done Done Sent false
8311 3401 Acked Write 6 Done Done Sent false
8312 3402 Acked Read 8 Done Done Sent false
8313 3403 Acked Read 3 Done Done Sent false
8314 3404 Acked Write 9 Done Done Sent false
8315 3405 Acked Write 9 Done Done Sent false
8316 3406 Acked Read 7 Done Done Sent false
8317 3407 Acked Write 6 Done Done Sent false
8318 3408 Acked Read 6 Done Done Sent false
8319 3409 Acked Read 2 Done Done Sent false
8320 3410 Acked Write 8 Done Done Sent false
8321 3411 Acked Read 2 Done Done Sent false
8322 3412 Acked Read 7 Done Done Sent false
8323 3413 Acked Write 9 Done Done Sent false
8324 STATES DS:0 DS:1 DS:2 TOTAL
8325 Sent 0 0 301 301
8326 Done 301 301 0 602
8327 Skipped 0 0 0 0
8328 Error 0 0 0 0
8329 Last Flush: Some(JobId(3367)) Some(JobId(3367)) Some(JobId(3112))
8330 Downstairs last five completed: 3112 3111 3110 3109 3108
8331 Upstairs last five completed: 3413 3412 3411 3410 3409
8332 CLIENT: Up:0 ds:301 act:2
83332025-02-25T20:48:27.233ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
83342025-02-25T20:48:28.236ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8335 [8830] got start action
8336 [8830] I am going to restart
8337 Starting downstairs at port 8830
8338 Make output file at "/tmp/dsc/downstairs-8830.txt"
8339 [8830][2] reports Starting
8340 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214)
8341 [8830][2] reports Running
83422025-02-25T20:48:29.237ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8343 ----------------------------------------------------------------
8344 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8345 States: Active Active Offline
8346 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8347 3113 Acked Write 4 Done Done Sent false
8348 3114 Acked Write 5 Done Done Sent false
8349 3115 Acked Write 8 Done Done Sent false
8350 3116 Acked Write 7 Done Done Sent false
8351 3117 Acked Write 1 Done Done Sent false
8352 3118 Acked Read 7 Done Done Sent false
8353 3119 Acked Write 6 Done Done Sent false
8354 3120 Acked Flush 0 Done Done Sent false
8355 3121 Acked Flush 0 Done Done Sent false
8356 3122 Acked Read 10 Done Done Sent false
8357 3123 Acked Read 5 Done Done Sent false
8358 3124 Acked Read 3 Done Done Sent false
8359 3125 Acked Write 3 Done Done Sent false
8360 3126 Acked Write 4 Done Done Sent false
8361 3127 Acked Write 2 Done Done Sent false
8362 3128 Acked Read 3 Done Done Sent false
8363 3129 Acked Write 8 Done Done Sent false
8364 3130 Acked Write 6 Done Done Sent false
8365 3131 Acked Read 7 Done Done Sent false
8366 3132 Acked Read 8 Done Done Sent false
8367 3133 Acked Write 9 Done Done Sent false
8368 3134 Acked Flush 0 Done Done Sent false
8369 3135 Acked Read 2 Done Done Sent false
8370 3136 Acked Read 10 Done Done Sent false
8371 3137 Acked Write 4 Done Done Sent false
8372 3138 Acked Write 8 Done Done Sent false
8373 3139 Acked Read 2 Done Done Sent false
8374 3140 Acked Write 1 Done Done Sent false
8375 3141 Acked Read 9 Done Done Sent false
8376 3142 Acked Read 7 Done Done Sent false
8377 3143 Acked Flush 0 Done Done Sent false
8378 3144 Acked Write 1 Done Done Sent false
8379 3145 Acked Write 5 Done Done Sent false
8380 3146 Acked Flush 0 Done Done Sent false
8381 3147 Acked Read 6 Done Done Sent false
8382 3148 Acked Write 3 Done Done Sent false
8383 3149 Acked Write 7 Done Done Sent false
8384 3150 Acked Flush 0 Done Done Sent false
8385 3151 Acked Write 9 Done Done Sent false
8386 3152 Acked Write 9 Done Done Sent false
8387 3153 Acked Write 5 Done Done Sent false
8388 3154 Acked Read 8 Done Done Sent false
8389 3155 Acked Read 6 Done Done Sent false
8390 3156 Acked Write 2 Done Done Sent false
8391 3157 Acked Write 9 Done Done Sent false
8392 3158 Acked Write 1 Done Done Sent false
8393 3159 Acked Flush 0 Done Done Sent false
8394 3160 Acked Write 9 Done Done Sent false
8395 3161 Acked Write 3 Done Done Sent false
8396 3162 Acked Write 6 Done Done Sent false
8397 3163 Acked Read 6 Done Done Sent false
8398 3164 Acked Read 10 Done Done Sent false
8399 3165 Acked Read 10 Done Done Sent false
8400 3166 Acked Write 6 Done Done Sent false
8401 3167 Acked Read 1 Done Done Sent false
8402 3168 Acked Read 1 Done Done Sent false
8403 3169 Acked Write 9 Done Done Sent false
8404 3170 Acked Read 5 Done Done Sent false
8405 3171 Acked Read 9 Done Done Sent false
8406 3172 Acked Flush 0 Done Done Sent false
8407 3173 Acked Write 7 Done Done Sent false
8408 3174 Acked Write 8 Done Done Sent false
8409 3175 Acked Read 4 Done Done Sent false
8410 3176 Acked Read 2 Done Done Sent false
8411 3177 Acked Read 7 Done Done Sent false
8412 3178 Acked Write 5 Done Done Sent false
8413 3179 Acked Read 8 Done Done Sent false
8414 3180 Acked Write 8 Done Done Sent false
8415 3181 Acked Read 7 Done Done Sent false
8416 3182 Acked Flush 0 Done Done Sent false
8417 3183 Acked Read 5 Done Done Sent false
8418 3184 Acked Flush 0 Done Done Sent false
8419 3185 Acked Write 9 Done Done Sent false
8420 3186 Acked Write 4 Done Done Sent false
8421 3187 Acked Read 7 Done Done Sent false
8422 3188 Acked Read 2 Done Done Sent false
8423 3189 Acked Write 2 Done Done Sent false
8424 3190 Acked Read 3 Done Done Sent false
8425 3191 Acked Read 3 Done Done Sent false
8426 3192 Acked Write 4 Done Done Sent false
8427 3193 Acked Flush 0 Done Done Sent false
8428 3194 Acked Read 5 Done Done Sent false
8429 3195 Acked Write 6 Done Done Sent false
8430 3196 Acked Flush 0 Done Done Sent false
8431 3197 Acked Read 9 Done Done Sent false
8432 3198 Acked Read 1 Done Done Sent false
8433 3199 Acked Write 7 Done Done Sent false
8434 3200 Acked Write 3 Done Done Sent false
8435 3201 Acked Flush 0 Done Done Sent false
8436 3202 Acked Flush 0 Done Done Sent false
8437 3203 Acked Write 5 Done Done Sent false
8438 3204 Acked Write 3 Done Done Sent false
8439 3205 Acked Read 5 Done Done Sent false
8440 3206 Acked Write 5 Done Done Sent false
8441 3207 Acked Read 10 Done Done Sent false
8442 3208 Acked Read 10 Done Done Sent false
8443 3209 Acked Read 5 Done Done Sent false
8444 3210 Acked Read 10 Done Done Sent false
8445 3211 Acked Read 1 Done Done Sent false
8446 3212 Acked Read 9 Done Done Sent false
8447 3213 Acked Read 7 Done Done Sent false
8448 3214 Acked Read 2 Done Done Sent false
8449 3215 Acked Flush 0 Done Done Sent false
8450 3216 Acked Write 1 Done Done Sent false
8451 3217 Acked Write 9 Done Done Sent false
8452 3218 Acked Read 3 Done Done Sent false
8453 3219 Acked Write 10 Done Done Sent false
8454 3220 Acked Write 7 Done Done Sent false
8455 3221 Acked Write 10 Done Done Sent false
8456 3222 Acked Read 8 Done Done Sent false
8457 3223 Acked Flush 0 Done Done Sent false
8458 3224 Acked Read 1 Done Done Sent false
8459 3225 Acked Read 5 Done Done Sent false
8460 3226 Acked Read 9 Done Done Sent false
8461 3227 Acked Read 1 Done Done Sent false
8462 3228 Acked Flush 0 Done Done Sent false
8463 3229 Acked Read 6 Done Done Sent false
8464 3230 Acked Write 10 Done Done Sent false
8465 3231 Acked Read 1 Done Done Sent false
8466 3232 Acked Read 2 Done Done Sent false
8467 3233 Acked Write 7 Done Done Sent false
8468 3234 Acked Write 3 Done Done Sent false
8469 3235 Acked Read 6 Done Done Sent false
8470 3236 Acked Write 10 Done Done Sent false
8471 3237 Acked Write 1 Done Done Sent false
8472 3238 Acked Write 10 Done Done Sent false
8473 3239 Acked Write 8 Done Done Sent false
8474 3240 Acked Read 5 Done Done Sent false
8475 3241 Acked Write 10 Done Done Sent false
8476 3242 Acked Read 1 Done Done Sent false
8477 3243 Acked Flush 0 Done Done Sent false
8478 3244 Acked Read 6 Done Done Sent false
8479 3245 Acked Write 7 Done Done Sent false
8480 3246 Acked Read 2 Done Done Sent false
8481 3247 Acked Write 5 Done Done Sent false
8482 3248 Acked Write 6 Done Done Sent false
8483 3249 Acked Write 8 Done Done Sent false
8484 3250 Acked Write 1 Done Done Sent false
8485 3251 Acked Read 8 Done Done Sent false
8486 3252 Acked Flush 0 Done Done Sent false
8487 3253 Acked Read 1 Done Done Sent false
8488 3254 Acked Read 4 Done Done Sent false
8489 3255 Acked Read 10 Done Done Sent false
8490 3256 Acked Write 5 Done Done Sent false
8491 3257 Acked Read 9 Done Done Sent false
8492 3258 Acked Read 2 Done Done Sent false
8493 3259 Acked Write 9 Done Done Sent false
8494 3260 Acked Flush 0 Done Done Sent false
8495 3261 Acked Read 3 Done Done Sent false
8496 3262 Acked Read 8 Done Done Sent false
8497 3263 Acked Write 1 Done Done Sent false
8498 3264 Acked Write 8 Done Done Sent false
8499 3265 Acked Flush 0 Done Done Sent false
8500 3266 Acked Read 3 Done Done Sent false
8501 3267 Acked Write 1 Done Done Sent false
8502 3268 Acked Read 2 Done Done Sent false
8503 3269 Acked Write 5 Done Done Sent false
8504 3270 Acked Write 6 Done Done Sent false
8505 3271 Acked Read 9 Done Done Sent false
8506 3272 Acked Read 9 Done Done Sent false
8507 3273 Acked Read 1 Done Done Sent false
8508 3274 Acked Write 4 Done Done Sent false
8509 3275 Acked Read 5 Done Done Sent false
8510 3276 Acked Write 3 Done Done Sent false
8511 3277 Acked Read 5 Done Done Sent false
8512 3278 Acked Write 1 Done Done Sent false
8513 3279 Acked Read 1 Done Done Sent false
8514 3280 Acked Read 2 Done Done Sent false
8515 3281 Acked Write 7 Done Done Sent false
8516 3282 Acked Read 6 Done Done Sent false
8517 3283 Acked Read 10 Done Done Sent false
8518 3284 Acked Read 7 Done Done Sent false
8519 3285 Acked Read 8 Done Done Sent false
8520 3286 Acked Read 2 Done Done Sent false
8521 3287 Acked Write 3 Done Done Sent false
8522 3288 Acked Read 5 Done Done Sent false
8523 3289 Acked Write 1 Done Done Sent false
8524 3290 Acked Read 1 Done Done Sent false
8525 3291 Acked Write 2 Done Done Sent false
8526 3292 Acked Write 4 Done Done Sent false
8527 3293 Acked Read 3 Done Done Sent false
8528 3294 Acked Write 6 Done Done Sent false
8529 3295 Acked Write 7 Done Done Sent false
8530 3296 Acked Write 3 Done Done Sent false
8531 3297 Acked Read 6 Done Done Sent false
8532 3298 Acked Write 4 Done Done Sent false
8533 3299 Acked Read 2 Done Done Sent false
8534 3300 Acked Read 3 Done Done Sent false
8535 3301 Acked Read 10 Done Done Sent false
8536 3302 Acked Read 2 Done Done Sent false
8537 3303 Acked Write 2 Done Done Sent false
8538 3304 Acked Read 6 Done Done Sent false
8539 3305 Acked Read 7 Done Done Sent false
8540 3306 Acked Read 2 Done Done Sent false
8541 3307 Acked Flush 0 Done Done Sent false
8542 3308 Acked Read 10 Done Done Sent false
8543 3309 Acked Read 8 Done Done Sent false
8544 3310 Acked Read 1 Done Done Sent false
8545 3311 Acked Write 6 Done Done Sent false
8546 3312 Acked Write 10 Done Done Sent false
8547 3313 Acked Read 6 Done Done Sent false
8548 3314 Acked Read 5 Done Done Sent false
8549 3315 Acked Read 6 Done Done Sent false
8550 3316 Acked Write 10 Done Done Sent false
8551 3317 Acked Read 6 Done Done Sent false
8552 3318 Acked Write 10 Done Done Sent false
8553 3319 Acked Read 6 Done Done Sent false
8554 3320 Acked Flush 0 Done Done Sent false
8555 3321 Acked Read 6 Done Done Sent false
8556 3322 Acked Read 5 Done Done Sent false
8557 3323 Acked Read 1 Done Done Sent false
8558 3324 Acked Read 5 Done Done Sent false
8559 3325 Acked Write 6 Done Done Sent false
8560 3326 Acked Read 2 Done Done Sent false
8561 3327 Acked Write 10 Done Done Sent false
8562 3328 Acked Read 7 Done Done Sent false
8563 3329 Acked Flush 0 Done Done Sent false
8564 3330 Acked Write 9 Done Done Sent false
8565 3331 Acked Read 8 Done Done Sent false
8566 3332 Acked Flush 0 Done Done Sent false
8567 3333 Acked Write 9 Done Done Sent false
8568 3334 Acked Write 1 Done Done Sent false
8569 3335 Acked Read 8 Done Done Sent false
8570 3336 Acked Write 6 Done Done Sent false
8571 3337 Acked Write 5 Done Done Sent false
8572 3338 Acked Write 5 Done Done Sent false
8573 3339 Acked Read 7 Done Done Sent false
8574 3340 Acked Write 5 Done Done Sent false
8575 3341 Acked Read 7 Done Done Sent false
8576 3342 Acked Read 3 Done Done Sent false
8577 3343 Acked Write 2 Done Done Sent false
8578 3344 Acked Read 1 Done Done Sent false
8579 3345 Acked Read 8 Done Done Sent false
8580 3346 Acked Write 5 Done Done Sent false
8581 3347 Acked Write 9 Done Done Sent false
8582 3348 Acked Read 3 Done Done Sent false
8583 3349 Acked Write 9 Done Done Sent false
8584 3350 Acked Write 9 Done Done Sent false
8585 3351 Acked Read 9 Done Done Sent false
8586 3352 Acked Read 5 Done Done Sent false
8587 3353 Acked Read 3 Done Done Sent false
8588 3354 Acked Flush 0 Done Done Sent false
8589 3355 Acked Write 7 Done Done Sent false
8590 3356 Acked Flush 0 Done Done Sent false
8591 3357 Acked Write 5 Done Done Sent false
8592 3358 Acked Write 6 Done Done Sent false
8593 3359 Acked Flush 0 Done Done Sent false
8594 3360 Acked Write 8 Done Done Sent false
8595 3361 Acked Write 4 Done Done Sent false
8596 3362 Acked Read 1 Done Done Sent false
8597 3363 Acked Write 7 Done Done Sent false
8598 3364 Acked Write 7 Done Done Sent false
8599 3365 Acked Write 9 Done Done Sent false
8600 3366 Acked Write 9 Done Done Sent false
8601 3367 Acked Flush 0 Done Done Sent false
8602 3368 Acked Read 4 Done Done Sent false
8603 3369 Acked Write 9 Done Done Sent false
8604 3370 Acked Write 7 Done Done Sent false
8605 3371 Acked Write 6 Done Done Sent false
8606 3372 Acked Read 5 Done Done Sent false
8607 3373 Acked Read 6 Done Done Sent false
8608 3374 Acked Read 3 Done Done Sent false
8609 3375 Acked Read 6 Done Done Sent false
8610 3376 Acked Read 4 Done Done Sent false
8611 3377 Acked Read 3 Done Done Sent false
8612 3378 Acked Write 2 Done Done Sent false
8613 3379 Acked Write 7 Done Done Sent false
8614 3380 Acked Read 6 Done Done Sent false
8615 3381 Acked Write 6 Done Done Sent false
8616 3382 Acked Write 6 Done Done Sent false
8617 3383 Acked Write 4 Done Done Sent false
8618 3384 Acked Write 8 Done Done Sent false
8619 3385 Acked Read 4 Done Done Sent false
8620 3386 Acked Write 2 Done Done Sent false
8621 3387 Acked Write 6 Done Done Sent false
8622 3388 Acked Read 7 Done Done Sent false
8623 3389 Acked Write 2 Done Done Sent false
8624 3390 Acked Read 10 Done Done Sent false
8625 3391 Acked Read 2 Done Done Sent false
8626 3392 Acked Write 4 Done Done Sent false
8627 3393 Acked Read 2 Done Done Sent false
8628 3394 Acked Write 1 Done Done Sent false
8629 3395 Acked Read 9 Done Done Sent false
8630 3396 Acked Write 9 Done Done Sent false
8631 3397 Acked Read 4 Done Done Sent false
8632 3398 Acked Read 9 Done Done Sent false
8633 3399 Acked Write 4 Done Done Sent false
8634 3400 Acked Read 8 Done Done Sent false
8635 3401 Acked Write 6 Done Done Sent false
8636 3402 Acked Read 8 Done Done Sent false
8637 3403 Acked Read 3 Done Done Sent false
8638 3404 Acked Write 9 Done Done Sent false
8639 3405 Acked Write 9 Done Done Sent false
8640 3406 Acked Read 7 Done Done Sent false
8641 3407 Acked Write 6 Done Done Sent false
8642 3408 Acked Read 6 Done Done Sent false
8643 3409 Acked Read 2 Done Done Sent false
8644 3410 Acked Write 8 Done Done Sent false
8645 3411 Acked Read 2 Done Done Sent false
8646 3412 Acked Read 7 Done Done Sent false
8647 3413 Acked Write 9 Done Done Sent false
8648 STATES DS:0 DS:1 DS:2 TOTAL
8649 Sent 0 0 301 301
8650 Done 301 301 0 602
8651 Skipped 0 0 0 0
8652 Error 0 0 0 0
8653 Last Flush: Some(JobId(3367)) Some(JobId(3367)) Some(JobId(3112))
8654 Downstairs last five completed: 3112 3111 3110 3109 3108
8655 Upstairs last five completed: 3413 3412 3411 3410 3409
8656 CLIENT: Up:0 ds:301 act:2
86572025-02-25T20:48:31.239ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86582025-02-25T20:48:32.241ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86592025-02-25T20:48:32.389ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86602025-02-25T20:48:32.389ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43186) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86612025-02-25T20:48:32.390ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86622025-02-25T20:48:32.390ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86632025-02-25T20:48:32.390ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86642025-02-25T20:48:32.391ZINFOcrucible: send last flush ID to this DS: Some(JobId(3112)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86652025-02-25T20:48:32.391ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86662025-02-25T20:48:32.391ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3112)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86672025-02-25T20:48:32.391ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3112)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8668 ----------------------------------------------------------------
8669 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8670 Downstairs last five completed: 3414 3413 3412 3411 3410
8671 Upstairs last five completed: 3414 3413 3412 3411 3410
8672 CLIENT: Up:0 ds:0 act:3
8673 Replay: All jobs finished
8674 Main task has work to do, go find it
8675 got dsc Stop(2)
8676 stop 2
8677 Feb 25 20:48:34.635 INFO request completed, latency_us: 386, response_code: 204, uri[8830] Got stop action so:false kr:false
8678 : /stop/cid/2, method: POST, req_id: ef3301af-8d83-4c42-a2dc-1d99eb78e704, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
86792025-02-25T20:48:34.636ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86802025-02-25T20:48:34.636ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86812025-02-25T20:48:34.636ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
86822025-02-25T20:48:34.636ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
8683 Feb 25 20:48:34.636 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: 89773de6-823a-4771-9563-9428313ac371, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
8684 [8830] Exited with: ExitStatus(unix_wait_status(9))
8685 [8830][2] reports Exit
8686 Feb 25 20:48:38.639 INFO request completed, latency_us: 266, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3e463323-79df-4cf1-9aca-e1390c71e630, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
8687 001/300 Read block 2789 len 36864
8688 002/300 Write block 4893 len 32768 data: 4 4 4 5 5 5 4 4
8689 003/300 Flush
8690 004/300 Write block 980 len 28672 data: 3 3 3 4 5 5 5
8691 005/300 Read block 2632 len 12288
8692 006/300 Read block 3418 len 32768
8693 007/300 Flush
8694 008/300 Write block 2314 len 28672 data: 3 3 3 3 3 3 3
8695 009/300 Write block 3525 len 8192 data: 3 3
8696 010/300 Read block 290 len 4096
8697 011/300 Read block 56 len 36864
8698 012/300 Write block 2362 len 12288 data: 2 3 3
8699 013/300 Read block 4800 len 24576
8700 014/300 Read block 2377 len 32768
8701 015/300 Write block 4082 len 12288 data: 3 2 2
8702 016/300 Write block 4461 len 12288 data: 4 4 3
8703 017/300 Read block 2101 len 20480
8704 018/300 Flush
8705 019/300 Read block 811 len 20480
8706 020/300 Read block 2077 len 16384
8707 021/300 Write block 214 len 28672 data: 4 4 4 3 4 4 4
8708 022/300 Flush
8709 023/300 Write block 4231 len 32768 data: 3 3 3 3 3 2 2 2
8710 024/300 Read block 3140 len 32768
8711 025/300 Read block 4847 len 20480
8712 026/300 Read block 1929 len 20480
8713 027/300 Write block 237 len 4096 data: 4
8714 028/300 Write block 1493 len 12288 data: 2 3 3
8715 029/300 Read block 2065 len 28672
8716 030/300 Read block 1642 len 16384
8717 031/300 Read block 4295 len 20480
8718 032/300 Write block 3121 len 24576 data: 2 2 2 2 2 3
8719 033/300 Read block 151 len 40960
8720 034/300 Flush
8721 035/300 Read block 2043 len 8192
8722 036/300 Write block 253 len 8192 data: 4 4
8723 037/300 Read block 2543 len 32768
8724 038/300 Write block 2678 len 32768 data: 2 2 4 5 5 5 5 3
8725 039/300 Read block 4416 len 28672
8726 040/300 Write block 4568 len 32768 data: 3 3 3 3 3 2 2 2
8727 041/300 Read block 4079 len 4096
8728 042/300 Flush
8729 043/300 Read block 4624 len 40960
8730 044/300 Write block 55 len 4096 data: 3
8731 045/300 Read block 1463 len 40960
8732 046/300 Write block 4596 len 36864 data: 2 2 3 2 3 3 4 4 4
8733 047/300 Write block 1196 len 32768 data: 3 3 3 3 3 3 4 3
8734 048/300 Write block 2724 len 16384 data: 2 2 2 2
8735 049/300 Read block 1332 len 36864
8736 050/300 Read block 864 len 8192
8737 051/300 Read block 3616 len 24576
8738 052/300 Write block 2007 len 40960 data: 3 3 5 5 3 3 3 3 2 2
8739 053/300 Write block 1981 len 16384 data: 2 2 2 2
8740 054/300 Write block 220 len 4096 data: 5
8741 055/300 Write block 2026 len 32768 data: 2 2 2 2 2 3 3 2
8742 056/300 Flush
8743 057/300 Read block 3716 len 16384
8744 058/300 Write block 3722 len 20480 data: 4 4 4 3 3
8745 059/300 Read block 1794 len 8192
8746 060/300 Write block 945 len 16384 data: 3 3 3 3
8747 061/300 Read block 2210 len 36864
8748 062/300 Read block 2220 len 28672
8749 063/300 Write block 2008 len 4096 data: 4
8750 064/300 Write block 3334 len 16384 data: 4 4 4 4
8751 065/300 Read block 1631 len 40960
8752 066/300 Read block 666 len 28672
8753 067/300 Write block 3001 len 16384 data: 3 3 4 3
8754 068/300 Read block 312 len 28672
8755 069/300 Write block 1663 len 4096 data: 2
8756 070/300 Write block 369 len 28672 data: 3 3 3 2 2 3 3
8757 071/300 Read block 3356 len 32768
8758 072/300 Write block 4851 len 20480 data: 4 3 3 3 3
8759 073/300 Write block 1100 len 36864 data: 3 3 3 3 3 2 2 2 2
8760 074/300 Write block 2951 len 32768 data: 2 2 2 2 2 3 3 3
8761 075/300 Write block 3126 len 8192 data: 4 3
8762 076/300 Read block 2916 len 24576
8763 077/300 Read block 1681 len 24576
8764 078/300 Write block 2281 len 36864 data: 2 2 2 2 2 2 2 2 2
8765 079/300 Read block 4678 len 12288
8766 080/300 Write block 2497 len 12288 data: 2 2 3
8767 081/300 Read block 4932 len 4096
8768 082/300 Flush
8769 083/300 Read block 4858 len 32768
8770 084/300 Read block 3785 len 16384
8771 085/300 Read block 2475 len 28672
8772 086/300 Write block 1744 len 40960 data: 2 2 2 2 3 3 4 3 3 3
8773 087/300 Read block 3809 len 36864
8774 088/300 Read block 1216 len 32768
8775 089/300 Read block 4869 len 40960
8776 090/300 Write block 1862 len 12288 data: 5 5 5
8777 091/300 Write block 4263 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8778 092/300 Read block 776 len 32768
8779 093/300 Read block 529 len 28672
8780 094/300 Read block 1 len 24576
8781 095/300 Read block 1036 len 40960
8782 096/300 Write block 2996 len 12288 data: 4 4 3
8783 097/300 Write block 4669 len 28672 data: 4 4 2 2 2 2 2
8784 098/300 Read block 4204 len 28672
8785 099/300 Read block 4873 len 24576
8786 100/300 Read block 2269 len 20480
8787 101/300 Read block 2327 len 40960
8788 102/300 Read block 3363 len 40960
8789 103/300 Read block 495 len 24576
8790 104/300 Read block 645 len 16384
8791 105/300 Read block 4942 len 24576
8792 106/300 Read block 2447 len 4096
8793 107/300 Read block 1884 len 20480
8794 108/300 Write block 4228 len 32768 data: 3 3 3 4 4 4 4 4
8795 109/300 Write block 2338 len 12288 data: 5 5 5
8796 110/300 Flush
8797 111/300 Write block 2280 len 32768 data: 2 3 3 3 3 3 3 3
8798 112/300 Write block 3244 len 16384 data: 4 3 3 2
8799 113/300 Read block 4490 len 40960
8800 114/300 Read block 719 len 8192
8801 115/300 Read block 4802 len 4096
8802 116/300 Write block 410 len 28672 data: 2 2 3 3 4 4 4
8803 117/300 Write block 4749 len 12288 data: 2 2 3
8804 118/300 Flush
8805 119/300 Read block 4049 len 20480
8806 120/300 Write block 584 len 4096 data: 4
8807 121/300 Read block 2438 len 40960
8808 122/300 Read block 4375 len 20480
8809 123/300 Write block 943 len 24576 data: 3 3 4 4 4 4
8810 124/300 Write block 4309 len 24576 data: 2 2 2 2 2 3
8811 125/300 Read block 2348 len 32768
8812 126/300 Write block 4387 len 16384 data: 3 3 3 3
8813 127/300 Read block 4557 len 36864
8814 128/300 Read block 621 len 32768
8815 129/300 Write block 2617 len 36864 data: 2 2 3 3 3 4 5 4 5
8816 130/300 Write block 577 len 20480 data: 5 5 6 6 5
8817 131/300 Flush
8818 132/300 Write block 3303 len 24576 data: 7 6 6 4 2 2
8819 133/300 Read block 2143 len 24576
8820 134/300 Read block 3303 len 32768
8821 135/300 Read block 1044 len 20480
8822 136/300 Write block 1416 len 28672 data: 4 4 3 3 3 3 3
8823 137/300 Read block 504 len 32768
8824 138/300 Write block 1690 len 24576 data: 2 2 2 2 2 2
8825 139/300 Write block 3706 len 28672 data: 4 5 5 4 4 4 4
8826 140/300 Read block 2810 len 4096
8827 141/300 Read block 2427 len 4096
8828 142/300 Read block 2021 len 12288
8829 143/300 Write block 1056 len 40960 data: 3 3 2 3 3 3 2 3 2 2
8830 144/300 Write block 2641 len 28672 data: 2 3 3 3 3 3 5
8831 145/300 Read block 2151 len 16384
8832 146/300 Write block 1182 len 20480 data: 2 2 2 2 3
8833 147/300 Write block 3556 len 8192 data: 3 3
8834 148/300 Read block 928 len 8192
8835 149/300 Write block 3136 len 40960 data: 4 4 3 3 3 2 3 3 4 4
8836 150/300 Flush
8837 151/300 Flush
8838 152/300 Read block 401 len 32768
8839 153/300 Write block 4621 len 8192 data: 2 2
8840 154/300 Write block 2868 len 12288 data: 4 4 4
8841 155/300 Read block 3109 len 4096
8842 156/300 Write block 1705 len 20480 data: 2 2 2 2 2
8843 157/300 Read block 2186 len 24576
8844 158/300 Flush
8845 159/300 Read block 528 len 12288
8846 160/300 Read block 4680 len 32768
8847 161/300 Write block 590 len 8192 data: 2 3
8848 162/300 Read block 348 len 8192
8849 163/300 Read block 1524 len 12288
8850 164/300 Write block 593 len 20480 data: 3 3 3 3 3
8851 165/300 Read block 1467 len 8192
8852 166/300 Write block 2055 len 24576 data: 2 3 3 3 3 3
8853 167/300 Read block 2858 len 32768
8854 168/300 Write block 2566 len 24576 data: 2 2 2 2 2 2
8855 169/300 Read block 2654 len 8192
8856 170/300 Read block 4578 len 32768
8857 171/300 Write block 3633 len 40960 data: 3 3 3 3 4 4 4 5 6 7
8858 172/300 Read block 2470 len 12288
8859 173/300 Flush
8860 174/300 Read block 4763 len 36864
8861 175/300 Read block 2091 len 28672
8862 176/300 Read block 123 len 20480
8863 177/300 Write block 1388 len 12288 data: 3 3 3
8864 178/300 Read block 3335 len 20480
8865 179/300 Write block 4611 len 40960 data: 3 3 3 3 3 3 4 3 2 2
8866 180/300 Read block 1202 len 24576
8867 181/300 Read block 4426 len 28672
8868 182/300 Read block 3565 len 4096
8869 183/300 Read block 3605 len 28672
8870 184/300 Read block 2002 len 4096
8871 185/300 Read block 4628 len 36864
8872 186/300 Write block 4896 len 12288 data: 6 6 6
8873 187/300 Write block 1214 len 8192 data: 3 3
8874 188/300 Flush
8875 189/300 Read block 4258 len 12288
8876 190/300 Flush
8877 191/300 Write block 2272 len 8192 data: 4 3
8878 192/300 Read block 2098 len 8192
8879 193/300 Read block 1080 len 24576
8880 194/300 Write block 925 len 20480 data: 3 3 2 2 2
8881 195/300 Write block 1979 len 20480 data: 2 2 3 3 3
8882 196/300 Read block 4432 len 8192
8883 197/300 Read block 1924 len 8192
8884 198/300 Read block 4953 len 16384
8885 199/300 Write block 1958 len 28672 data: 3 3 2 2 2 2 2
8886 200/300 Write block 4748 len 12288 data: 2 3 3
8887 201/300 Read block 3973 len 8192
8888 202/300 Read block 2633 len 8192
8889 203/300 Read block 3046 len 32768
8890 204/300 Read block 740 len 20480
8891 205/300 Read block 4758 len 24576
8892 206/300 Read block 2463 len 4096
8893 207/300 Flush
8894 208/300 Read block 2054 len 28672
8895 209/300 Read block 4464 len 8192
8896 210/300 Flush
8897 211/300 Write block 3852 len 32768 data: 2 2 3 3 3 3 2 2
8898 212/300 Write block 1629 len 12288 data: 4 4 4
8899 213/300 Read block 3177 len 36864
8900 214/300 Write block 1088 len 36864 data: 2 2 2 2 2 2 2 3 3
8901 215/300 Write block 1506 len 8192 data: 4 3
8902 216/300 Read block 1142 len 20480
8903 217/300 Read block 19 len 12288
8904 218/300 Read block 1074 len 12288
8905 219/300 Write block 3616 len 16384 data: 4 5 4 5
8906 220/300 Read block 2596 len 4096
8907 221/300 Flush
8908 222/300 Write block 4418 len 20480 data: 5 4 5 6 7
8909 223/300 Read block 2212 len 24576
8910 224/300 Flush
8911 225/300 Read block 710 len 20480
8912 226/300 Write block 332 len 32768 data: 5 4 4 4 4 2 2 2
8913 227/300 Read block 1265 len 8192
8914 228/300 Read block 3322 len 24576
8915 229/300 Read block 2024 len 16384
8916 230/300 Read block 3674 len 16384
8917 231/300 Read block 3995 len 28672
8918 232/300 Write block 966 len 8192 data: 2 2
8919 233/300 Write block 1321 len 40960 data: 4 4 6 6 7 6 5 5 4 3
8920 234/300 Write block 306 len 28672 data: 3 3 3 3 3 3 3
8921 235/300 Read block 1364 len 12288
8922 236/300 Read block 851 len 32768
8923 237/300 Flush
8924 238/300 Write block 299 len 4096 data: 2
8925 239/300 Write block 418 len 40960 data: 3 3 2 2 4 4 4 5 5 4
8926 240/300 Write block 1764 len 20480 data: 3 3 2 2 3
8927 241/300 Read block 3581 len 32768
8928 242/300 Read block 1057 len 28672
8929 243/300 Read block 4120 len 4096
8930 244/300 Write block 2259 len 12288 data: 5 4 4
8931 245/300 Write block 709 len 28672 data: 2 3 4 4 5 4 4
8932 246/300 Write block 171 len 24576 data: 2 2 2 2 2 3
8933 247/300 Write block 1227 len 4096 data: 2
8934 248/300 Write block 4515 len 4096 data: 3
8935 249/300 Write block 1790 len 36864 data: 5 4 5 5 4 4 4 4 4
8936 250/300 Flush
8937 251/300 Read block 236 len 20480
8938 252/300 Write block 310 len 12288 data: 4 4 4
8939 253/300 Write block 1874 len 36864 data: 4 5 5 5 5 5 5 3 3
8940 254/300 Write block 4226 len 8192 data: 3 3
8941 255/300 Write block 4278 len 12288 data: 2 2 2
8942 256/300 Flush
8943 257/300 Write block 4771 len 28672 data: 2 2 2 2 2 2 3
8944 258/300 Read block 3092 len 40960
8945 259/300 Write block 3008 len 40960 data: 5 6 5 5 5 4 3 3 3 2
8946 260/300 Read block 60 len 40960
8947 261/300 Read block 2123 len 12288
8948 262/300 Write block 550 len 8192 data: 3 3
8949 263/300 Read block 2557 len 24576
8950 264/300 Write block 1062 len 28672 data: 3 4 3 3 2 4 4
8951 265/300 Read block 4106 len 20480
8952 266/300 Flush
8953 267/300 Write block 4390 len 24576 data: 4 5 4 3 3 3
8954 268/300 Read block 2413 len 40960
8955 269/300 Read block 3863 len 40960
8956 270/300 Read block 166 len 36864
8957 271/300 Read block 4583 len 36864
8958 272/300 Write block 1459 len 20480 data: 6 5 5 6 4
8959 273/300 Flush
8960 274/300 Read block 4667 len 12288
8961 275/300 Read block 736 len 12288
8962 276/300 Write block 3638 len 12288 data: 5 5 6
8963 277/300 Read block 3278 len 36864
8964 278/300 Write block 855 len 36864 data: 3 4 3 3 3 3 3 3 4
8965 279/300 Write block 2838 len 32768 data: 5 5 5 4 3 3 3 4
8966 280/300 Write block 3268 len 12288 data: 3 2 3
8967 281/300 Read block 2239 len 16384
8968 282/300 Write block 2986 len 12288 data: 5 6 8
8969 283/300 Read block 3459 len 20480
8970 284/300 Write block 2674 len 32768 data: 3 3 2 2 3 3 5 6
8971 285/300 Write block 30 len 24576 data: 3 2 2 2 2 3
8972 286/300 Read block 1014 len 16384
8973 287/300 Read block 675 len 40960
8974 288/300 Read block 3230 len 20480
8975 289/300 Write block 2033 len 24576 data: 3 2 3 4 4 3
8976 290/300 Write block 1711 len 40960 data: 2 2 2 2 2 2 2 2 3 3
8977 291/300 Flush
8978 292/300 Read block 2948 len 12288
8979 293/300 Read block 1895 len 28672
8980 294/300 Read block 907 len 36864
8981 295/300 Write block 1632 len 16384 data: 4 2 2 2
8982 296/300 Read block 372 len 40960
8983 297/300 Read block 4599 len 12288
8984 298/300 Read block 2136 len 20480
8985 299/300 Flush
8986 300/300 Write block 353 len 8192 data: 4 4
8987 301/300 Read block 3441 len 28672
8988 Main task has work to do, go find it
8989 got dsc Start(2)
8990 start 2
8991 Feb 25 20:48:38.893 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: dff8e67c-a4c9-42e8-b8c3-538e5d95be24, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
8992 Replay: started 2, returned:Ok(())
8993 ----------------------------------------------------------------
8994 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8995 States: Active Active Offline
8996 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8997 3415 Acked Read 9 Done Done Sent false
8998 3416 Acked Write 8 Done Done Sent false
8999 3417 Acked Flush 0 Done Done Sent false
9000 3418 Acked Write 7 Done Done Sent false
9001 3419 Acked Read 3 Done Done Sent false
9002 3420 Acked Read 8 Done Done Sent false
9003 3421 Acked Flush 0 Done Done Sent false
9004 3422 Acked Write 7 Done Done Sent false
9005 3423 Acked Write 2 Done Done Sent false
9006 3424 Acked Read 1 Done Done Sent false
9007 3425 Acked Read 9 Done Done Sent false
9008 3426 Acked Write 3 Done Done Sent false
9009 3427 Acked Read 6 Done Done Sent false
9010 3428 Acked Read 8 Done Done Sent false
9011 3429 Acked Write 3 Done Done Sent false
9012 3430 Acked Write 3 Done Done Sent false
9013 3431 Acked Read 5 Done Done Sent false
9014 3432 Acked Flush 0 Done Done Sent false
9015 3433 Acked Read 5 Done Done Sent false
9016 3434 Acked Read 4 Done Done Sent false
9017 3435 Acked Write 7 Done Done Sent false
9018 3436 Acked Flush 0 Done Done Sent false
9019 3437 Acked Write 8 Done Done Sent false
9020 3438 Acked Read 8 Done Done Sent false
9021 3439 Acked Read 5 Done Done Sent false
9022 3440 Acked Read 5 Done Done Sent false
9023 3441 Acked Write 1 Done Done Sent false
9024 3442 Acked Write 3 Done Done Sent false
9025 3443 Acked Read 7 Done Done Sent false
9026 3444 Acked Read 4 Done Done Sent false
9027 3445 Acked Read 5 Done Done Sent false
9028 3446 Acked Write 6 Done Done Sent false
9029 3447 Acked Read 10 Done Done Sent false
9030 3448 Acked Flush 0 Done Done Sent false
9031 3449 Acked Read 2 Done Done Sent false
9032 3450 Acked Write 2 Done Done Sent false
9033 3451 Acked Read 8 Done Done Sent false
9034 3452 Acked Write 8 Done Done Sent false
9035 3453 Acked Read 7 Done Done Sent false
9036 3454 Acked Write 8 Done Done Sent false
9037 3455 Acked Read 1 Done Done Sent false
9038 3456 Acked Flush 0 Done Done Sent false
9039 3457 Acked Read 10 Done Done Sent false
9040 3458 Acked Write 1 Done Done Sent false
9041 3459 Acked Read 10 Done Done Sent false
9042 3460 Acked Write 9 Done Done Sent false
9043 3461 Acked Write 8 Done Done Sent false
9044 3462 Acked Write 4 Done Done Sent false
9045 3463 Acked Read 9 Done Done Sent false
9046 3464 Acked Read 2 Done Done Sent false
9047 3465 Acked Read 6 Done Done Sent false
9048 3466 Acked Write 10 Done Done Sent false
9049 3467 Acked Write 4 Done Done Sent false
9050 3468 Acked Write 1 Done Done Sent false
9051 3469 Acked Write 8 Done Done Sent false
9052 3470 Acked Flush 0 Done Done Sent false
9053 3471 Acked Read 4 Done Done Sent false
9054 3472 Acked Write 5 Done Done Sent false
9055 3473 Acked Read 2 Done Done Sent false
9056 3474 Acked Write 4 Done Done Sent false
9057 3475 Acked Read 9 Done Done Sent false
9058 3476 Acked Read 7 Done Done Sent false
9059 3477 Acked Write 1 Done Done Sent false
9060 3478 Acked Write 4 Done Done Sent false
9061 3479 Acked Read 10 Done Done Sent false
9062 3480 Acked Read 7 Done Done Sent false
9063 3481 Acked Write 4 Done Done Sent false
9064 3482 Acked Read 7 Done Done Sent false
9065 3483 Acked Write 1 Done Done Sent false
9066 3484 Acked Write 7 Done Done Sent false
9067 3485 Acked Read 8 Done Done Sent false
9068 3486 Acked Write 5 Done Done Sent false
9069 3487 Acked Write 9 Done Done Sent false
9070 3488 Acked Write 8 Done Done Sent false
9071 3489 Acked Write 2 Done Done Sent false
9072 3490 Acked Read 6 Done Done Sent false
9073 3491 Acked Read 6 Done Done Sent false
9074 3492 Acked Write 9 Done Done Sent false
9075 3493 Acked Read 3 Done Done Sent false
9076 3494 Acked Write 3 Done Done Sent false
9077 3495 Acked Read 1 Done Done Sent false
9078 3496 Acked Flush 0 Done Done Sent false
9079 3497 Acked Read 8 Done Done Sent false
9080 3498 Acked Read 4 Done Done Sent false
9081 3499 Acked Read 7 Done Done Sent false
9082 3500 Acked Write 10 Done Done Sent false
9083 3501 Acked Read 9 Done Done Sent false
9084 3502 Acked Read 8 Done Done Sent false
9085 3503 Acked Read 10 Done Done Sent false
9086 3504 Acked Write 3 Done Done Sent false
9087 3505 Acked Write 10 Done Done Sent false
9088 3506 Acked Read 8 Done Done Sent false
9089 3507 Acked Read 7 Done Done Sent false
9090 3508 Acked Read 6 Done Done Sent false
9091 3509 Acked Read 10 Done Done Sent false
9092 3510 Acked Write 3 Done Done Sent false
9093 3511 Acked Write 7 Done Done Sent false
9094 3512 Acked Read 7 Done Done Sent false
9095 3513 Acked Read 6 Done Done Sent false
9096 3514 Acked Read 5 Done Done Sent false
9097 3515 Acked Read 10 Done Done Sent false
9098 3516 Acked Read 10 Done Done Sent false
9099 3517 Acked Read 6 Done Done Sent false
9100 3518 Acked Read 4 Done Done Sent false
9101 3519 Acked Read 6 Done Done Sent false
9102 3520 Acked Read 1 Done Done Sent false
9103 3521 Acked Read 5 Done Done Sent false
9104 3522 Acked Write 8 Done Done Sent false
9105 3523 Acked Write 3 Done Done Sent false
9106 3524 Acked Flush 0 Done Done Sent false
9107 3525 Acked Write 8 Done Done Sent false
9108 3526 Acked Write 4 Done Done Sent false
9109 3527 Acked Read 10 Done Done Sent false
9110 3528 Acked Read 2 Done Done Sent false
9111 3529 Acked Read 1 Done Done Sent false
9112 3530 Acked Write 7 Done Done Sent false
9113 3531 Acked Write 3 Done Done Sent false
9114 3532 Acked Flush 0 Done Done Sent false
9115 3533 Acked Read 5 Done Done Sent false
9116 3534 Acked Write 1 Done Done Sent false
9117 3535 Acked Read 10 Done Done Sent false
9118 3536 Acked Read 5 Done Done Sent false
9119 3537 Acked Write 6 Done Done Sent false
9120 3538 Acked Write 6 Done Done Sent false
9121 3539 Acked Read 8 Done Done Sent false
9122 3540 Acked Write 4 Done Done Sent false
9123 3541 Acked Read 9 Done Done Sent false
9124 3542 Acked Read 8 Done Done Sent false
9125 3543 Acked Write 9 Done Done Sent false
9126 3544 Acked Write 5 Done Done Sent false
9127 3545 Acked Flush 0 Done Done Sent false
9128 3546 Acked Write 6 Done Done Sent false
9129 3547 Acked Read 6 Done Done Sent false
9130 3548 Acked Read 8 Done Done Sent false
9131 3549 Acked Read 5 Done Done Sent false
9132 3550 Acked Write 7 Done Done Sent false
9133 3551 Acked Read 8 Done Done Sent false
9134 3552 Acked Write 6 Done Done Sent false
9135 3553 Acked Write 7 Done Done Sent false
9136 3554 Acked Read 1 Done Done Sent false
9137 3555 Acked Read 1 Done Done Sent false
9138 3556 Acked Read 3 Done Done Sent false
9139 3557 Acked Write 10 Done Done Sent false
9140 3558 Acked Write 7 Done Done Sent false
9141 3559 Acked Read 4 Done Done Sent false
9142 3560 Acked Write 5 Done Done Sent false
9143 3561 Acked Write 2 Done Done Sent false
9144 3562 Acked Read 2 Done Done Sent false
9145 3563 Acked Write 10 Done Done Sent false
9146 3564 Acked Flush 0 Done Done Sent false
9147 3565 Acked Flush 0 Done Done Sent false
9148 3566 Acked Read 8 Done Done Sent false
9149 3567 Acked Write 2 Done Done Sent false
9150 3568 Acked Write 3 Done Done Sent false
9151 3569 Acked Read 1 Done Done Sent false
9152 3570 Acked Write 5 Done Done Sent false
9153 3571 Acked Read 6 Done Done Sent false
9154 3572 Acked Flush 0 Done Done Sent false
9155 3573 Acked Read 3 Done Done Sent false
9156 3574 Acked Read 8 Done Done Sent false
9157 3575 Acked Write 2 Done Done Sent false
9158 3576 Acked Read 2 Done Done Sent false
9159 3577 Acked Read 3 Done Done Sent false
9160 3578 Acked Write 5 Done Done Sent false
9161 3579 Acked Read 2 Done Done Sent false
9162 3580 Acked Write 6 Done Done Sent false
9163 3581 Acked Read 8 Done Done Sent false
9164 3582 Acked Write 6 Done Done Sent false
9165 3583 Acked Read 2 Done Done Sent false
9166 3584 Acked Read 8 Done Done Sent false
9167 3585 Acked Write 10 Done Done Sent false
9168 3586 Acked Read 3 Done Done Sent false
9169 3587 Acked Flush 0 Done Done Sent false
9170 3588 Acked Read 9 Done Done Sent false
9171 3589 Acked Read 7 Done Done Sent false
9172 3590 Acked Read 5 Done Done Sent false
9173 3591 Acked Write 3 Done Done Sent false
9174 3592 Acked Read 5 Done Done Sent false
9175 3593 Acked Write 10 Done Done Sent false
9176 3594 Acked Read 6 Done Done Sent false
9177 3595 Acked Read 7 Done Done Sent false
9178 3596 Acked Read 1 Done Done Sent false
9179 3597 Acked Read 7 Done Done Sent false
9180 3598 Acked Read 1 Done Done Sent false
9181 3599 Acked Read 9 Done Done Sent false
9182 3600 Acked Write 3 Done Done Sent false
9183 3601 Acked Write 2 Done Done Sent false
9184 3602 Acked Flush 0 Done Done Sent false
9185 3603 Acked Read 3 Done Done Sent false
9186 3604 Acked Flush 0 Done Done Sent false
9187 3605 Acked Write 2 Done Done Sent false
9188 3606 Acked Read 2 Done Done Sent false
9189 3607 Acked Read 6 Done Done Sent false
9190 3608 Acked Write 5 Done Done Sent false
9191 3609 Acked Write 5 Done Done Sent false
9192 3610 Acked Read 2 Done Done Sent false
9193 3611 Acked Read 2 Done Done Sent false
9194 3612 Acked Read 4 Done Done Sent false
9195 3613 Acked Write 7 Done Done Sent false
9196 3614 Acked Write 3 Done Done Sent false
9197 3615 Acked Read 2 Done Done Sent false
9198 3616 Acked Read 2 Done Done Sent false
9199 3617 Acked Read 8 Done Done Sent false
9200 3618 Acked Read 5 Done Done Sent false
9201 3619 Acked Read 6 Done Done Sent false
9202 3620 Acked Read 1 Done Done Sent false
9203 3621 Acked Flush 0 Done Done Sent false
9204 3622 Acked Read 7 Done Done Sent false
9205 3623 Acked Read 2 Done Done Sent false
9206 3624 Acked Flush 0 Done Done Sent false
9207 3625 Acked Write 8 Done Done Sent false
9208 3626 Acked Write 3 Done Done Sent false
9209 3627 Acked Read 9 Done Done Sent false
9210 3628 Acked Write 9 Done Done Sent false
9211 3629 Acked Write 2 Done Done Sent false
9212 3630 Acked Read 5 Done Done Sent false
9213 3631 Acked Read 3 Done Done Sent false
9214 3632 Acked Read 3 Done Done Sent false
9215 3633 Acked Write 4 Done Done Sent false
9216 3634 Acked Read 1 Done Done Sent false
9217 3635 Acked Flush 0 Done Done Sent false
9218 3636 Acked Write 5 Done Done Sent false
9219 3637 Acked Read 6 Done Done Sent false
9220 3638 Acked Flush 0 Done Done Sent false
9221 3639 Acked Read 5 Done Done Sent false
9222 3640 Acked Write 8 Done Done Sent false
9223 3641 Acked Read 2 Done Done Sent false
9224 3642 Acked Read 6 Done Done Sent false
9225 3643 Acked Read 4 Done Done Sent false
9226 3644 Acked Read 4 Done Done Sent false
9227 3645 Acked Read 7 Done Done Sent false
9228 3646 Acked Write 2 Done Done Sent false
9229 3647 Acked Write 10 Done Done Sent false
9230 3648 Acked Write 7 Done Done Sent false
9231 3649 Acked Read 3 Done Done Sent false
9232 3650 Acked Read 8 Done Done Sent false
9233 3651 Acked Flush 0 Done Done Sent false
9234 3652 Acked Write 1 Done Done Sent false
9235 3653 Acked Write 10 Done Done Sent false
9236 3654 Acked Write 5 Done Done Sent false
9237 3655 Acked Read 8 Done Done Sent false
9238 3656 Acked Read 7 Done Done Sent false
9239 3657 Acked Read 1 Done Done Sent false
9240 3658 Acked Write 3 Done Done Sent false
9241 3659 Acked Write 7 Done Done Sent false
9242 3660 Acked Write 6 Done Done Sent false
9243 3661 Acked Write 1 Done Done Sent false
9244 3662 Acked Write 1 Done Done Sent false
9245 3663 Acked Write 9 Done Done Sent false
9246 3664 Acked Flush 0 Done Done Sent false
9247 3665 Acked Read 5 Done Done Sent false
9248 3666 Acked Write 3 Done Done Sent false
9249 3667 Acked Write 9 Done Done Sent false
9250 3668 Acked Write 2 Done Done Sent false
9251 3669 Acked Write 3 Done Done Sent false
9252 3670 Acked Flush 0 Done Done Sent false
9253 3671 Acked Write 7 Done Done Sent false
9254 3672 Acked Read 10 Done Done Sent false
9255 3673 Acked Write 10 Done Done Sent false
9256 3674 Acked Read 10 Done Done Sent false
9257 3675 Acked Read 3 Done Done Sent false
9258 3676 Acked Write 2 Done Done Sent false
9259 3677 Acked Read 6 Done Done Sent false
9260 3678 Acked Write 7 Done Done Sent false
9261 3679 Acked Read 5 Done Done Sent false
9262 3680 Acked Flush 0 Done Done Sent false
9263 3681 Acked Write 6 Done Done Sent false
9264 3682 Acked Read 10 Done Done Sent false
9265 3683 Acked Read 10 Done Done Sent false
9266 3684 Acked Read 9 Done Done Sent false
9267 3685 Acked Read 9 Done Done Sent false
9268 3686 Acked Write 5 Done Done Sent false
9269 3687 Acked Flush 0 Done Done Sent false
9270 3688 Acked Read 3 Done Done Sent false
9271 3689 Acked Read 3 Done Done Sent false
9272 3690 Acked Write 3 Done Done Sent false
9273 3691 Acked Read 9 Done Done Sent false
9274 3692 Acked Write 9 Done Done Sent false
9275 3693 Acked Write 8 Done Done Sent false
9276 3694 Acked Write 3 Done Done Sent false
9277 3695 Acked Read 4 Done Done Sent false
9278 3696 Acked Write 3 Done Done Sent false
9279 3697 Acked Read 5 Done Done Sent false
9280 3698 Acked Write 8 Done Done Sent false
9281 3699 Acked Write 6 Done Done Sent false
9282 3700 Acked Read 4 Done Done Sent false
9283 3701 Acked Read 10 Done Done Sent false
9284 3702 Acked Read 5 Done Done Sent false
9285 3703 Acked Write 6 Done Done Sent false
9286 3704 Acked Write 10 Done Done Sent false
9287 3705 Acked Flush 0 Done Done Sent false
9288 3706 Acked Read 3 Done Done Sent false
9289 3707 Acked Read 7 Done Done Sent false
9290 3708 Acked Read 9 Done Done Sent false
9291 3709 Acked Write 4 Done Done Sent false
9292 3710 Acked Read 10 Done Done Sent false
9293 3711 Acked Read 3 Done Done Sent false
9294 3712 Acked Read 5 Done Done Sent false
9295 3713 Acked Flush 0 Done Done Sent false
9296 3714 Acked Write 2 Done Done Sent false
9297 3715 Acked Read 7 Done Done Sent false
9298 STATES DS:0 DS:1 DS:2 TOTAL
9299 Sent 0 0 301 301
9300 Done 301 301 0 602
9301 Skipped 0 0 0 0
9302 Error 0 0 0 0
9303 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
9304 Downstairs last five completed: 3414 3413 3412 3411 3410
9305 Upstairs last five completed: 3715 3714 3713 3712 3711
9306 CLIENT: Up:0 ds:301 act:2
93072025-02-25T20:48:39.395ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
93082025-02-25T20:48:41.256ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
93092025-02-25T20:48:42.257ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
9310 ----------------------------------------------------------------
9311 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9312 States: Active Active Offline
9313 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9314 3415 Acked Read 9 Done Done Sent false
9315 3416 Acked Write 8 Done Done Sent false
9316 3417 Acked Flush 0 Done Done Sent false
9317 3418 Acked Write 7 Done Done Sent false
9318 3419 Acked Read 3 Done Done Sent false
9319 3420 Acked Read 8 Done Done Sent false
9320 3421 Acked Flush 0 Done Done Sent false
9321 3422 Acked Write 7 Done Done Sent false
9322 3423 Acked Write 2 Done Done Sent false
9323 3424 Acked Read 1 Done Done Sent false
9324 3425 Acked Read 9 Done Done Sent false
9325 3426 Acked Write 3 Done Done Sent false
9326 3427 Acked Read 6 Done Done Sent false
9327 3428 Acked Read 8 Done Done Sent false
9328 3429 Acked Write 3 Done Done Sent false
9329 3430 Acked Write 3 Done Done Sent false
9330 3431 Acked Read 5 Done Done Sent false
9331 3432 Acked Flush 0 Done Done Sent false
9332 3433 Acked Read 5 Done Done Sent false
9333 3434 Acked Read 4 Done Done Sent false
9334 3435 Acked Write 7 Done Done Sent false
9335 3436 Acked Flush 0 Done Done Sent false
9336 3437 Acked Write 8 Done Done Sent false
9337 3438 Acked Read 8 Done Done Sent false
9338 3439 Acked Read 5 Done Done Sent false
9339 3440 Acked Read 5 Done Done Sent false
9340 3441 Acked Write 1 Done Done Sent false
9341 3442 Acked Write 3 Done Done Sent false
9342 3443 Acked Read 7 Done Done Sent false
9343 3444 Acked Read 4 Done Done Sent false
9344 3445 Acked Read 5 Done Done Sent false
9345 3446 Acked Write 6 Done Done Sent false
9346 3447 Acked Read 10 Done Done Sent false
9347 3448 Acked Flush 0 Done Done Sent false
9348 3449 Acked Read 2 Done Done Sent false
9349 3450 Acked Write 2 Done Done Sent false
9350 3451 Acked Read 8 Done Done Sent false
9351 3452 Acked Write 8 Done Done Sent false
9352 3453 Acked Read 7 Done Done Sent false
9353 3454 Acked Write 8 Done Done Sent false
9354 3455 Acked Read 1 Done Done Sent false
9355 3456 Acked Flush 0 Done Done Sent false
9356 3457 Acked Read 10 Done Done Sent false
9357 3458 Acked Write 1 Done Done Sent false
9358 3459 Acked Read 10 Done Done Sent false
9359 3460 Acked Write 9 Done Done Sent false
9360 3461 Acked Write 8 Done Done Sent false
9361 3462 Acked Write 4 Done Done Sent false
9362 3463 Acked Read 9 Done Done Sent false
9363 3464 Acked Read 2 Done Done Sent false
9364 3465 Acked Read 6 Done Done Sent false
9365 3466 Acked Write 10 Done Done Sent false
9366 3467 Acked Write 4 Done Done Sent false
9367 3468 Acked Write 1 Done Done Sent false
9368 3469 Acked Write 8 Done Done Sent false
9369 3470 Acked Flush 0 Done Done Sent false
9370 3471 Acked Read 4 Done Done Sent false
9371 3472 Acked Write 5 Done Done Sent false
9372 3473 Acked Read 2 Done Done Sent false
9373 3474 Acked Write 4 Done Done Sent false
9374 3475 Acked Read 9 Done Done Sent false
9375 3476 Acked Read 7 Done Done Sent false
9376 3477 Acked Write 1 Done Done Sent false
9377 3478 Acked Write 4 Done Done Sent false
9378 3479 Acked Read 10 Done Done Sent false
9379 3480 Acked Read 7 Done Done Sent false
9380 3481 Acked Write 4 Done Done Sent false
9381 3482 Acked Read 7 Done Done Sent false
9382 3483 Acked Write 1 Done Done Sent false
9383 3484 Acked Write 7 Done Done Sent false
9384 3485 Acked Read 8 Done Done Sent false
9385 3486 Acked Write 5 Done Done Sent false
9386 3487 Acked Write 9 Done Done Sent false
9387 3488 Acked Write 8 Done Done Sent false
9388 3489 Acked Write 2 Done Done Sent false
9389 3490 Acked Read 6 Done Done Sent false
9390 3491 Acked Read 6 Done Done Sent false
9391 3492 Acked Write 9 Done Done Sent false
9392 3493 Acked Read 3 Done Done Sent false
9393 3494 Acked Write 3 Done Done Sent false
9394 3495 Acked Read 1 Done Done Sent false
9395 3496 Acked Flush 0 Done Done Sent false
9396 3497 Acked Read 8 Done Done Sent false
9397 3498 Acked Read 4 Done Done Sent false
9398 3499 Acked Read 7 Done Done Sent false
9399 3500 Acked Write 10 Done Done Sent false
9400 3501 Acked Read 9 Done Done Sent false
9401 3502 Acked Read 8 Done Done Sent false
9402 3503 Acked Read 10 Done Done Sent false
9403 3504 Acked Write 3 Done Done Sent false
9404 3505 Acked Write 10 Done Done Sent false
9405 3506 Acked Read 8 Done Done Sent false
9406 3507 Acked Read 7 Done Done Sent false
9407 3508 Acked Read 6 Done Done Sent false
9408 3509 Acked Read 10 Done Done Sent false
9409 3510 Acked Write 3 Done Done Sent false
9410 3511 Acked Write 7 Done Done Sent false
9411 3512 Acked Read 7 Done Done Sent false
9412 3513 Acked Read 6 Done Done Sent false
9413 3514 Acked Read 5 Done Done Sent false
9414 3515 Acked Read 10 Done Done Sent false
9415 3516 Acked Read 10 Done Done Sent false
9416 3517 Acked Read 6 Done Done Sent false
9417 3518 Acked Read 4 Done Done Sent false
9418 3519 Acked Read 6 Done Done Sent false
9419 3520 Acked Read 1 Done Done Sent false
9420 3521 Acked Read 5 Done Done Sent false
9421 3522 Acked Write 8 Done Done Sent false
9422 3523 Acked Write 3 Done Done Sent false
9423 3524 Acked Flush 0 Done Done Sent false
9424 3525 Acked Write 8 Done Done Sent false
9425 3526 Acked Write 4 Done Done Sent false
9426 3527 Acked Read 10 Done Done Sent false
9427 3528 Acked Read 2 Done Done Sent false
9428 3529 Acked Read 1 Done Done Sent false
9429 3530 Acked Write 7 Done Done Sent false
9430 3531 Acked Write 3 Done Done Sent false
9431 3532 Acked Flush 0 Done Done Sent false
9432 3533 Acked Read 5 Done Done Sent false
9433 3534 Acked Write 1 Done Done Sent false
9434 3535 Acked Read 10 Done Done Sent false
9435 3536 Acked Read 5 Done Done Sent false
9436 3537 Acked Write 6 Done Done Sent false
9437 3538 Acked Write 6 Done Done Sent false
9438 3539 Acked Read 8 Done Done Sent false
9439 3540 Acked Write 4 Done Done Sent false
9440 3541 Acked Read 9 Done Done Sent false
9441 3542 Acked Read 8 Done Done Sent false
9442 3543 Acked Write 9 Done Done Sent false
9443 3544 Acked Write 5 Done Done Sent false
9444 3545 Acked Flush 0 Done Done Sent false
9445 3546 Acked Write 6 Done Done Sent false
9446 3547 Acked Read 6 Done Done Sent false
9447 3548 Acked Read 8 Done Done Sent false
9448 3549 Acked Read 5 Done Done Sent false
9449 3550 Acked Write 7 Done Done Sent false
9450 3551 Acked Read 8 Done Done Sent false
9451 3552 Acked Write 6 Done Done Sent false
9452 3553 Acked Write 7 Done Done Sent false
9453 3554 Acked Read 1 Done Done Sent false
9454 3555 Acked Read 1 Done Done Sent false
9455 3556 Acked Read 3 Done Done Sent false
9456 3557 Acked Write 10 Done Done Sent false
9457 3558 Acked Write 7 Done Done Sent false
9458 3559 Acked Read 4 Done Done Sent false
9459 3560 Acked Write 5 Done Done Sent false
9460 3561 Acked Write 2 Done Done Sent false
9461 3562 Acked Read 2 Done Done Sent false
9462 3563 Acked Write 10 Done Done Sent false
9463 3564 Acked Flush 0 Done Done Sent false
9464 3565 Acked Flush 0 Done Done Sent false
9465 3566 Acked Read 8 Done Done Sent false
9466 3567 Acked Write 2 Done Done Sent false
9467 3568 Acked Write 3 Done Done Sent false
9468 3569 Acked Read 1 Done Done Sent false
9469 3570 Acked Write 5 Done Done Sent false
9470 3571 Acked Read 6 Done Done Sent false
9471 3572 Acked Flush 0 Done Done Sent false
9472 3573 Acked Read 3 Done Done Sent false
9473 3574 Acked Read 8 Done Done Sent false
9474 3575 Acked Write 2 Done Done Sent false
9475 3576 Acked Read 2 Done Done Sent false
9476 3577 Acked Read 3 Done Done Sent false
9477 3578 Acked Write 5 Done Done Sent false
9478 3579 Acked Read 2 Done Done Sent false
9479 3580 Acked Write 6 Done Done Sent false
9480 3581 Acked Read 8 Done Done Sent false
9481 3582 Acked Write 6 Done Done Sent false
9482 3583 Acked Read 2 Done Done Sent false
9483 3584 Acked Read 8 Done Done Sent false
9484 3585 Acked Write 10 Done Done Sent false
9485 3586 Acked Read 3 Done Done Sent false
9486 3587 Acked Flush 0 Done Done Sent false
9487 3588 Acked Read 9 Done Done Sent false
9488 3589 Acked Read 7 Done Done Sent false
9489 3590 Acked Read 5 Done Done Sent false
9490 3591 Acked Write 3 Done Done Sent false
9491 3592 Acked Read 5 Done Done Sent false
9492 3593 Acked Write 10 Done Done Sent false
9493 3594 Acked Read 6 Done Done Sent false
9494 3595 Acked Read 7 Done Done Sent false
9495 3596 Acked Read 1 Done Done Sent false
9496 3597 Acked Read 7 Done Done Sent false
9497 3598 Acked Read 1 Done Done Sent false
9498 3599 Acked Read 9 Done Done Sent false
9499 3600 Acked Write 3 Done Done Sent false
9500 3601 Acked Write 2 Done Done Sent false
9501 3602 Acked Flush 0 Done Done Sent false
9502 3603 Acked Read 3 Done Done Sent false
9503 3604 Acked Flush 0 Done Done Sent false
9504 3605 Acked Write 2 Done Done Sent false
9505 3606 Acked Read 2 Done Done Sent false
9506 3607 Acked Read 6 Done Done Sent false
9507 3608 Acked Write 5 Done Done Sent false
9508 3609 Acked Write 5 Done Done Sent false
9509 3610 Acked Read 2 Done Done Sent false
9510 3611 Acked Read 2 Done Done Sent false
9511 3612 Acked Read 4 Done Done Sent false
9512 3613 Acked Write 7 Done Done Sent false
9513 3614 Acked Write 3 Done Done Sent false
9514 3615 Acked Read 2 Done Done Sent false
9515 3616 Acked Read 2 Done Done Sent false
9516 3617 Acked Read 8 Done Done Sent false
9517 3618 Acked Read 5 Done Done Sent false
9518 3619 Acked Read 6 Done Done Sent false
9519 3620 Acked Read 1 Done Done Sent false
9520 3621 Acked Flush 0 Done Done Sent false
9521 3622 Acked Read 7 Done Done Sent false
9522 3623 Acked Read 2 Done Done Sent false
9523 3624 Acked Flush 0 Done Done Sent false
9524 3625 Acked Write 8 Done Done Sent false
9525 3626 Acked Write 3 Done Done Sent false
9526 3627 Acked Read 9 Done Done Sent false
9527 3628 Acked Write 9 Done Done Sent false
9528 3629 Acked Write 2 Done Done Sent false
9529 3630 Acked Read 5 Done Done Sent false
9530 3631 Acked Read 3 Done Done Sent false
9531 3632 Acked Read 3 Done Done Sent false
9532 3633 Acked Write 4 Done Done Sent false
9533 3634 Acked Read 1 Done Done Sent false
9534 3635 Acked Flush 0 Done Done Sent false
9535 3636 Acked Write 5 Done Done Sent false
9536 3637 Acked Read 6 Done Done Sent false
9537 3638 Acked Flush 0 Done Done Sent false
9538 3639 Acked Read 5 Done Done Sent false
9539 3640 Acked Write 8 Done Done Sent false
9540 3641 Acked Read 2 Done Done Sent false
9541 3642 Acked Read 6 Done Done Sent false
9542 3643 Acked Read 4 Done Done Sent false
9543 3644 Acked Read 4 Done Done Sent false
9544 3645 Acked Read 7 Done Done Sent false
9545 3646 Acked Write 2 Done Done Sent false
9546 3647 Acked Write 10 Done Done Sent false
9547 3648 Acked Write 7 Done Done Sent false
9548 3649 Acked Read 3 Done Done Sent false
9549 3650 Acked Read 8 Done Done Sent false
9550 3651 Acked Flush 0 Done Done Sent false
9551 3652 Acked Write 1 Done Done Sent false
9552 3653 Acked Write 10 Done Done Sent false
9553 3654 Acked Write 5 Done Done Sent false
9554 3655 Acked Read 8 Done Done Sent false
9555 3656 Acked Read 7 Done Done Sent false
9556 3657 Acked Read 1 Done Done Sent false
9557 3658 Acked Write 3 Done Done Sent false
9558 3659 Acked Write 7 Done Done Sent false
9559 3660 Acked Write 6 Done Done Sent false
9560 3661 Acked Write 1 Done Done Sent false
9561 3662 Acked Write 1 Done Done Sent false
9562 3663 Acked Write 9 Done Done Sent false
9563 3664 Acked Flush 0 Done Done Sent false
9564 3665 Acked Read 5 Done Done Sent false
9565 3666 Acked Write 3 Done Done Sent false
9566 3667 Acked Write 9 Done Done Sent false
9567 3668 Acked Write 2 Done Done Sent false
9568 3669 Acked Write 3 Done Done Sent false
9569 3670 Acked Flush 0 Done Done Sent false
9570 3671 Acked Write 7 Done Done Sent false
9571 3672 Acked Read 10 Done Done Sent false
9572 3673 Acked Write 10 Done Done Sent false
9573 3674 Acked Read 10 Done Done Sent false
9574 3675 Acked Read 3 Done Done Sent false
9575 3676 Acked Write 2 Done Done Sent false
9576 3677 Acked Read 6 Done Done Sent false
9577 3678 Acked Write 7 Done Done Sent false
9578 3679 Acked Read 5 Done Done Sent false
9579 3680 Acked Flush 0 Done Done Sent false
9580 3681 Acked Write 6 Done Done Sent false
9581 3682 Acked Read 10 Done Done Sent false
9582 3683 Acked Read 10 Done Done Sent false
9583 3684 Acked Read 9 Done Done Sent false
9584 3685 Acked Read 9 Done Done Sent false
9585 3686 Acked Write 5 Done Done Sent false
9586 3687 Acked Flush 0 Done Done Sent false
9587 3688 Acked Read 3 Done Done Sent false
9588 3689 Acked Read 3 Done Done Sent false
9589 3690 Acked Write 3 Done Done Sent false
9590 3691 Acked Read 9 Done Done Sent false
9591 3692 Acked Write 9 Done Done Sent false
9592 3693 Acked Write 8 Done Done Sent false
9593 3694 Acked Write 3 Done Done Sent false
9594 3695 Acked Read 4 Done Done Sent false
9595 3696 Acked Write 3 Done Done Sent false
9596 3697 Acked Read 5 Done Done Sent false
9597 3698 Acked Write 8 Done Done Sent false
9598 3699 Acked Write 6 Done Done Sent false
9599 3700 Acked Read 4 Done Done Sent false
9600 3701 Acked Read 10 Done Done Sent false
9601 3702 Acked Read 5 Done Done Sent false
9602 3703 Acked Write 6 Done Done Sent false
9603 3704 Acked Write 10 Done Done Sent false
9604 3705 Acked Flush 0 Done Done Sent false
9605 3706 Acked Read 3 Done Done Sent false
9606 3707 Acked Read 7 Done Done Sent false
9607 3708 Acked Read 9 Done Done Sent false
9608 3709 Acked Write 4 Done Done Sent false
9609 3710 Acked Read 10 Done Done Sent false
9610 3711 Acked Read 3 Done Done Sent false
9611 3712 Acked Read 5 Done Done Sent false
9612 3713 Acked Flush 0 Done Done Sent false
9613 3714 Acked Write 2 Done Done Sent false
9614 3715 Acked Read 7 Done Done Sent false
9615 STATES DS:0 DS:1 DS:2 TOTAL
9616 Sent 0 0 301 301
9617 Done 301 301 0 602
9618 Skipped 0 0 0 0
9619 Error 0 0 0 0
9620 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
9621 Downstairs last five completed: 3414 3413 3412 3411 3410
9622 Upstairs last five completed: 3715 3714 3713 3712 3711
9623 CLIENT: Up:0 ds:301 act:2
96242025-02-25T20:48:43.395ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96252025-02-25T20:48:44.260ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96262025-02-25T20:48:44.637ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96272025-02-25T20:48:44.637ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96282025-02-25T20:48:44.637ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96292025-02-25T20:48:44.637ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96302025-02-25T20:48:44.637ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
96312025-02-25T20:48:46.263ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
9632 ----------------------------------------------------------------
9633 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9634 States: Active Active Offline
9635 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9636 3415 Acked Read 9 Done Done Sent false
9637 3416 Acked Write 8 Done Done Sent false
9638 3417 Acked Flush 0 Done Done Sent false
9639 3418 Acked Write 7 Done Done Sent false
9640 3419 Acked Read 3 Done Done Sent false
9641 3420 Acked Read 8 Done Done Sent false
9642 3421 Acked Flush 0 Done Done Sent false
9643 3422 Acked Write 7 Done Done Sent false
9644 3423 Acked Write 2 Done Done Sent false
9645 3424 Acked Read 1 Done Done Sent false
9646 3425 Acked Read 9 Done Done Sent false
9647 3426 Acked Write 3 Done Done Sent false
9648 3427 Acked Read 6 Done Done Sent false
9649 3428 Acked Read 8 Done Done Sent false
9650 3429 Acked Write 3 Done Done Sent false
9651 3430 Acked Write 3 Done Done Sent false
9652 3431 Acked Read 5 Done Done Sent false
9653 3432 Acked Flush 0 Done Done Sent false
9654 3433 Acked Read 5 Done Done Sent false
9655 3434 Acked Read 4 Done Done Sent false
9656 3435 Acked Write 7 Done Done Sent false
9657 3436 Acked Flush 0 Done Done Sent false
9658 3437 Acked Write 8 Done Done Sent false
9659 3438 Acked Read 8 Done Done Sent false
9660 3439 Acked Read 5 Done Done Sent false
9661 3440 Acked Read 5 Done Done Sent false
9662 3441 Acked Write 1 Done Done Sent false
9663 3442 Acked Write 3 Done Done Sent false
9664 3443 Acked Read 7 Done Done Sent false
9665 3444 Acked Read 4 Done Done Sent false
9666 3445 Acked Read 5 Done Done Sent false
9667 3446 Acked Write 6 Done Done Sent false
9668 3447 Acked Read 10 Done Done Sent false
9669 3448 Acked Flush 0 Done Done Sent false
9670 3449 Acked Read 2 Done Done Sent false
9671 3450 Acked Write 2 Done Done Sent false
9672 3451 Acked Read 8 Done Done Sent false
9673 3452 Acked Write 8 Done Done Sent false
9674 3453 Acked Read 7 Done Done Sent false
9675 3454 Acked Write 8 Done Done Sent false
9676 3455 Acked Read 1 Done Done Sent false
9677 3456 Acked Flush 0 Done Done Sent false
9678 3457 Acked Read 10 Done Done Sent false
9679 3458 Acked Write 1 Done Done Sent false
9680 3459 Acked Read 10 Done Done Sent false
9681 3460 Acked Write 9 Done Done Sent false
9682 3461 Acked Write 8 Done Done Sent false
9683 3462 Acked Write 4 Done Done Sent false
9684 3463 Acked Read 9 Done Done Sent false
9685 3464 Acked Read 2 Done Done Sent false
9686 3465 Acked Read 6 Done Done Sent false
9687 3466 Acked Write 10 Done Done Sent false
9688 3467 Acked Write 4 Done Done Sent false
9689 3468 Acked Write 1 Done Done Sent false
9690 3469 Acked Write 8 Done Done Sent false
9691 3470 Acked Flush 0 Done Done Sent false
9692 3471 Acked Read 4 Done Done Sent false
9693 3472 Acked Write 5 Done Done Sent false
9694 3473 Acked Read 2 Done Done Sent false
9695 3474 Acked Write 4 Done Done Sent false
9696 3475 Acked Read 9 Done Done Sent false
9697 3476 Acked Read 7 Done Done Sent false
9698 3477 Acked Write 1 Done Done Sent false
9699 3478 Acked Write 4 Done Done Sent false
9700 3479 Acked Read 10 Done Done Sent false
9701 3480 Acked Read 7 Done Done Sent false
9702 3481 Acked Write 4 Done Done Sent false
9703 3482 Acked Read 7 Done Done Sent false
9704 3483 Acked Write 1 Done Done Sent false
9705 3484 Acked Write 7 Done Done Sent false
9706 3485 Acked Read 8 Done Done Sent false
9707 3486 Acked Write 5 Done Done Sent false
9708 3487 Acked Write 9 Done Done Sent false
9709 3488 Acked Write 8 Done Done Sent false
9710 3489 Acked Write 2 Done Done Sent false
9711 3490 Acked Read 6 Done Done Sent false
9712 3491 Acked Read 6 Done Done Sent false
9713 3492 Acked Write 9 Done Done Sent false
9714 3493 Acked Read 3 Done Done Sent false
9715 3494 Acked Write 3 Done Done Sent false
9716 3495 Acked Read 1 Done Done Sent false
9717 3496 Acked Flush 0 Done Done Sent false
9718 3497 Acked Read 8 Done Done Sent false
9719 3498 Acked Read 4 Done Done Sent false
9720 3499 Acked Read 7 Done Done Sent false
9721 3500 Acked Write 10 Done Done Sent false
9722 3501 Acked Read 9 Done Done Sent false
9723 3502 Acked Read 8 Done Done Sent false
9724 3503 Acked Read 10 Done Done Sent false
9725 3504 Acked Write 3 Done Done Sent false
9726 3505 Acked Write 10 Done Done Sent false
9727 3506 Acked Read 8 Done Done Sent false
9728 3507 Acked Read 7 Done Done Sent false
9729 3508 Acked Read 6 Done Done Sent false
9730 3509 Acked Read 10 Done Done Sent false
9731 3510 Acked Write 3 Done Done Sent false
9732 3511 Acked Write 7 Done Done Sent false
9733 3512 Acked Read 7 Done Done Sent false
9734 3513 Acked Read 6 Done Done Sent false
9735 3514 Acked Read 5 Done Done Sent false
9736 3515 Acked Read 10 Done Done Sent false
9737 3516 Acked Read 10 Done Done Sent false
9738 3517 Acked Read 6 Done Done Sent false
9739 3518 Acked Read 4 Done Done Sent false
9740 3519 Acked Read 6 Done Done Sent false
9741 3520 Acked Read 1 Done Done Sent false
9742 3521 Acked Read 5 Done Done Sent false
9743 3522 Acked Write 8 Done Done Sent false
9744 3523 Acked Write 3 Done Done Sent false
9745 3524 Acked Flush 0 Done Done Sent false
9746 3525 Acked Write 8 Done Done Sent false
9747 3526 Acked Write 4 Done Done Sent false
9748 3527 Acked Read 10 Done Done Sent false
9749 3528 Acked Read 2 Done Done Sent false
9750 3529 Acked Read 1 Done Done Sent false
9751 3530 Acked Write 7 Done Done Sent false
9752 3531 Acked Write 3 Done Done Sent false
9753 3532 Acked Flush 0 Done Done Sent false
9754 3533 Acked Read 5 Done Done Sent false
9755 3534 Acked Write 1 Done Done Sent false
9756 3535 Acked Read 10 Done Done Sent false
9757 3536 Acked Read 5 Done Done Sent false
9758 3537 Acked Write 6 Done Done Sent false
9759 3538 Acked Write 6 Done Done Sent false
9760 3539 Acked Read 8 Done Done Sent false
9761 3540 Acked Write 4 Done Done Sent false
9762 3541 Acked Read 9 Done Done Sent false
9763 3542 Acked Read 8 Done Done Sent false
9764 3543 Acked Write 9 Done Done Sent false
9765 3544 Acked Write 5 Done Done Sent false
9766 3545 Acked Flush 0 Done Done Sent false
9767 3546 Acked Write 6 Done Done Sent false
9768 3547 Acked Read 6 Done Done Sent false
9769 3548 Acked Read 8 Done Done Sent false
9770 3549 Acked Read 5 Done Done Sent false
9771 3550 Acked Write 7 Done Done Sent false
9772 3551 Acked Read 8 Done Done Sent false
9773 3552 Acked Write 6 Done Done Sent false
9774 3553 Acked Write 7 Done Done Sent false
9775 3554 Acked Read 1 Done Done Sent false
9776 3555 Acked Read 1 Done Done Sent false
9777 3556 Acked Read 3 Done Done Sent false
9778 3557 Acked Write 10 Done Done Sent false
9779 3558 Acked Write 7 Done Done Sent false
9780 3559 Acked Read 4 Done Done Sent false
9781 3560 Acked Write 5 Done Done Sent false
9782 3561 Acked Write 2 Done Done Sent false
9783 3562 Acked Read 2 Done Done Sent false
9784 3563 Acked Write 10 Done Done Sent false
9785 3564 Acked Flush 0 Done Done Sent false
9786 3565 Acked Flush 0 Done Done Sent false
9787 3566 Acked Read 8 Done Done Sent false
9788 3567 Acked Write 2 Done Done Sent false
9789 3568 Acked Write 3 Done Done Sent false
9790 3569 Acked Read 1 Done Done Sent false
9791 3570 Acked Write 5 Done Done Sent false
9792 3571 Acked Read 6 Done Done Sent false
9793 3572 Acked Flush 0 Done Done Sent false
9794 3573 Acked Read 3 Done Done Sent false
9795 3574 Acked Read 8 Done Done Sent false
9796 3575 Acked Write 2 Done Done Sent false
9797 3576 Acked Read 2 Done Done Sent false
9798 3577 Acked Read 3 Done Done Sent false
9799 3578 Acked Write 5 Done Done Sent false
9800 3579 Acked Read 2 Done Done Sent false
9801 3580 Acked Write 6 Done Done Sent false
9802 3581 Acked Read 8 Done Done Sent false
9803 3582 Acked Write 6 Done Done Sent false
9804 3583 Acked Read 2 Done Done Sent false
9805 3584 Acked Read 8 Done Done Sent false
9806 3585 Acked Write 10 Done Done Sent false
9807 3586 Acked Read 3 Done Done Sent false
9808 3587 Acked Flush 0 Done Done Sent false
9809 3588 Acked Read 9 Done Done Sent false
9810 3589 Acked Read 7 Done Done Sent false
9811 3590 Acked Read 5 Done Done Sent false
9812 3591 Acked Write 3 Done Done Sent false
9813 3592 Acked Read 5 Done Done Sent false
9814 3593 Acked Write 10 Done Done Sent false
9815 3594 Acked Read 6 Done Done Sent false
9816 3595 Acked Read 7 Done Done Sent false
9817 3596 Acked Read 1 Done Done Sent false
9818 3597 Acked Read 7 Done Done Sent false
9819 3598 Acked Read 1 Done Done Sent false
9820 3599 Acked Read 9 Done Done Sent false
9821 3600 Acked Write 3 Done Done Sent false
9822 3601 Acked Write 2 Done Done Sent false
9823 3602 Acked Flush 0 Done Done Sent false
9824 3603 Acked Read 3 Done Done Sent false
9825 3604 Acked Flush 0 Done Done Sent false
9826 3605 Acked Write 2 Done Done Sent false
9827 3606 Acked Read 2 Done Done Sent false
9828 3607 Acked Read 6 Done Done Sent false
9829 3608 Acked Write 5 Done Done Sent false
9830 3609 Acked Write 5 Done Done Sent false
9831 3610 Acked Read 2 Done Done Sent false
9832 3611 Acked Read 2 Done Done Sent false
9833 3612 Acked Read 4 Done Done Sent false
9834 3613 Acked Write 7 Done Done Sent false
9835 3614 Acked Write 3 Done Done Sent false
9836 3615 Acked Read 2 Done Done Sent false
9837 3616 Acked Read 2 Done Done Sent false
9838 3617 Acked Read 8 Done Done Sent false
9839 3618 Acked Read 5 Done Done Sent false
9840 3619 Acked Read 6 Done Done Sent false
9841 3620 Acked Read 1 Done Done Sent false
9842 3621 Acked Flush 0 Done Done Sent false
9843 3622 Acked Read 7 Done Done Sent false
9844 3623 Acked Read 2 Done Done Sent false
9845 3624 Acked Flush 0 Done Done Sent false
9846 3625 Acked Write 8 Done Done Sent false
9847 3626 Acked Write 3 Done Done Sent false
9848 3627 Acked Read 9 Done Done Sent false
9849 3628 Acked Write 9 Done Done Sent false
9850 3629 Acked Write 2 Done Done Sent false
9851 3630 Acked Read 5 Done Done Sent false
9852 3631 Acked Read 3 Done Done Sent false
9853 3632 Acked Read 3 Done Done Sent false
9854 3633 Acked Write 4 Done Done Sent false
9855 3634 Acked Read 1 Done Done Sent false
9856 3635 Acked Flush 0 Done Done Sent false
9857 3636 Acked Write 5 Done Done Sent false
9858 3637 Acked Read 6 Done Done Sent false
9859 3638 Acked Flush 0 Done Done Sent false
9860 3639 Acked Read 5 Done Done Sent false
9861 3640 Acked Write 8 Done Done Sent false
9862 3641 Acked Read 2 Done Done Sent false
9863 3642 Acked Read 6 Done Done Sent false
9864 3643 Acked Read 4 Done Done Sent false
9865 3644 Acked Read 4 Done Done Sent false
9866 3645 Acked Read 7 Done Done Sent false
9867 3646 Acked Write 2 Done Done Sent false
9868 3647 Acked Write 10 Done Done Sent false
9869 3648 Acked Write 7 Done Done Sent false
9870 3649 Acked Read 3 Done Done Sent false
9871 3650 Acked Read 8 Done Done Sent false
9872 3651 Acked Flush 0 Done Done Sent false
9873 3652 Acked Write 1 Done Done Sent false
9874 3653 Acked Write 10 Done Done Sent false
9875 3654 Acked Write 5 Done Done Sent false
9876 3655 Acked Read 8 Done Done Sent false
9877 3656 Acked Read 7 Done Done Sent false
9878 3657 Acked Read 1 Done Done Sent false
9879 3658 Acked Write 3 Done Done Sent false
9880 3659 Acked Write 7 Done Done Sent false
9881 3660 Acked Write 6 Done Done Sent false
9882 3661 Acked Write 1 Done Done Sent false
9883 3662 Acked Write 1 Done Done Sent false
9884 3663 Acked Write 9 Done Done Sent false
9885 3664 Acked Flush 0 Done Done Sent false
9886 3665 Acked Read 5 Done Done Sent false
9887 3666 Acked Write 3 Done Done Sent false
9888 3667 Acked Write 9 Done Done Sent false
9889 3668 Acked Write 2 Done Done Sent false
9890 3669 Acked Write 3 Done Done Sent false
9891 3670 Acked Flush 0 Done Done Sent false
9892 3671 Acked Write 7 Done Done Sent false
9893 3672 Acked Read 10 Done Done Sent false
9894 3673 Acked Write 10 Done Done Sent false
9895 3674 Acked Read 10 Done Done Sent false
9896 3675 Acked Read 3 Done Done Sent false
9897 3676 Acked Write 2 Done Done Sent false
9898 3677 Acked Read 6 Done Done Sent false
9899 3678 Acked Write 7 Done Done Sent false
9900 3679 Acked Read 5 Done Done Sent false
9901 3680 Acked Flush 0 Done Done Sent false
9902 3681 Acked Write 6 Done Done Sent false
9903 3682 Acked Read 10 Done Done Sent false
9904 3683 Acked Read 10 Done Done Sent false
9905 3684 Acked Read 9 Done Done Sent false
9906 3685 Acked Read 9 Done Done Sent false
9907 3686 Acked Write 5 Done Done Sent false
9908 3687 Acked Flush 0 Done Done Sent false
9909 3688 Acked Read 3 Done Done Sent false
9910 3689 Acked Read 3 Done Done Sent false
9911 3690 Acked Write 3 Done Done Sent false
9912 3691 Acked Read 9 Done Done Sent false
9913 3692 Acked Write 9 Done Done Sent false
9914 3693 Acked Write 8 Done Done Sent false
9915 3694 Acked Write 3 Done Done Sent false
9916 3695 Acked Read 4 Done Done Sent false
9917 3696 Acked Write 3 Done Done Sent false
9918 3697 Acked Read 5 Done Done Sent false
9919 3698 Acked Write 8 Done Done Sent false
9920 3699 Acked Write 6 Done Done Sent false
9921 3700 Acked Read 4 Done Done Sent false
9922 3701 Acked Read 10 Done Done Sent false
9923 3702 Acked Read 5 Done Done Sent false
9924 3703 Acked Write 6 Done Done Sent false
9925 3704 Acked Write 10 Done Done Sent false
9926 3705 Acked Flush 0 Done Done Sent false
9927 3706 Acked Read 3 Done Done Sent false
9928 3707 Acked Read 7 Done Done Sent false
9929 3708 Acked Read 9 Done Done Sent false
9930 3709 Acked Write 4 Done Done Sent false
9931 3710 Acked Read 10 Done Done Sent false
9932 3711 Acked Read 3 Done Done Sent false
9933 3712 Acked Read 5 Done Done Sent false
9934 3713 Acked Flush 0 Done Done Sent false
9935 3714 Acked Write 2 Done Done Sent false
9936 3715 Acked Read 7 Done Done Sent false
9937 STATES DS:0 DS:1 DS:2 TOTAL
9938 Sent 0 0 301 301
9939 Done 301 301 0 602
9940 Skipped 0 0 0 0
9941 Error 0 0 0 0
9942 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
9943 Downstairs last five completed: 3414 3413 3412 3411 3410
9944 Upstairs last five completed: 3715 3714 3713 3712 3711
9945 CLIENT: Up:0 ds:301 act:2
99462025-02-25T20:48:47.398ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
99472025-02-25T20:48:48.266ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
99482025-02-25T20:48:49.266ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
9949 ----------------------------------------------------------------
9950 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9951 States: Active Active Offline
9952 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9953 3415 Acked Read 9 Done Done Sent false
9954 3416 Acked Write 8 Done Done Sent false
9955 3417 Acked Flush 0 Done Done Sent false
9956 3418 Acked Write 7 Done Done Sent false
9957 3419 Acked Read 3 Done Done Sent false
9958 3420 Acked Read 8 Done Done Sent false
9959 3421 Acked Flush 0 Done Done Sent false
9960 3422 Acked Write 7 Done Done Sent false
9961 3423 Acked Write 2 Done Done Sent false
9962 3424 Acked Read 1 Done Done Sent false
9963 3425 Acked Read 9 Done Done Sent false
9964 3426 Acked Write 3 Done Done Sent false
9965 3427 Acked Read 6 Done Done Sent false
9966 3428 Acked Read 8 Done Done Sent false
9967 3429 Acked Write 3 Done Done Sent false
9968 3430 Acked Write 3 Done Done Sent false
9969 3431 Acked Read 5 Done Done Sent false
9970 3432 Acked Flush 0 Done Done Sent false
9971 3433 Acked Read 5 Done Done Sent false
9972 3434 Acked Read 4 Done Done Sent false
9973 3435 Acked Write 7 Done Done Sent false
9974 3436 Acked Flush 0 Done Done Sent false
9975 3437 Acked Write 8 Done Done Sent false
9976 3438 Acked Read 8 Done Done Sent false
9977 3439 Acked Read 5 Done Done Sent false
9978 3440 Acked Read 5 Done Done Sent false
9979 3441 Acked Write 1 Done Done Sent false
9980 3442 Acked Write 3 Done Done Sent false
9981 3443 Acked Read 7 Done Done Sent false
9982 3444 Acked Read 4 Done Done Sent false
9983 3445 Acked Read 5 Done Done Sent false
9984 3446 Acked Write 6 Done Done Sent false
9985 3447 Acked Read 10 Done Done Sent false
9986 3448 Acked Flush 0 Done Done Sent false
9987 3449 Acked Read 2 Done Done Sent false
9988 3450 Acked Write 2 Done Done Sent false
9989 3451 Acked Read 8 Done Done Sent false
9990 3452 Acked Write 8 Done Done Sent false
9991 3453 Acked Read 7 Done Done Sent false
9992 3454 Acked Write 8 Done Done Sent false
9993 3455 Acked Read 1 Done Done Sent false
9994 3456 Acked Flush 0 Done Done Sent false
9995 3457 Acked Read 10 Done Done Sent false
9996 3458 Acked Write 1 Done Done Sent false
9997 3459 Acked Read 10 Done Done Sent false
9998 3460 Acked Write 9 Done Done Sent false
9999 3461 Acked Write 8 Done Done Sent false
10000 3462 Acked Write 4 Done Done Sent false
10001 3463 Acked Read 9 Done Done Sent false
10002 3464 Acked Read 2 Done Done Sent false
10003 3465 Acked Read 6 Done Done Sent false
10004 3466 Acked Write 10 Done Done Sent false
10005 3467 Acked Write 4 Done Done Sent false
10006 3468 Acked Write 1 Done Done Sent false
10007 3469 Acked Write 8 Done Done Sent false
10008 3470 Acked Flush 0 Done Done Sent false
10009 3471 Acked Read 4 Done Done Sent false
10010 3472 Acked Write 5 Done Done Sent false
10011 3473 Acked Read 2 Done Done Sent false
10012 3474 Acked Write 4 Done Done Sent false
10013 3475 Acked Read 9 Done Done Sent false
10014 3476 Acked Read 7 Done Done Sent false
10015 3477 Acked Write 1 Done Done Sent false
10016 3478 Acked Write 4 Done Done Sent false
10017 3479 Acked Read 10 Done Done Sent false
10018 3480 Acked Read 7 Done Done Sent false
10019 3481 Acked Write 4 Done Done Sent false
10020 3482 Acked Read 7 Done Done Sent false
10021 3483 Acked Write 1 Done Done Sent false
10022 3484 Acked Write 7 Done Done Sent false
10023 3485 Acked Read 8 Done Done Sent false
10024 3486 Acked Write 5 Done Done Sent false
10025 3487 Acked Write 9 Done Done Sent false
10026 3488 Acked Write 8 Done Done Sent false
10027 3489 Acked Write 2 Done Done Sent false
10028 3490 Acked Read 6 Done Done Sent false
10029 3491 Acked Read 6 Done Done Sent false
10030 3492 Acked Write 9 Done Done Sent false
10031 3493 Acked Read 3 Done Done Sent false
10032 3494 Acked Write 3 Done Done Sent false
10033 3495 Acked Read 1 Done Done Sent false
10034 3496 Acked Flush 0 Done Done Sent false
10035 3497 Acked Read 8 Done Done Sent false
10036 3498 Acked Read 4 Done Done Sent false
10037 3499 Acked Read 7 Done Done Sent false
10038 3500 Acked Write 10 Done Done Sent false
10039 3501 Acked Read 9 Done Done Sent false
10040 3502 Acked Read 8 Done Done Sent false
10041 3503 Acked Read 10 Done Done Sent false
10042 3504 Acked Write 3 Done Done Sent false
10043 3505 Acked Write 10 Done Done Sent false
10044 3506 Acked Read 8 Done Done Sent false
10045 3507 Acked Read 7 Done Done Sent false
10046 3508 Acked Read 6 Done Done Sent false
10047 3509 Acked Read 10 Done Done Sent false
10048 3510 Acked Write 3 Done Done Sent false
10049 3511 Acked Write 7 Done Done Sent false
10050 3512 Acked Read 7 Done Done Sent false
10051 3513 Acked Read 6 Done Done Sent false
10052 3514 Acked Read 5 Done Done Sent false
10053 3515 Acked Read 10 Done Done Sent false
10054 3516 Acked Read 10 Done Done Sent false
10055 3517 Acked Read 6 Done Done Sent false
10056 3518 Acked Read 4 Done Done Sent false
10057 3519 Acked Read 6 Done Done Sent false
10058 3520 Acked Read 1 Done Done Sent false
10059 3521 Acked Read 5 Done Done Sent false
10060 3522 Acked Write 8 Done Done Sent false
10061 3523 Acked Write 3 Done Done Sent false
10062 3524 Acked Flush 0 Done Done Sent false
10063 3525 Acked Write 8 Done Done Sent false
10064 3526 Acked Write 4 Done Done Sent false
10065 3527 Acked Read 10 Done Done Sent false
10066 3528 Acked Read 2 Done Done Sent false
10067 3529 Acked Read 1 Done Done Sent false
10068 3530 Acked Write 7 Done Done Sent false
10069 3531 Acked Write 3 Done Done Sent false
10070 3532 Acked Flush 0 Done Done Sent false
10071 3533 Acked Read 5 Done Done Sent false
10072 3534 Acked Write 1 Done Done Sent false
10073 3535 Acked Read 10 Done Done Sent false
10074 3536 Acked Read 5 Done Done Sent false
10075 3537 Acked Write 6 Done Done Sent false
10076 3538 Acked Write 6 Done Done Sent false
10077 3539 Acked Read 8 Done Done Sent false
10078 3540 Acked Write 4 Done Done Sent false
10079 3541 Acked Read 9 Done Done Sent false
10080 3542 Acked Read 8 Done Done Sent false
10081 3543 Acked Write 9 Done Done Sent false
10082 3544 Acked Write 5 Done Done Sent false
10083 3545 Acked Flush 0 Done Done Sent false
10084 3546 Acked Write 6 Done Done Sent false
10085 3547 Acked Read 6 Done Done Sent false
10086 3548 Acked Read 8 Done Done Sent false
10087 3549 Acked Read 5 Done Done Sent false
10088 3550 Acked Write 7 Done Done Sent false
10089 3551 Acked Read 8 Done Done Sent false
10090 3552 Acked Write 6 Done Done Sent false
10091 3553 Acked Write 7 Done Done Sent false
10092 3554 Acked Read 1 Done Done Sent false
10093 3555 Acked Read 1 Done Done Sent false
10094 3556 Acked Read 3 Done Done Sent false
10095 3557 Acked Write 10 Done Done Sent false
10096 3558 Acked Write 7 Done Done Sent false
10097 3559 Acked Read 4 Done Done Sent false
10098 3560 Acked Write 5 Done Done Sent false
10099 3561 Acked Write 2 Done Done Sent false
10100 3562 Acked Read 2 Done Done Sent false
10101 3563 Acked Write 10 Done Done Sent false
10102 3564 Acked Flush 0 Done Done Sent false
10103 3565 Acked Flush 0 Done Done Sent false
10104 3566 Acked Read 8 Done Done Sent false
10105 3567 Acked Write 2 Done Done Sent false
10106 3568 Acked Write 3 Done Done Sent false
10107 3569 Acked Read 1 Done Done Sent false
10108 3570 Acked Write 5 Done Done Sent false
10109 3571 Acked Read 6 Done Done Sent false
10110 3572 Acked Flush 0 Done Done Sent false
10111 3573 Acked Read 3 Done Done Sent false
10112 3574 Acked Read 8 Done Done Sent false
10113 3575 Acked Write 2 Done Done Sent false
10114 3576 Acked Read 2 Done Done Sent false
10115 3577 Acked Read 3 Done Done Sent false
10116 3578 Acked Write 5 Done Done Sent false
10117 3579 Acked Read 2 Done Done Sent false
10118 3580 Acked Write 6 Done Done Sent false
10119 3581 Acked Read 8 Done Done Sent false
10120 3582 Acked Write 6 Done Done Sent false
10121 3583 Acked Read 2 Done Done Sent false
10122 3584 Acked Read 8 Done Done Sent false
10123 3585 Acked Write 10 Done Done Sent false
10124 3586 Acked Read 3 Done Done Sent false
10125 3587 Acked Flush 0 Done Done Sent false
10126 3588 Acked Read 9 Done Done Sent false
10127 3589 Acked Read 7 Done Done Sent false
10128 3590 Acked Read 5 Done Done Sent false
10129 3591 Acked Write 3 Done Done Sent false
10130 3592 Acked Read 5 Done Done Sent false
10131 3593 Acked Write 10 Done Done Sent false
10132 3594 Acked Read 6 Done Done Sent false
10133 3595 Acked Read 7 Done Done Sent false
10134 3596 Acked Read 1 Done Done Sent false
10135 3597 Acked Read 7 Done Done Sent false
10136 3598 Acked Read 1 Done Done Sent false
10137 3599 Acked Read 9 Done Done Sent false
10138 3600 Acked Write 3 Done Done Sent false
10139 3601 Acked Write 2 Done Done Sent false
10140 3602 Acked Flush 0 Done Done Sent false
10141 3603 Acked Read 3 Done Done Sent false
10142 3604 Acked Flush 0 Done Done Sent false
10143 3605 Acked Write 2 Done Done Sent false
10144 3606 Acked Read 2 Done Done Sent false
10145 3607 Acked Read 6 Done Done Sent false
10146 3608 Acked Write 5 Done Done Sent false
10147 3609 Acked Write 5 Done Done Sent false
10148 3610 Acked Read 2 Done Done Sent false
10149 3611 Acked Read 2 Done Done Sent false
10150 3612 Acked Read 4 Done Done Sent false
10151 3613 Acked Write 7 Done Done Sent false
10152 3614 Acked Write 3 Done Done Sent false
10153 3615 Acked Read 2 Done Done Sent false
10154 3616 Acked Read 2 Done Done Sent false
10155 3617 Acked Read 8 Done Done Sent false
10156 3618 Acked Read 5 Done Done Sent false
10157 3619 Acked Read 6 Done Done Sent false
10158 3620 Acked Read 1 Done Done Sent false
10159 3621 Acked Flush 0 Done Done Sent false
10160 3622 Acked Read 7 Done Done Sent false
10161 3623 Acked Read 2 Done Done Sent false
10162 3624 Acked Flush 0 Done Done Sent false
10163 3625 Acked Write 8 Done Done Sent false
10164 3626 Acked Write 3 Done Done Sent false
10165 3627 Acked Read 9 Done Done Sent false
10166 3628 Acked Write 9 Done Done Sent false
10167 3629 Acked Write 2 Done Done Sent false
10168 3630 Acked Read 5 Done Done Sent false
10169 3631 Acked Read 3 Done Done Sent false
10170 3632 Acked Read 3 Done Done Sent false
10171 3633 Acked Write 4 Done Done Sent false
10172 3634 Acked Read 1 Done Done Sent false
10173 3635 Acked Flush 0 Done Done Sent false
10174 3636 Acked Write 5 Done Done Sent false
10175 3637 Acked Read 6 Done Done Sent false
10176 3638 Acked Flush 0 Done Done Sent false
10177 3639 Acked Read 5 Done Done Sent false
10178 3640 Acked Write 8 Done Done Sent false
10179 3641 Acked Read 2 Done Done Sent false
10180 3642 Acked Read 6 Done Done Sent false
10181 3643 Acked Read 4 Done Done Sent false
10182 3644 Acked Read 4 Done Done Sent false
10183 3645 Acked Read 7 Done Done Sent false
10184 3646 Acked Write 2 Done Done Sent false
10185 3647 Acked Write 10 Done Done Sent false
10186 3648 Acked Write 7 Done Done Sent false
10187 3649 Acked Read 3 Done Done Sent false
10188 3650 Acked Read 8 Done Done Sent false
10189 3651 Acked Flush 0 Done Done Sent false
10190 3652 Acked Write 1 Done Done Sent false
10191 3653 Acked Write 10 Done Done Sent false
10192 3654 Acked Write 5 Done Done Sent false
10193 3655 Acked Read 8 Done Done Sent false
10194 3656 Acked Read 7 Done Done Sent false
10195 3657 Acked Read 1 Done Done Sent false
10196 3658 Acked Write 3 Done Done Sent false
10197 3659 Acked Write 7 Done Done Sent false
10198 3660 Acked Write 6 Done Done Sent false
10199 3661 Acked Write 1 Done Done Sent false
10200 3662 Acked Write 1 Done Done Sent false
10201 3663 Acked Write 9 Done Done Sent false
10202 3664 Acked Flush 0 Done Done Sent false
10203 3665 Acked Read 5 Done Done Sent false
10204 3666 Acked Write 3 Done Done Sent false
10205 3667 Acked Write 9 Done Done Sent false
10206 3668 Acked Write 2 Done Done Sent false
10207 3669 Acked Write 3 Done Done Sent false
10208 3670 Acked Flush 0 Done Done Sent false
10209 3671 Acked Write 7 Done Done Sent false
10210 3672 Acked Read 10 Done Done Sent false
10211 3673 Acked Write 10 Done Done Sent false
10212 3674 Acked Read 10 Done Done Sent false
10213 3675 Acked Read 3 Done Done Sent false
10214 3676 Acked Write 2 Done Done Sent false
10215 3677 Acked Read 6 Done Done Sent false
10216 3678 Acked Write 7 Done Done Sent false
10217 3679 Acked Read 5 Done Done Sent false
10218 3680 Acked Flush 0 Done Done Sent false
10219 3681 Acked Write 6 Done Done Sent false
10220 3682 Acked Read 10 Done Done Sent false
10221 3683 Acked Read 10 Done Done Sent false
10222 3684 Acked Read 9 Done Done Sent false
10223 3685 Acked Read 9 Done Done Sent false
10224 3686 Acked Write 5 Done Done Sent false
10225 3687 Acked Flush 0 Done Done Sent false
10226 3688 Acked Read 3 Done Done Sent false
10227 3689 Acked Read 3 Done Done Sent false
10228 3690 Acked Write 3 Done Done Sent false
10229 3691 Acked Read 9 Done Done Sent false
10230 3692 Acked Write 9 Done Done Sent false
10231 3693 Acked Write 8 Done Done Sent false
10232 3694 Acked Write 3 Done Done Sent false
10233 3695 Acked Read 4 Done Done Sent false
10234 3696 Acked Write 3 Done Done Sent false
10235 3697 Acked Read 5 Done Done Sent false
10236 3698 Acked Write 8 Done Done Sent false
10237 3699 Acked Write 6 Done Done Sent false
10238 3700 Acked Read 4 Done Done Sent false
10239 3701 Acked Read 10 Done Done Sent false
10240 3702 Acked Read 5 Done Done Sent false
10241 3703 Acked Write 6 Done Done Sent false
10242 3704 Acked Write 10 Done Done Sent false
10243 3705 Acked Flush 0 Done Done Sent false
10244 3706 Acked Read 3 Done Done Sent false
10245 3707 Acked Read 7 Done Done Sent false
10246 3708 Acked Read 9 Done Done Sent false
10247 3709 Acked Write 4 Done Done Sent false
10248 3710 Acked Read 10 Done Done Sent false
10249 3711 Acked Read 3 Done Done Sent false
10250 3712 Acked Read 5 Done Done Sent false
10251 3713 Acked Flush 0 Done Done Sent false
10252 3714 Acked Write 2 Done Done Sent false
10253 3715 Acked Read 7 Done Done Sent false
10254 STATES DS:0 DS:1 DS:2 TOTAL
10255 Sent 0 0 301 301
10256 Done 301 301 0 602
10257 Skipped 0 0 0 0
10258 Error 0 0 0 0
10259 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
10260 Downstairs last five completed: 3414 3413 3412 3411 3410
10261 Upstairs last five completed: 3715 3714 3713 3712 3711
10262 CLIENT: Up:0 ds:301 act:2
102632025-02-25T20:48:51.401ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102642025-02-25T20:48:52.272ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102652025-02-25T20:48:53.273ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102662025-02-25T20:48:54.274ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102672025-02-25T20:48:54.638ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102682025-02-25T20:48:54.638ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102692025-02-25T20:48:54.638ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102702025-02-25T20:48:54.638ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
102712025-02-25T20:48:54.638ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
10272 ----------------------------------------------------------------
10273 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10274 States: Active Active Offline
10275 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10276 3415 Acked Read 9 Done Done Sent false
10277 3416 Acked Write 8 Done Done Sent false
10278 3417 Acked Flush 0 Done Done Sent false
10279 3418 Acked Write 7 Done Done Sent false
10280 3419 Acked Read 3 Done Done Sent false
10281 3420 Acked Read 8 Done Done Sent false
10282 3421 Acked Flush 0 Done Done Sent false
10283 3422 Acked Write 7 Done Done Sent false
10284 3423 Acked Write 2 Done Done Sent false
10285 3424 Acked Read 1 Done Done Sent false
10286 3425 Acked Read 9 Done Done Sent false
10287 3426 Acked Write 3 Done Done Sent false
10288 3427 Acked Read 6 Done Done Sent false
10289 3428 Acked Read 8 Done Done Sent false
10290 3429 Acked Write 3 Done Done Sent false
10291 3430 Acked Write 3 Done Done Sent false
10292 3431 Acked Read 5 Done Done Sent false
10293 3432 Acked Flush 0 Done Done Sent false
10294 3433 Acked Read 5 Done Done Sent false
10295 3434 Acked Read 4 Done Done Sent false
10296 3435 Acked Write 7 Done Done Sent false
10297 3436 Acked Flush 0 Done Done Sent false
10298 3437 Acked Write 8 Done Done Sent false
10299 3438 Acked Read 8 Done Done Sent false
10300 3439 Acked Read 5 Done Done Sent false
10301 3440 Acked Read 5 Done Done Sent false
10302 3441 Acked Write 1 Done Done Sent false
10303 3442 Acked Write 3 Done Done Sent false
10304 3443 Acked Read 7 Done Done Sent false
10305 3444 Acked Read 4 Done Done Sent false
10306 3445 Acked Read 5 Done Done Sent false
10307 3446 Acked Write 6 Done Done Sent false
10308 3447 Acked Read 10 Done Done Sent false
10309 3448 Acked Flush 0 Done Done Sent false
10310 3449 Acked Read 2 Done Done Sent false
10311 3450 Acked Write 2 Done Done Sent false
10312 3451 Acked Read 8 Done Done Sent false
10313 3452 Acked Write 8 Done Done Sent false
10314 3453 Acked Read 7 Done Done Sent false
10315 3454 Acked Write 8 Done Done Sent false
10316 3455 Acked Read 1 Done Done Sent false
10317 3456 Acked Flush 0 Done Done Sent false
10318 3457 Acked Read 10 Done Done Sent false
10319 3458 Acked Write 1 Done Done Sent false
10320 3459 Acked Read 10 Done Done Sent false
10321 3460 Acked Write 9 Done Done Sent false
10322 3461 Acked Write 8 Done Done Sent false
10323 3462 Acked Write 4 Done Done Sent false
10324 3463 Acked Read 9 Done Done Sent false
10325 3464 Acked Read 2 Done Done Sent false
10326 3465 Acked Read 6 Done Done Sent false
10327 3466 Acked Write 10 Done Done Sent false
10328 3467 Acked Write 4 Done Done Sent false
10329 3468 Acked Write 1 Done Done Sent false
10330 3469 Acked Write 8 Done Done Sent false
10331 3470 Acked Flush 0 Done Done Sent false
10332 3471 Acked Read 4 Done Done Sent false
10333 3472 Acked Write 5 Done Done Sent false
10334 3473 Acked Read 2 Done Done Sent false
10335 3474 Acked Write 4 Done Done Sent false
10336 3475 Acked Read 9 Done Done Sent false
10337 3476 Acked Read 7 Done Done Sent false
10338 3477 Acked Write 1 Done Done Sent false
10339 3478 Acked Write 4 Done Done Sent false
10340 3479 Acked Read 10 Done Done Sent false
10341 3480 Acked Read 7 Done Done Sent false
10342 3481 Acked Write 4 Done Done Sent false
10343 3482 Acked Read 7 Done Done Sent false
10344 3483 Acked Write 1 Done Done Sent false
10345 3484 Acked Write 7 Done Done Sent false
10346 3485 Acked Read 8 Done Done Sent false
10347 3486 Acked Write 5 Done Done Sent false
10348 3487 Acked Write 9 Done Done Sent false
10349 3488 Acked Write 8 Done Done Sent false
10350 3489 Acked Write 2 Done Done Sent false
10351 3490 Acked Read 6 Done Done Sent false
10352 3491 Acked Read 6 Done Done Sent false
10353 3492 Acked Write 9 Done Done Sent false
10354 3493 Acked Read 3 Done Done Sent false
10355 3494 Acked Write 3 Done Done Sent false
10356 3495 Acked Read 1 Done Done Sent false
10357 3496 Acked Flush 0 Done Done Sent false
10358 3497 Acked Read 8 Done Done Sent false
10359 3498 Acked Read 4 Done Done Sent false
10360 3499 Acked Read 7 Done Done Sent false
10361 3500 Acked Write 10 Done Done Sent false
10362 3501 Acked Read 9 Done Done Sent false
10363 3502 Acked Read 8 Done Done Sent false
10364 3503 Acked Read 10 Done Done Sent false
10365 3504 Acked Write 3 Done Done Sent false
10366 3505 Acked Write 10 Done Done Sent false
10367 3506 Acked Read 8 Done Done Sent false
10368 3507 Acked Read 7 Done Done Sent false
10369 3508 Acked Read 6 Done Done Sent false
10370 3509 Acked Read 10 Done Done Sent false
10371 3510 Acked Write 3 Done Done Sent false
10372 3511 Acked Write 7 Done Done Sent false
10373 3512 Acked Read 7 Done Done Sent false
10374 3513 Acked Read 6 Done Done Sent false
10375 3514 Acked Read 5 Done Done Sent false
10376 3515 Acked Read 10 Done Done Sent false
10377 3516 Acked Read 10 Done Done Sent false
10378 3517 Acked Read 6 Done Done Sent false
10379 3518 Acked Read 4 Done Done Sent false
10380 3519 Acked Read 6 Done Done Sent false
10381 3520 Acked Read 1 Done Done Sent false
10382 3521 Acked Read 5 Done Done Sent false
10383 3522 Acked Write 8 Done Done Sent false
10384 3523 Acked Write 3 Done Done Sent false
10385 3524 Acked Flush 0 Done Done Sent false
10386 3525 Acked Write 8 Done Done Sent false
10387 3526 Acked Write 4 Done Done Sent false
10388 3527 Acked Read 10 Done Done Sent false
10389 3528 Acked Read 2 Done Done Sent false
10390 3529 Acked Read 1 Done Done Sent false
10391 3530 Acked Write 7 Done Done Sent false
10392 3531 Acked Write 3 Done Done Sent false
10393 3532 Acked Flush 0 Done Done Sent false
10394 3533 Acked Read 5 Done Done Sent false
10395 3534 Acked Write 1 Done Done Sent false
10396 3535 Acked Read 10 Done Done Sent false
10397 3536 Acked Read 5 Done Done Sent false
10398 3537 Acked Write 6 Done Done Sent false
10399 3538 Acked Write 6 Done Done Sent false
10400 3539 Acked Read 8 Done Done Sent false
10401 3540 Acked Write 4 Done Done Sent false
10402 3541 Acked Read 9 Done Done Sent false
10403 3542 Acked Read 8 Done Done Sent false
10404 3543 Acked Write 9 Done Done Sent false
10405 3544 Acked Write 5 Done Done Sent false
10406 3545 Acked Flush 0 Done Done Sent false
10407 3546 Acked Write 6 Done Done Sent false
10408 3547 Acked Read 6 Done Done Sent false
10409 3548 Acked Read 8 Done Done Sent false
10410 3549 Acked Read 5 Done Done Sent false
10411 3550 Acked Write 7 Done Done Sent false
10412 3551 Acked Read 8 Done Done Sent false
10413 3552 Acked Write 6 Done Done Sent false
10414 3553 Acked Write 7 Done Done Sent false
10415 3554 Acked Read 1 Done Done Sent false
10416 3555 Acked Read 1 Done Done Sent false
10417 3556 Acked Read 3 Done Done Sent false
10418 3557 Acked Write 10 Done Done Sent false
10419 3558 Acked Write 7 Done Done Sent false
10420 3559 Acked Read 4 Done Done Sent false
10421 3560 Acked Write 5 Done Done Sent false
10422 3561 Acked Write 2 Done Done Sent false
10423 3562 Acked Read 2 Done Done Sent false
10424 3563 Acked Write 10 Done Done Sent false
10425 3564 Acked Flush 0 Done Done Sent false
10426 3565 Acked Flush 0 Done Done Sent false
10427 3566 Acked Read 8 Done Done Sent false
10428 3567 Acked Write 2 Done Done Sent false
10429 3568 Acked Write 3 Done Done Sent false
10430 3569 Acked Read 1 Done Done Sent false
10431 3570 Acked Write 5 Done Done Sent false
10432 3571 Acked Read 6 Done Done Sent false
10433 3572 Acked Flush 0 Done Done Sent false
10434 3573 Acked Read 3 Done Done Sent false
10435 3574 Acked Read 8 Done Done Sent false
10436 3575 Acked Write 2 Done Done Sent false
10437 3576 Acked Read 2 Done Done Sent false
10438 3577 Acked Read 3 Done Done Sent false
10439 3578 Acked Write 5 Done Done Sent false
10440 3579 Acked Read 2 Done Done Sent false
10441 3580 Acked Write 6 Done Done Sent false
10442 3581 Acked Read 8 Done Done Sent false
10443 3582 Acked Write 6 Done Done Sent false
10444 3583 Acked Read 2 Done Done Sent false
10445 3584 Acked Read 8 Done Done Sent false
10446 3585 Acked Write 10 Done Done Sent false
10447 3586 Acked Read 3 Done Done Sent false
10448 3587 Acked Flush 0 Done Done Sent false
10449 3588 Acked Read 9 Done Done Sent false
10450 3589 Acked Read 7 Done Done Sent false
10451 3590 Acked Read 5 Done Done Sent false
10452 3591 Acked Write 3 Done Done Sent false
10453 3592 Acked Read 5 Done Done Sent false
10454 3593 Acked Write 10 Done Done Sent false
10455 3594 Acked Read 6 Done Done Sent false
10456 3595 Acked Read 7 Done Done Sent false
10457 3596 Acked Read 1 Done Done Sent false
10458 3597 Acked Read 7 Done Done Sent false
10459 3598 Acked Read 1 Done Done Sent false
10460 3599 Acked Read 9 Done Done Sent false
10461 3600 Acked Write 3 Done Done Sent false
10462 3601 Acked Write 2 Done Done Sent false
10463 3602 Acked Flush 0 Done Done Sent false
10464 3603 Acked Read 3 Done Done Sent false
10465 3604 Acked Flush 0 Done Done Sent false
10466 3605 Acked Write 2 Done Done Sent false
10467 3606 Acked Read 2 Done Done Sent false
10468 3607 Acked Read 6 Done Done Sent false
10469 3608 Acked Write 5 Done Done Sent false
10470 3609 Acked Write 5 Done Done Sent false
10471 3610 Acked Read 2 Done Done Sent false
10472 3611 Acked Read 2 Done Done Sent false
10473 3612 Acked Read 4 Done Done Sent false
10474 3613 Acked Write 7 Done Done Sent false
10475 3614 Acked Write 3 Done Done Sent false
10476 3615 Acked Read 2 Done Done Sent false
10477 3616 Acked Read 2 Done Done Sent false
10478 3617 Acked Read 8 Done Done Sent false
10479 3618 Acked Read 5 Done Done Sent false
10480 3619 Acked Read 6 Done Done Sent false
10481 3620 Acked Read 1 Done Done Sent false
10482 3621 Acked Flush 0 Done Done Sent false
10483 3622 Acked Read 7 Done Done Sent false
10484 3623 Acked Read 2 Done Done Sent false
10485 3624 Acked Flush 0 Done Done Sent false
10486 3625 Acked Write 8 Done Done Sent false
10487 3626 Acked Write 3 Done Done Sent false
10488 3627 Acked Read 9 Done Done Sent false
10489 3628 Acked Write 9 Done Done Sent false
10490 3629 Acked Write 2 Done Done Sent false
10491 3630 Acked Read 5 Done Done Sent false
10492 3631 Acked Read 3 Done Done Sent false
10493 3632 Acked Read 3 Done Done Sent false
10494 3633 Acked Write 4 Done Done Sent false
10495 3634 Acked Read 1 Done Done Sent false
10496 3635 Acked Flush 0 Done Done Sent false
10497 3636 Acked Write 5 Done Done Sent false
10498 3637 Acked Read 6 Done Done Sent false
10499 3638 Acked Flush 0 Done Done Sent false
10500 3639 Acked Read 5 Done Done Sent false
10501 3640 Acked Write 8 Done Done Sent false
10502 3641 Acked Read 2 Done Done Sent false
10503 3642 Acked Read 6 Done Done Sent false
10504 3643 Acked Read 4 Done Done Sent false
10505 3644 Acked Read 4 Done Done Sent false
10506 3645 Acked Read 7 Done Done Sent false
10507 3646 Acked Write 2 Done Done Sent false
10508 3647 Acked Write 10 Done Done Sent false
10509 3648 Acked Write 7 Done Done Sent false
10510 3649 Acked Read 3 Done Done Sent false
10511 3650 Acked Read 8 Done Done Sent false
10512 3651 Acked Flush 0 Done Done Sent false
10513 3652 Acked Write 1 Done Done Sent false
10514 3653 Acked Write 10 Done Done Sent false
10515 3654 Acked Write 5 Done Done Sent false
10516 3655 Acked Read 8 Done Done Sent false
10517 3656 Acked Read 7 Done Done Sent false
10518 3657 Acked Read 1 Done Done Sent false
10519 3658 Acked Write 3 Done Done Sent false
10520 3659 Acked Write 7 Done Done Sent false
10521 3660 Acked Write 6 Done Done Sent false
10522 3661 Acked Write 1 Done Done Sent false
10523 3662 Acked Write 1 Done Done Sent false
10524 3663 Acked Write 9 Done Done Sent false
10525 3664 Acked Flush 0 Done Done Sent false
10526 3665 Acked Read 5 Done Done Sent false
10527 3666 Acked Write 3 Done Done Sent false
10528 3667 Acked Write 9 Done Done Sent false
10529 3668 Acked Write 2 Done Done Sent false
10530 3669 Acked Write 3 Done Done Sent false
10531 3670 Acked Flush 0 Done Done Sent false
10532 3671 Acked Write 7 Done Done Sent false
10533 3672 Acked Read 10 Done Done Sent false
10534 3673 Acked Write 10 Done Done Sent false
10535 3674 Acked Read 10 Done Done Sent false
10536 3675 Acked Read 3 Done Done Sent false
10537 3676 Acked Write 2 Done Done Sent false
10538 3677 Acked Read 6 Done Done Sent false
10539 3678 Acked Write 7 Done Done Sent false
10540 3679 Acked Read 5 Done Done Sent false
10541 3680 Acked Flush 0 Done Done Sent false
10542 3681 Acked Write 6 Done Done Sent false
10543 3682 Acked Read 10 Done Done Sent false
10544 3683 Acked Read 10 Done Done Sent false
10545 3684 Acked Read 9 Done Done Sent false
10546 3685 Acked Read 9 Done Done Sent false
10547 3686 Acked Write 5 Done Done Sent false
10548 3687 Acked Flush 0 Done Done Sent false
10549 3688 Acked Read 3 Done Done Sent false
10550 3689 Acked Read 3 Done Done Sent false
10551 3690 Acked Write 3 Done Done Sent false
10552 3691 Acked Read 9 Done Done Sent false
10553 3692 Acked Write 9 Done Done Sent false
10554 3693 Acked Write 8 Done Done Sent false
10555 3694 Acked Write 3 Done Done Sent false
10556 3695 Acked Read 4 Done Done Sent false
10557 3696 Acked Write 3 Done Done Sent false
10558 3697 Acked Read 5 Done Done Sent false
10559 3698 Acked Write 8 Done Done Sent false
10560 3699 Acked Write 6 Done Done Sent false
10561 3700 Acked Read 4 Done Done Sent false
10562 3701 Acked Read 10 Done Done Sent false
10563 3702 Acked Read 5 Done Done Sent false
10564 3703 Acked Write 6 Done Done Sent false
10565 3704 Acked Write 10 Done Done Sent false
10566 3705 Acked Flush 0 Done Done Sent false
10567 3706 Acked Read 3 Done Done Sent false
10568 3707 Acked Read 7 Done Done Sent false
10569 3708 Acked Read 9 Done Done Sent false
10570 3709 Acked Write 4 Done Done Sent false
10571 3710 Acked Read 10 Done Done Sent false
10572 3711 Acked Read 3 Done Done Sent false
10573 3712 Acked Read 5 Done Done Sent false
10574 3713 Acked Flush 0 Done Done Sent false
10575 3714 Acked Write 2 Done Done Sent false
10576 3715 Acked Read 7 Done Done Sent false
10577 STATES DS:0 DS:1 DS:2 TOTAL
10578 Sent 0 0 301 301
10579 Done 301 301 0 602
10580 Skipped 0 0 0 0
10581 Error 0 0 0 0
10582 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
10583 Downstairs last five completed: 3414 3413 3412 3411 3410
10584 Upstairs last five completed: 3715 3714 3713 3712 3711
10585 CLIENT: Up:0 ds:301 act:2
105862025-02-25T20:48:56.277ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
105872025-02-25T20:48:57.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
105882025-02-25T20:48:58.281ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
10589 [8830] got start action
10590 [8830] I am going to restart
10591 Starting downstairs at port 8830
10592 Make output file at "/tmp/dsc/downstairs-8830.txt"
10593 [8830][2] reports Starting
10594 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
10595 [8830][2] reports Running
10596 ----------------------------------------------------------------
10597 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10598 States: Active Active Offline
10599 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10600 3415 Acked Read 9 Done Done Sent false
10601 3416 Acked Write 8 Done Done Sent false
10602 3417 Acked Flush 0 Done Done Sent false
10603 3418 Acked Write 7 Done Done Sent false
10604 3419 Acked Read 3 Done Done Sent false
10605 3420 Acked Read 8 Done Done Sent false
10606 3421 Acked Flush 0 Done Done Sent false
10607 3422 Acked Write 7 Done Done Sent false
10608 3423 Acked Write 2 Done Done Sent false
10609 3424 Acked Read 1 Done Done Sent false
10610 3425 Acked Read 9 Done Done Sent false
10611 3426 Acked Write 3 Done Done Sent false
10612 3427 Acked Read 6 Done Done Sent false
10613 3428 Acked Read 8 Done Done Sent false
10614 3429 Acked Write 3 Done Done Sent false
10615 3430 Acked Write 3 Done Done Sent false
10616 3431 Acked Read 5 Done Done Sent false
10617 3432 Acked Flush 0 Done Done Sent false
10618 3433 Acked Read 5 Done Done Sent false
10619 3434 Acked Read 4 Done Done Sent false
10620 3435 Acked Write 7 Done Done Sent false
10621 3436 Acked Flush 0 Done Done Sent false
10622 3437 Acked Write 8 Done Done Sent false
10623 3438 Acked Read 8 Done Done Sent false
10624 3439 Acked Read 5 Done Done Sent false
10625 3440 Acked Read 5 Done Done Sent false
10626 3441 Acked Write 1 Done Done Sent false
10627 3442 Acked Write 3 Done Done Sent false
10628 3443 Acked Read 7 Done Done Sent false
10629 3444 Acked Read 4 Done Done Sent false
10630 3445 Acked Read 5 Done Done Sent false
10631 3446 Acked Write 6 Done Done Sent false
10632 3447 Acked Read 10 Done Done Sent false
10633 3448 Acked Flush 0 Done Done Sent false
10634 3449 Acked Read 2 Done Done Sent false
10635 3450 Acked Write 2 Done Done Sent false
10636 3451 Acked Read 8 Done Done Sent false
10637 3452 Acked Write 8 Done Done Sent false
10638 3453 Acked Read 7 Done Done Sent false
10639 3454 Acked Write 8 Done Done Sent false
10640 3455 Acked Read 1 Done Done Sent false
10641 3456 Acked Flush 0 Done Done Sent false
10642 3457 Acked Read 10 Done Done Sent false
10643 3458 Acked Write 1 Done Done Sent false
10644 3459 Acked Read 10 Done Done Sent false
10645 3460 Acked Write 9 Done Done Sent false
10646 3461 Acked Write 8 Done Done Sent false
10647 3462 Acked Write 4 Done Done Sent false
10648 3463 Acked Read 9 Done Done Sent false
10649 3464 Acked Read 2 Done Done Sent false
10650 3465 Acked Read 6 Done Done Sent false
10651 3466 Acked Write 10 Done Done Sent false
10652 3467 Acked Write 4 Done Done Sent false
10653 3468 Acked Write 1 Done Done Sent false
10654 3469 Acked Write 8 Done Done Sent false
10655 3470 Acked Flush 0 Done Done Sent false
10656 3471 Acked Read 4 Done Done Sent false
10657 3472 Acked Write 5 Done Done Sent false
10658 3473 Acked Read 2 Done Done Sent false
10659 3474 Acked Write 4 Done Done Sent false
10660 3475 Acked Read 9 Done Done Sent false
10661 3476 Acked Read 7 Done Done Sent false
10662 3477 Acked Write 1 Done Done Sent false
10663 3478 Acked Write 4 Done Done Sent false
10664 3479 Acked Read 10 Done Done Sent false
10665 3480 Acked Read 7 Done Done Sent false
10666 3481 Acked Write 4 Done Done Sent false
10667 3482 Acked Read 7 Done Done Sent false
10668 3483 Acked Write 1 Done Done Sent false
10669 3484 Acked Write 7 Done Done Sent false
10670 3485 Acked Read 8 Done Done Sent false
10671 3486 Acked Write 5 Done Done Sent false
10672 3487 Acked Write 9 Done Done Sent false
10673 3488 Acked Write 8 Done Done Sent false
10674 3489 Acked Write 2 Done Done Sent false
10675 3490 Acked Read 6 Done Done Sent false
10676 3491 Acked Read 6 Done Done Sent false
10677 3492 Acked Write 9 Done Done Sent false
10678 3493 Acked Read 3 Done Done Sent false
10679 3494 Acked Write 3 Done Done Sent false
10680 3495 Acked Read 1 Done Done Sent false
10681 3496 Acked Flush 0 Done Done Sent false
10682 3497 Acked Read 8 Done Done Sent false
10683 3498 Acked Read 4 Done Done Sent false
10684 3499 Acked Read 7 Done Done Sent false
10685 3500 Acked Write 10 Done Done Sent false
10686 3501 Acked Read 9 Done Done Sent false
10687 3502 Acked Read 8 Done Done Sent false
10688 3503 Acked Read 10 Done Done Sent false
10689 3504 Acked Write 3 Done Done Sent false
10690 3505 Acked Write 10 Done Done Sent false
10691 3506 Acked Read 8 Done Done Sent false
10692 3507 Acked Read 7 Done Done Sent false
10693 3508 Acked Read 6 Done Done Sent false
10694 3509 Acked Read 10 Done Done Sent false
10695 3510 Acked Write 3 Done Done Sent false
10696 3511 Acked Write 7 Done Done Sent false
10697 3512 Acked Read 7 Done Done Sent false
10698 3513 Acked Read 6 Done Done Sent false
10699 3514 Acked Read 5 Done Done Sent false
10700 3515 Acked Read 10 Done Done Sent false
10701 3516 Acked Read 10 Done Done Sent false
10702 3517 Acked Read 6 Done Done Sent false
10703 3518 Acked Read 4 Done Done Sent false
10704 3519 Acked Read 6 Done Done Sent false
10705 3520 Acked Read 1 Done Done Sent false
10706 3521 Acked Read 5 Done Done Sent false
10707 3522 Acked Write 8 Done Done Sent false
10708 3523 Acked Write 3 Done Done Sent false
10709 3524 Acked Flush 0 Done Done Sent false
10710 3525 Acked Write 8 Done Done Sent false
10711 3526 Acked Write 4 Done Done Sent false
10712 3527 Acked Read 10 Done Done Sent false
10713 3528 Acked Read 2 Done Done Sent false
10714 3529 Acked Read 1 Done Done Sent false
10715 3530 Acked Write 7 Done Done Sent false
10716 3531 Acked Write 3 Done Done Sent false
10717 3532 Acked Flush 0 Done Done Sent false
10718 3533 Acked Read 5 Done Done Sent false
10719 3534 Acked Write 1 Done Done Sent false
10720 3535 Acked Read 10 Done Done Sent false
10721 3536 Acked Read 5 Done Done Sent false
10722 3537 Acked Write 6 Done Done Sent false
10723 3538 Acked Write 6 Done Done Sent false
10724 3539 Acked Read 8 Done Done Sent false
10725 3540 Acked Write 4 Done Done Sent false
10726 3541 Acked Read 9 Done Done Sent false
10727 3542 Acked Read 8 Done Done Sent false
10728 3543 Acked Write 9 Done Done Sent false
10729 3544 Acked Write 5 Done Done Sent false
10730 3545 Acked Flush 0 Done Done Sent false
10731 3546 Acked Write 6 Done Done Sent false
10732 3547 Acked Read 6 Done Done Sent false
10733 3548 Acked Read 8 Done Done Sent false
10734 3549 Acked Read 5 Done Done Sent false
10735 3550 Acked Write 7 Done Done Sent false
10736 3551 Acked Read 8 Done Done Sent false
10737 3552 Acked Write 6 Done Done Sent false
10738 3553 Acked Write 7 Done Done Sent false
10739 3554 Acked Read 1 Done Done Sent false
10740 3555 Acked Read 1 Done Done Sent false
10741 3556 Acked Read 3 Done Done Sent false
10742 3557 Acked Write 10 Done Done Sent false
10743 3558 Acked Write 7 Done Done Sent false
10744 3559 Acked Read 4 Done Done Sent false
10745 3560 Acked Write 5 Done Done Sent false
10746 3561 Acked Write 2 Done Done Sent false
10747 3562 Acked Read 2 Done Done Sent false
10748 3563 Acked Write 10 Done Done Sent false
10749 3564 Acked Flush 0 Done Done Sent false
10750 3565 Acked Flush 0 Done Done Sent false
10751 3566 Acked Read 8 Done Done Sent false
10752 3567 Acked Write 2 Done Done Sent false
10753 3568 Acked Write 3 Done Done Sent false
10754 3569 Acked Read 1 Done Done Sent false
10755 3570 Acked Write 5 Done Done Sent false
10756 3571 Acked Read 6 Done Done Sent false
10757 3572 Acked Flush 0 Done Done Sent false
10758 3573 Acked Read 3 Done Done Sent false
10759 3574 Acked Read 8 Done Done Sent false
10760 3575 Acked Write 2 Done Done Sent false
10761 3576 Acked Read 2 Done Done Sent false
10762 3577 Acked Read 3 Done Done Sent false
10763 3578 Acked Write 5 Done Done Sent false
10764 3579 Acked Read 2 Done Done Sent false
10765 3580 Acked Write 6 Done Done Sent false
10766 3581 Acked Read 8 Done Done Sent false
10767 3582 Acked Write 6 Done Done Sent false
10768 3583 Acked Read 2 Done Done Sent false
10769 3584 Acked Read 8 Done Done Sent false
10770 3585 Acked Write 10 Done Done Sent false
10771 3586 Acked Read 3 Done Done Sent false
10772 3587 Acked Flush 0 Done Done Sent false
10773 3588 Acked Read 9 Done Done Sent false
10774 3589 Acked Read 7 Done Done Sent false
10775 3590 Acked Read 5 Done Done Sent false
10776 3591 Acked Write 3 Done Done Sent false
10777 3592 Acked Read 5 Done Done Sent false
10778 3593 Acked Write 10 Done Done Sent false
10779 3594 Acked Read 6 Done Done Sent false
10780 3595 Acked Read 7 Done Done Sent false
10781 3596 Acked Read 1 Done Done Sent false
10782 3597 Acked Read 7 Done Done Sent false
10783 3598 Acked Read 1 Done Done Sent false
10784 3599 Acked Read 9 Done Done Sent false
10785 3600 Acked Write 3 Done Done Sent false
10786 3601 Acked Write 2 Done Done Sent false
10787 3602 Acked Flush 0 Done Done Sent false
10788 3603 Acked Read 3 Done Done Sent false
10789 3604 Acked Flush 0 Done Done Sent false
10790 3605 Acked Write 2 Done Done Sent false
10791 3606 Acked Read 2 Done Done Sent false
10792 3607 Acked Read 6 Done Done Sent false
10793 3608 Acked Write 5 Done Done Sent false
10794 3609 Acked Write 5 Done Done Sent false
10795 3610 Acked Read 2 Done Done Sent false
10796 3611 Acked Read 2 Done Done Sent false
10797 3612 Acked Read 4 Done Done Sent false
10798 3613 Acked Write 7 Done Done Sent false
10799 3614 Acked Write 3 Done Done Sent false
10800 3615 Acked Read 2 Done Done Sent false
10801 3616 Acked Read 2 Done Done Sent false
10802 3617 Acked Read 8 Done Done Sent false
10803 3618 Acked Read 5 Done Done Sent false
10804 3619 Acked Read 6 Done Done Sent false
10805 3620 Acked Read 1 Done Done Sent false
10806 3621 Acked Flush 0 Done Done Sent false
10807 3622 Acked Read 7 Done Done Sent false
10808 3623 Acked Read 2 Done Done Sent false
10809 3624 Acked Flush 0 Done Done Sent false
10810 3625 Acked Write 8 Done Done Sent false
10811 3626 Acked Write 3 Done Done Sent false
10812 3627 Acked Read 9 Done Done Sent false
10813 3628 Acked Write 9 Done Done Sent false
10814 3629 Acked Write 2 Done Done Sent false
10815 3630 Acked Read 5 Done Done Sent false
10816 3631 Acked Read 3 Done Done Sent false
10817 3632 Acked Read 3 Done Done Sent false
10818 3633 Acked Write 4 Done Done Sent false
10819 3634 Acked Read 1 Done Done Sent false
10820 3635 Acked Flush 0 Done Done Sent false
10821 3636 Acked Write 5 Done Done Sent false
10822 3637 Acked Read 6 Done Done Sent false
10823 3638 Acked Flush 0 Done Done Sent false
10824 3639 Acked Read 5 Done Done Sent false
10825 3640 Acked Write 8 Done Done Sent false
10826 3641 Acked Read 2 Done Done Sent false
10827 3642 Acked Read 6 Done Done Sent false
10828 3643 Acked Read 4 Done Done Sent false
10829 3644 Acked Read 4 Done Done Sent false
10830 3645 Acked Read 7 Done Done Sent false
10831 3646 Acked Write 2 Done Done Sent false
10832 3647 Acked Write 10 Done Done Sent false
10833 3648 Acked Write 7 Done Done Sent false
10834 3649 Acked Read 3 Done Done Sent false
10835 3650 Acked Read 8 Done Done Sent false
10836 3651 Acked Flush 0 Done Done Sent false
10837 3652 Acked Write 1 Done Done Sent false
10838 3653 Acked Write 10 Done Done Sent false
10839 3654 Acked Write 5 Done Done Sent false
10840 3655 Acked Read 8 Done Done Sent false
10841 3656 Acked Read 7 Done Done Sent false
10842 3657 Acked Read 1 Done Done Sent false
10843 3658 Acked Write 3 Done Done Sent false
10844 3659 Acked Write 7 Done Done Sent false
10845 3660 Acked Write 6 Done Done Sent false
10846 3661 Acked Write 1 Done Done Sent false
10847 3662 Acked Write 1 Done Done Sent false
10848 3663 Acked Write 9 Done Done Sent false
10849 3664 Acked Flush 0 Done Done Sent false
10850 3665 Acked Read 5 Done Done Sent false
10851 3666 Acked Write 3 Done Done Sent false
10852 3667 Acked Write 9 Done Done Sent false
10853 3668 Acked Write 2 Done Done Sent false
10854 3669 Acked Write 3 Done Done Sent false
10855 3670 Acked Flush 0 Done Done Sent false
10856 3671 Acked Write 7 Done Done Sent false
10857 3672 Acked Read 10 Done Done Sent false
10858 3673 Acked Write 10 Done Done Sent false
10859 3674 Acked Read 10 Done Done Sent false
10860 3675 Acked Read 3 Done Done Sent false
10861 3676 Acked Write 2 Done Done Sent false
10862 3677 Acked Read 6 Done Done Sent false
10863 3678 Acked Write 7 Done Done Sent false
10864 3679 Acked Read 5 Done Done Sent false
10865 3680 Acked Flush 0 Done Done Sent false
10866 3681 Acked Write 6 Done Done Sent false
10867 3682 Acked Read 10 Done Done Sent false
10868 3683 Acked Read 10 Done Done Sent false
10869 3684 Acked Read 9 Done Done Sent false
10870 3685 Acked Read 9 Done Done Sent false
10871 3686 Acked Write 5 Done Done Sent false
10872 3687 Acked Flush 0 Done Done Sent false
10873 3688 Acked Read 3 Done Done Sent false
10874 3689 Acked Read 3 Done Done Sent false
10875 3690 Acked Write 3 Done Done Sent false
10876 3691 Acked Read 9 Done Done Sent false
10877 3692 Acked Write 9 Done Done Sent false
10878 3693 Acked Write 8 Done Done Sent false
10879 3694 Acked Write 3 Done Done Sent false
10880 3695 Acked Read 4 Done Done Sent false
10881 3696 Acked Write 3 Done Done Sent false
10882 3697 Acked Read 5 Done Done Sent false
10883 3698 Acked Write 8 Done Done Sent false
10884 3699 Acked Write 6 Done Done Sent false
10885 3700 Acked Read 4 Done Done Sent false
10886 3701 Acked Read 10 Done Done Sent false
10887 3702 Acked Read 5 Done Done Sent false
10888 3703 Acked Write 6 Done Done Sent false
10889 3704 Acked Write 10 Done Done Sent false
10890 3705 Acked Flush 0 Done Done Sent false
10891 3706 Acked Read 3 Done Done Sent false
10892 3707 Acked Read 7 Done Done Sent false
10893 3708 Acked Read 9 Done Done Sent false
10894 3709 Acked Write 4 Done Done Sent false
10895 3710 Acked Read 10 Done Done Sent false
10896 3711 Acked Read 3 Done Done Sent false
10897 3712 Acked Read 5 Done Done Sent false
10898 3713 Acked Flush 0 Done Done Sent false
10899 3714 Acked Write 2 Done Done Sent false
10900 3715 Acked Read 7 Done Done Sent false
10901 STATES DS:0 DS:1 DS:2 TOTAL
10902 Sent 0 0 301 301
10903 Done 301 301 0 602
10904 Skipped 0 0 0 0
10905 Error 0 0 0 0
10906 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
10907 Downstairs last five completed: 3414 3413 3412 3411 3410
10908 Upstairs last five completed: 3715 3714 3713 3712 3711
10909 CLIENT: Up:0 ds:301 act:2
109102025-02-25T20:48:59.405ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
109112025-02-25T20:49:01.287ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
109122025-02-25T20:49:01.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
109132025-02-25T20:49:02.290ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
10914 ----------------------------------------------------------------
10915 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10916 States: Active Active Offline
10917 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10918 3415 Acked Read 9 Done Done Sent false
10919 3416 Acked Write 8 Done Done Sent false
10920 3417 Acked Flush 0 Done Done Sent false
10921 3418 Acked Write 7 Done Done Sent false
10922 3419 Acked Read 3 Done Done Sent false
10923 3420 Acked Read 8 Done Done Sent false
10924 3421 Acked Flush 0 Done Done Sent false
10925 3422 Acked Write 7 Done Done Sent false
10926 3423 Acked Write 2 Done Done Sent false
10927 3424 Acked Read 1 Done Done Sent false
10928 3425 Acked Read 9 Done Done Sent false
10929 3426 Acked Write 3 Done Done Sent false
10930 3427 Acked Read 6 Done Done Sent false
10931 3428 Acked Read 8 Done Done Sent false
10932 3429 Acked Write 3 Done Done Sent false
10933 3430 Acked Write 3 Done Done Sent false
10934 3431 Acked Read 5 Done Done Sent false
10935 3432 Acked Flush 0 Done Done Sent false
10936 3433 Acked Read 5 Done Done Sent false
10937 3434 Acked Read 4 Done Done Sent false
10938 3435 Acked Write 7 Done Done Sent false
10939 3436 Acked Flush 0 Done Done Sent false
10940 3437 Acked Write 8 Done Done Sent false
10941 3438 Acked Read 8 Done Done Sent false
10942 3439 Acked Read 5 Done Done Sent false
10943 3440 Acked Read 5 Done Done Sent false
10944 3441 Acked Write 1 Done Done Sent false
10945 3442 Acked Write 3 Done Done Sent false
10946 3443 Acked Read 7 Done Done Sent false
10947 3444 Acked Read 4 Done Done Sent false
10948 3445 Acked Read 5 Done Done Sent false
10949 3446 Acked Write 6 Done Done Sent false
10950 3447 Acked Read 10 Done Done Sent false
10951 3448 Acked Flush 0 Done Done Sent false
10952 3449 Acked Read 2 Done Done Sent false
10953 3450 Acked Write 2 Done Done Sent false
10954 3451 Acked Read 8 Done Done Sent false
10955 3452 Acked Write 8 Done Done Sent false
10956 3453 Acked Read 7 Done Done Sent false
10957 3454 Acked Write 8 Done Done Sent false
10958 3455 Acked Read 1 Done Done Sent false
10959 3456 Acked Flush 0 Done Done Sent false
10960 3457 Acked Read 10 Done Done Sent false
10961 3458 Acked Write 1 Done Done Sent false
10962 3459 Acked Read 10 Done Done Sent false
10963 3460 Acked Write 9 Done Done Sent false
10964 3461 Acked Write 8 Done Done Sent false
10965 3462 Acked Write 4 Done Done Sent false
10966 3463 Acked Read 9 Done Done Sent false
10967 3464 Acked Read 2 Done Done Sent false
10968 3465 Acked Read 6 Done Done Sent false
10969 3466 Acked Write 10 Done Done Sent false
10970 3467 Acked Write 4 Done Done Sent false
10971 3468 Acked Write 1 Done Done Sent false
10972 3469 Acked Write 8 Done Done Sent false
10973 3470 Acked Flush 0 Done Done Sent false
10974 3471 Acked Read 4 Done Done Sent false
10975 3472 Acked Write 5 Done Done Sent false
10976 3473 Acked Read 2 Done Done Sent false
10977 3474 Acked Write 4 Done Done Sent false
10978 3475 Acked Read 9 Done Done Sent false
10979 3476 Acked Read 7 Done Done Sent false
10980 3477 Acked Write 1 Done Done Sent false
10981 3478 Acked Write 4 Done Done Sent false
10982 3479 Acked Read 10 Done Done Sent false
10983 3480 Acked Read 7 Done Done Sent false
10984 3481 Acked Write 4 Done Done Sent false
10985 3482 Acked Read 7 Done Done Sent false
10986 3483 Acked Write 1 Done Done Sent false
10987 3484 Acked Write 7 Done Done Sent false
10988 3485 Acked Read 8 Done Done Sent false
10989 3486 Acked Write 5 Done Done Sent false
10990 3487 Acked Write 9 Done Done Sent false
10991 3488 Acked Write 8 Done Done Sent false
10992 3489 Acked Write 2 Done Done Sent false
10993 3490 Acked Read 6 Done Done Sent false
10994 3491 Acked Read 6 Done Done Sent false
10995 3492 Acked Write 9 Done Done Sent false
10996 3493 Acked Read 3 Done Done Sent false
10997 3494 Acked Write 3 Done Done Sent false
10998 3495 Acked Read 1 Done Done Sent false
10999 3496 Acked Flush 0 Done Done Sent false
11000 3497 Acked Read 8 Done Done Sent false
11001 3498 Acked Read 4 Done Done Sent false
11002 3499 Acked Read 7 Done Done Sent false
11003 3500 Acked Write 10 Done Done Sent false
11004 3501 Acked Read 9 Done Done Sent false
11005 3502 Acked Read 8 Done Done Sent false
11006 3503 Acked Read 10 Done Done Sent false
11007 3504 Acked Write 3 Done Done Sent false
11008 3505 Acked Write 10 Done Done Sent false
11009 3506 Acked Read 8 Done Done Sent false
11010 3507 Acked Read 7 Done Done Sent false
11011 3508 Acked Read 6 Done Done Sent false
11012 3509 Acked Read 10 Done Done Sent false
11013 3510 Acked Write 3 Done Done Sent false
11014 3511 Acked Write 7 Done Done Sent false
11015 3512 Acked Read 7 Done Done Sent false
11016 3513 Acked Read 6 Done Done Sent false
11017 3514 Acked Read 5 Done Done Sent false
11018 3515 Acked Read 10 Done Done Sent false
11019 3516 Acked Read 10 Done Done Sent false
11020 3517 Acked Read 6 Done Done Sent false
11021 3518 Acked Read 4 Done Done Sent false
11022 3519 Acked Read 6 Done Done Sent false
11023 3520 Acked Read 1 Done Done Sent false
11024 3521 Acked Read 5 Done Done Sent false
11025 3522 Acked Write 8 Done Done Sent false
11026 3523 Acked Write 3 Done Done Sent false
11027 3524 Acked Flush 0 Done Done Sent false
11028 3525 Acked Write 8 Done Done Sent false
11029 3526 Acked Write 4 Done Done Sent false
11030 3527 Acked Read 10 Done Done Sent false
11031 3528 Acked Read 2 Done Done Sent false
11032 3529 Acked Read 1 Done Done Sent false
11033 3530 Acked Write 7 Done Done Sent false
11034 3531 Acked Write 3 Done Done Sent false
11035 3532 Acked Flush 0 Done Done Sent false
11036 3533 Acked Read 5 Done Done Sent false
11037 3534 Acked Write 1 Done Done Sent false
11038 3535 Acked Read 10 Done Done Sent false
11039 3536 Acked Read 5 Done Done Sent false
11040 3537 Acked Write 6 Done Done Sent false
11041 3538 Acked Write 6 Done Done Sent false
11042 3539 Acked Read 8 Done Done Sent false
11043 3540 Acked Write 4 Done Done Sent false
11044 3541 Acked Read 9 Done Done Sent false
11045 3542 Acked Read 8 Done Done Sent false
11046 3543 Acked Write 9 Done Done Sent false
11047 3544 Acked Write 5 Done Done Sent false
11048 3545 Acked Flush 0 Done Done Sent false
11049 3546 Acked Write 6 Done Done Sent false
11050 3547 Acked Read 6 Done Done Sent false
11051 3548 Acked Read 8 Done Done Sent false
11052 3549 Acked Read 5 Done Done Sent false
11053 3550 Acked Write 7 Done Done Sent false
11054 3551 Acked Read 8 Done Done Sent false
11055 3552 Acked Write 6 Done Done Sent false
11056 3553 Acked Write 7 Done Done Sent false
11057 3554 Acked Read 1 Done Done Sent false
11058 3555 Acked Read 1 Done Done Sent false
11059 3556 Acked Read 3 Done Done Sent false
11060 3557 Acked Write 10 Done Done Sent false
11061 3558 Acked Write 7 Done Done Sent false
11062 3559 Acked Read 4 Done Done Sent false
11063 3560 Acked Write 5 Done Done Sent false
11064 3561 Acked Write 2 Done Done Sent false
11065 3562 Acked Read 2 Done Done Sent false
11066 3563 Acked Write 10 Done Done Sent false
11067 3564 Acked Flush 0 Done Done Sent false
11068 3565 Acked Flush 0 Done Done Sent false
11069 3566 Acked Read 8 Done Done Sent false
11070 3567 Acked Write 2 Done Done Sent false
11071 3568 Acked Write 3 Done Done Sent false
11072 3569 Acked Read 1 Done Done Sent false
11073 3570 Acked Write 5 Done Done Sent false
11074 3571 Acked Read 6 Done Done Sent false
11075 3572 Acked Flush 0 Done Done Sent false
11076 3573 Acked Read 3 Done Done Sent false
11077 3574 Acked Read 8 Done Done Sent false
11078 3575 Acked Write 2 Done Done Sent false
11079 3576 Acked Read 2 Done Done Sent false
11080 3577 Acked Read 3 Done Done Sent false
11081 3578 Acked Write 5 Done Done Sent false
11082 3579 Acked Read 2 Done Done Sent false
11083 3580 Acked Write 6 Done Done Sent false
11084 3581 Acked Read 8 Done Done Sent false
11085 3582 Acked Write 6 Done Done Sent false
11086 3583 Acked Read 2 Done Done Sent false
11087 3584 Acked Read 8 Done Done Sent false
11088 3585 Acked Write 10 Done Done Sent false
11089 3586 Acked Read 3 Done Done Sent false
11090 3587 Acked Flush 0 Done Done Sent false
11091 3588 Acked Read 9 Done Done Sent false
11092 3589 Acked Read 7 Done Done Sent false
11093 3590 Acked Read 5 Done Done Sent false
11094 3591 Acked Write 3 Done Done Sent false
11095 3592 Acked Read 5 Done Done Sent false
11096 3593 Acked Write 10 Done Done Sent false
11097 3594 Acked Read 6 Done Done Sent false
11098 3595 Acked Read 7 Done Done Sent false
11099 3596 Acked Read 1 Done Done Sent false
11100 3597 Acked Read 7 Done Done Sent false
11101 3598 Acked Read 1 Done Done Sent false
11102 3599 Acked Read 9 Done Done Sent false
11103 3600 Acked Write 3 Done Done Sent false
11104 3601 Acked Write 2 Done Done Sent false
11105 3602 Acked Flush 0 Done Done Sent false
11106 3603 Acked Read 3 Done Done Sent false
11107 3604 Acked Flush 0 Done Done Sent false
11108 3605 Acked Write 2 Done Done Sent false
11109 3606 Acked Read 2 Done Done Sent false
11110 3607 Acked Read 6 Done Done Sent false
11111 3608 Acked Write 5 Done Done Sent false
11112 3609 Acked Write 5 Done Done Sent false
11113 3610 Acked Read 2 Done Done Sent false
11114 3611 Acked Read 2 Done Done Sent false
11115 3612 Acked Read 4 Done Done Sent false
11116 3613 Acked Write 7 Done Done Sent false
11117 3614 Acked Write 3 Done Done Sent false
11118 3615 Acked Read 2 Done Done Sent false
11119 3616 Acked Read 2 Done Done Sent false
11120 3617 Acked Read 8 Done Done Sent false
11121 3618 Acked Read 5 Done Done Sent false
11122 3619 Acked Read 6 Done Done Sent false
11123 3620 Acked Read 1 Done Done Sent false
11124 3621 Acked Flush 0 Done Done Sent false
11125 3622 Acked Read 7 Done Done Sent false
11126 3623 Acked Read 2 Done Done Sent false
11127 3624 Acked Flush 0 Done Done Sent false
11128 3625 Acked Write 8 Done Done Sent false
11129 3626 Acked Write 3 Done Done Sent false
11130 3627 Acked Read 9 Done Done Sent false
11131 3628 Acked Write 9 Done Done Sent false
11132 3629 Acked Write 2 Done Done Sent false
11133 3630 Acked Read 5 Done Done Sent false
11134 3631 Acked Read 3 Done Done Sent false
11135 3632 Acked Read 3 Done Done Sent false
11136 3633 Acked Write 4 Done Done Sent false
11137 3634 Acked Read 1 Done Done Sent false
11138 3635 Acked Flush 0 Done Done Sent false
11139 3636 Acked Write 5 Done Done Sent false
11140 3637 Acked Read 6 Done Done Sent false
11141 3638 Acked Flush 0 Done Done Sent false
11142 3639 Acked Read 5 Done Done Sent false
11143 3640 Acked Write 8 Done Done Sent false
11144 3641 Acked Read 2 Done Done Sent false
11145 3642 Acked Read 6 Done Done Sent false
11146 3643 Acked Read 4 Done Done Sent false
11147 3644 Acked Read 4 Done Done Sent false
11148 3645 Acked Read 7 Done Done Sent false
11149 3646 Acked Write 2 Done Done Sent false
11150 3647 Acked Write 10 Done Done Sent false
11151 3648 Acked Write 7 Done Done Sent false
11152 3649 Acked Read 3 Done Done Sent false
11153 3650 Acked Read 8 Done Done Sent false
11154 3651 Acked Flush 0 Done Done Sent false
11155 3652 Acked Write 1 Done Done Sent false
11156 3653 Acked Write 10 Done Done Sent false
11157 3654 Acked Write 5 Done Done Sent false
11158 3655 Acked Read 8 Done Done Sent false
11159 3656 Acked Read 7 Done Done Sent false
11160 3657 Acked Read 1 Done Done Sent false
11161 3658 Acked Write 3 Done Done Sent false
11162 3659 Acked Write 7 Done Done Sent false
11163 3660 Acked Write 6 Done Done Sent false
11164 3661 Acked Write 1 Done Done Sent false
11165 3662 Acked Write 1 Done Done Sent false
11166 3663 Acked Write 9 Done Done Sent false
11167 3664 Acked Flush 0 Done Done Sent false
11168 3665 Acked Read 5 Done Done Sent false
11169 3666 Acked Write 3 Done Done Sent false
11170 3667 Acked Write 9 Done Done Sent false
11171 3668 Acked Write 2 Done Done Sent false
11172 3669 Acked Write 3 Done Done Sent false
11173 3670 Acked Flush 0 Done Done Sent false
11174 3671 Acked Write 7 Done Done Sent false
11175 3672 Acked Read 10 Done Done Sent false
11176 3673 Acked Write 10 Done Done Sent false
11177 3674 Acked Read 10 Done Done Sent false
11178 3675 Acked Read 3 Done Done Sent false
11179 3676 Acked Write 2 Done Done Sent false
11180 3677 Acked Read 6 Done Done Sent false
11181 3678 Acked Write 7 Done Done Sent false
11182 3679 Acked Read 5 Done Done Sent false
11183 3680 Acked Flush 0 Done Done Sent false
11184 3681 Acked Write 6 Done Done Sent false
11185 3682 Acked Read 10 Done Done Sent false
11186 3683 Acked Read 10 Done Done Sent false
11187 3684 Acked Read 9 Done Done Sent false
11188 3685 Acked Read 9 Done Done Sent false
11189 3686 Acked Write 5 Done Done Sent false
11190 3687 Acked Flush 0 Done Done Sent false
11191 3688 Acked Read 3 Done Done Sent false
11192 3689 Acked Read 3 Done Done Sent false
11193 3690 Acked Write 3 Done Done Sent false
11194 3691 Acked Read 9 Done Done Sent false
11195 3692 Acked Write 9 Done Done Sent false
11196 3693 Acked Write 8 Done Done Sent false
11197 3694 Acked Write 3 Done Done Sent false
11198 3695 Acked Read 4 Done Done Sent false
11199 3696 Acked Write 3 Done Done Sent false
11200 3697 Acked Read 5 Done Done Sent false
11201 3698 Acked Write 8 Done Done Sent false
11202 3699 Acked Write 6 Done Done Sent false
11203 3700 Acked Read 4 Done Done Sent false
11204 3701 Acked Read 10 Done Done Sent false
11205 3702 Acked Read 5 Done Done Sent false
11206 3703 Acked Write 6 Done Done Sent false
11207 3704 Acked Write 10 Done Done Sent false
11208 3705 Acked Flush 0 Done Done Sent false
11209 3706 Acked Read 3 Done Done Sent false
11210 3707 Acked Read 7 Done Done Sent false
11211 3708 Acked Read 9 Done Done Sent false
11212 3709 Acked Write 4 Done Done Sent false
11213 3710 Acked Read 10 Done Done Sent false
11214 3711 Acked Read 3 Done Done Sent false
11215 3712 Acked Read 5 Done Done Sent false
11216 3713 Acked Flush 0 Done Done Sent false
11217 3714 Acked Write 2 Done Done Sent false
11218 3715 Acked Read 7 Done Done Sent false
11219 STATES DS:0 DS:1 DS:2 TOTAL
11220 Sent 0 0 301 301
11221 Done 301 301 0 602
11222 Skipped 0 0 0 0
11223 Error 0 0 0 0
11224 Last Flush: Some(JobId(3713)) Some(JobId(3713)) Some(JobId(3414))
11225 Downstairs last five completed: 3414 3413 3412 3411 3410
11226 Upstairs last five completed: 3715 3714 3713 3712 3711
11227 CLIENT: Up:0 ds:301 act:2
112282025-02-25T20:49:03.406ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112292025-02-25T20:49:04.293ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112302025-02-25T20:49:04.639ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112312025-02-25T20:49:04.639ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54376) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112322025-02-25T20:49:04.641ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112332025-02-25T20:49:04.641ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112342025-02-25T20:49:04.641ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112352025-02-25T20:49:04.641ZINFOcrucible: send last flush ID to this DS: Some(JobId(3414)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112362025-02-25T20:49:04.641ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112372025-02-25T20:49:04.641ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3414)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112382025-02-25T20:49:04.642ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3414)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11239 ----------------------------------------------------------------
11240 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11241 Downstairs last five completed: 3716 3715 3714 3713 3712
11242 Upstairs last five completed: 3716 3715 3714 3713 3712
11243 CLIENT: Up:0 ds:0 act:3
11244 Replay: All jobs finished
11245 Main task has work to do, go find it
11246 got dsc Stop(1)
11247 stop 1
11248 Feb 25 20:49:06.908 INFO request completed, latency_us: 349, response_code: 204, uri: [8820] Got stop action so:false kr:false
11249 /stop/cid/1, method: POST, req_id: 6968e49a-5b4a-48c6-b773-81a3edfd7cc6, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
112502025-02-25T20:49:06.908ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112512025-02-25T20:49:06.908ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
112522025-02-25T20:49:06.908ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11253 Feb 25 20:49:06.908 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5095bee2-33c2-40d9-8af7-d27bfd45c7ff, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
112542025-02-25T20:49:06.909ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11255 [8820] Exited with: ExitStatus(unix_wait_status(9))
11256 [8820][1] reports Exit
11257 Feb 25 20:49:10.911 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9e3e4d31-2a28-4dad-8d4b-33c7e4ffca04, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
11258 001/300 Read block 4011 len 40960
11259 002/300 Read block 899 len 36864
11260 003/300 Write block 2507 len 20480 data: 3 3 2 2 3
11261 004/300 Write block 3252 len 24576 data: 4 4 3 3 2 3
11262 005/300 Read block 2240 len 20480
11263 006/300 Read block 321 len 12288
11264 007/300 Read block 3864 len 12288
11265 008/300 Write block 1007 len 40960 data: 3 3 3 2 2 4 4 4 4 4
11266 009/300 Write block 2375 len 20480 data: 2 2 4 4 4
11267 010/300 Write block 4118 len 24576 data: 3 3 3 3 3 4
11268 011/300 Write block 3068 len 16384 data: 3 3 3 2
11269 012/300 Read block 2694 len 24576
11270 013/300 Write block 279 len 32768 data: 3 4 5 5 4 4 3 3
11271 014/300 Read block 1185 len 8192
11272 015/300 Read block 490 len 24576
11273 016/300 Flush
11274 017/300 Read block 1928 len 40960
11275 018/300 Write block 1720 len 16384 data: 4 3 3 3
11276 019/300 Flush
11277 020/300 Read block 3304 len 8192
11278 021/300 Read block 2307 len 32768
11279 022/300 Read block 2730 len 8192
11280 023/300 Write block 882 len 24576 data: 3 3 4 4 3 3
11281 024/300 Read block 3011 len 36864
11282 025/300 Write block 2480 len 40960 data: 2 2 2 3 3 2 2 3 3 3
11283 026/300 Flush
11284 027/300 Read block 1769 len 20480
11285 028/300 Read block 4424 len 20480
11286 029/300 Flush
11287 030/300 Read block 1235 len 28672
11288 031/300 Write block 4215 len 40960 data: 3 3 4 4 3 3 3 2 2 2
11289 032/300 Read block 1384 len 12288
11290 033/300 Read block 3121 len 8192
11291 034/300 Write block 4781 len 40960 data: 3 4 4 3 4 4 4 4 3 3
11292 035/300 Write block 2424 len 40960 data: 3 3 5 5 5 5 5 5 4 2
11293 036/300 Read block 2053 len 20480
11294 037/300 Write block 4080 len 24576 data: 3 3 4 3 3 2
11295 038/300 Write block 401 len 8192 data: 2 2
11296 039/300 Flush
11297 040/300 Read block 856 len 36864
11298 041/300 Write block 4421 len 24576 data: 7 8 4 5 5 6
11299 042/300 Read block 3046 len 12288
11300 043/300 Write block 309 len 4096 data: 4
11301 044/300 Read block 1097 len 24576
11302 045/300 Write block 4344 len 4096 data: 4
11303 046/300 Write block 2159 len 36864 data: 3 3 3 2 2 2 3 3 3
11304 047/300 Read block 1884 len 36864
11305 048/300 Read block 873 len 8192
11306 049/300 Write block 1367 len 28672 data: 2 2 2 2 2 2 2
11307 050/300 Read block 4968 len 20480
11308 051/300 Read block 3884 len 20480
11309 052/300 Read block 462 len 20480
11310 053/300 Write block 345 len 20480 data: 2 2 2 2 2
11311 054/300 Write block 1181 len 40960 data: 2 3 3 3 3 4 3 3 3 4
11312 055/300 Read block 1506 len 24576
11313 056/300 Write block 2579 len 40960 data: 2 2 2 3 3 4 4 4 3 3
11314 057/300 Read block 4255 len 16384
11315 058/300 Read block 459 len 36864
11316 059/300 Write block 1961 len 40960 data: 3 3 3 3 2 2 2 2 2 3
11317 060/300 Write block 3658 len 36864 data: 4 4 4 4 4 4 3 3 3
11318 061/300 Read block 1212 len 32768
11319 062/300 Read block 2339 len 40960
11320 063/300 Read block 2234 len 36864
11321 064/300 Read block 962 len 20480
11322 065/300 Write block 2881 len 12288 data: 2 3 3
11323 066/300 Read block 694 len 4096
11324 067/300 Read block 1009 len 24576
11325 068/300 Read block 3884 len 40960
11326 069/300 Read block 1862 len 4096
11327 070/300 Write block 1562 len 4096 data: 3
11328 071/300 Write block 1430 len 24576 data: 2 3 3 3 3 3
11329 072/300 Flush
11330 073/300 Read block 2244 len 16384
11331 074/300 Write block 1059 len 20480 data: 4 4 4 4 5
11332 075/300 Write block 3509 len 40960 data: 2 2 2 2 2 2 2 2 2 2
11333 076/300 Read block 758 len 28672
11334 077/300 Read block 4699 len 28672
11335 078/300 Read block 1705 len 36864
11336 079/300 Flush
11337 080/300 Read block 1534 len 40960
11338 081/300 Flush
11339 082/300 Write block 4433 len 36864 data: 3 3 3 3 3 3 3 4 4
11340 083/300 Write block 1934 len 24576 data: 3 3 3 3 3 3
11341 084/300 Write block 2162 len 32768 data: 3 3 3 4 4 4 3 3
11342 085/300 Flush
11343 086/300 Write block 3027 len 4096 data: 4
11344 087/300 Read block 2890 len 20480
11345 088/300 Write block 848 len 36864 data: 4 2 2 2 2 2 4 4 5
11346 089/300 Write block 2615 len 8192 data: 2 2
11347 090/300 Write block 3467 len 16384 data: 2 2 2 2
11348 091/300 Write block 93 len 4096 data: 5
11349 092/300 Read block 3573 len 4096
11350 093/300 Flush
11351 094/300 Read block 4013 len 40960
11352 095/300 Read block 4946 len 8192
11353 096/300 Flush
11354 097/300 Read block 4373 len 28672
11355 098/300 Read block 3350 len 20480
11356 099/300 Write block 3973 len 36864 data: 3 3 3 2 3 3 3 3 3
11357 100/300 Write block 3570 len 24576 data: 3 3 3 3 4 3
11358 101/300 Read block 4914 len 32768
11359 102/300 Write block 2178 len 24576 data: 3 3 3 3 3 3
11360 103/300 Read block 154 len 24576
11361 104/300 Write block 887 len 8192 data: 4 3
11362 105/300 Flush
11363 106/300 Read block 4698 len 32768
11364 107/300 Flush
11365 108/300 Read block 2193 len 28672
11366 109/300 Write block 1102 len 36864 data: 4 4 4 3 3 3 3 2 3
11367 110/300 Write block 1726 len 24576 data: 2 3 2 2 3 3
11368 111/300 Write block 1345 len 20480 data: 3 3 3 3 2
11369 112/300 Write block 1363 len 8192 data: 4 3
11370 113/300 Read block 3121 len 32768
11371 114/300 Read block 4537 len 40960
11372 115/300 Read block 678 len 20480
11373 116/300 Write block 4522 len 40960 data: 2 2 3 3 5 5 6 6 5 5
11374 117/300 Read block 3669 len 12288
11375 118/300 Write block 1476 len 28672 data: 3 3 4 4 4 4 2
11376 119/300 Write block 4391 len 40960 data: 6 5 4 4 4 3 3 4 4 4
11377 120/300 Write block 4711 len 32768 data: 2 2 2 2 3 4 3 3
11378 121/300 Flush
11379 122/300 Read block 3306 len 28672
11380 123/300 Write block 3416 len 40960 data: 2 4 4 4 4 5 6 5 5 4
11381 124/300 Read block 4015 len 24576
11382 125/300 Flush
11383 126/300 Write block 3302 len 12288 data: 6 8 7
11384 127/300 Write block 3593 len 16384 data: 2 2 2 2
11385 128/300 Read block 2352 len 28672
11386 129/300 Read block 2707 len 32768
11387 130/300 Write block 1603 len 20480 data: 3 3 3 3 3
11388 131/300 Write block 811 len 24576 data: 3 4 4 4 4 4
11389 132/300 Write block 1116 len 24576 data: 3 3 3 2 3 4
11390 133/300 Read block 979 len 4096
11391 134/300 Flush
11392 135/300 Flush
11393 136/300 Read block 2073 len 12288
11394 137/300 Write block 4234 len 24576 data: 5 5 3 3 3 2
11395 138/300 Write block 4871 len 8192 data: 2 2
11396 139/300 Read block 2344 len 40960
11397 140/300 Read block 4621 len 8192
11398 141/300 Write block 1918 len 20480 data: 3 2 2 2 2
11399 142/300 Read block 3853 len 28672
11400 143/300 Write block 4595 len 8192 data: 3 3
11401 144/300 Read block 3810 len 40960
11402 145/300 Read block 1027 len 24576
11403 146/300 Write block 4792 len 40960 data: 2 2 4 5 6 6 6 6 3 2
11404 147/300 Write block 3989 len 8192 data: 3 3
11405 148/300 Write block 3629 len 8192 data: 2 3
11406 149/300 Write block 1598 len 12288 data: 3 3 4
11407 150/300 Write block 1668 len 16384 data: 2 4 4 4
11408 151/300 Write block 406 len 36864 data: 2 2 2 3 3 3 4 4 5
11409 152/300 Read block 1990 len 28672
11410 153/300 Read block 1678 len 4096
11411 154/300 Read block 963 len 4096
11412 155/300 Read block 395 len 20480
11413 156/300 Flush
11414 157/300 Write block 1104 len 36864 data: 5 4 4 4 4 3 4 3 4
11415 158/300 Write block 4577 len 12288 data: 2 3 3
11416 159/300 Write block 3600 len 12288 data: 3 3 2
11417 160/300 Read block 4882 len 36864
11418 161/300 Write block 1210 len 28672 data: 4 4 4 3 4 4 3
11419 162/300 Read block 2141 len 24576
11420 163/300 Read block 4779 len 12288
11421 164/300 Flush
11422 165/300 Flush
11423 166/300 Read block 1534 len 16384
11424 167/300 Read block 4944 len 12288
11425 168/300 Read block 323 len 36864
11426 169/300 Write block 2699 len 36864 data: 2 2 2 2 2 2 3 4 4
11427 170/300 Read block 4987 len 8192
11428 171/300 Read block 4326 len 24576
11429 172/300 Read block 3657 len 12288
11430 173/300 Flush
11431 174/300 Read block 4031 len 16384
11432 175/300 Write block 3571 len 16384 data: 4 4 4 5
11433 176/300 Write block 4481 len 16384 data: 4 3 2 3
11434 177/300 Read block 1268 len 8192
11435 178/300 Read block 1790 len 20480
11436 179/300 Read block 360 len 32768
11437 180/300 Read block 3465 len 8192
11438 181/300 Read block 1829 len 28672
11439 182/300 Flush
11440 183/300 Flush
11441 184/300 Write block 4518 len 16384 data: 2 2 2 2
11442 185/300 Write block 4872 len 8192 data: 3 2
11443 186/300 Write block 2914 len 36864 data: 3 3 3 3 3 4 4 4 4
11444 187/300 Read block 2316 len 36864
11445 188/300 Write block 1418 len 20480 data: 4 4 4 4 4
11446 189/300 Read block 680 len 36864
11447 190/300 Read block 2449 len 20480
11448 191/300 Write block 1347 len 12288 data: 4 4 3
11449 192/300 Read block 3748 len 16384
11450 193/300 Write block 4709 len 36864 data: 2 2 3 3 3 3 4 5 4
11451 194/300 Read block 1022 len 36864
11452 195/300 Read block 3439 len 36864
11453 196/300 Flush
11454 197/300 Read block 3406 len 20480
11455 198/300 Read block 3500 len 4096
11456 199/300 Read block 2518 len 28672
11457 200/300 Read block 3104 len 28672
11458 201/300 Read block 2105 len 36864
11459 202/300 Read block 3789 len 36864
11460 203/300 Flush
11461 204/300 Read block 3825 len 28672
11462 205/300 Read block 2514 len 16384
11463 206/300 Write block 1819 len 40960 data: 3 2 2 2 3 3 4 4 4 4
11464 207/300 Read block 3711 len 24576
11465 208/300 Write block 4529 len 32768 data: 7 6 6 4 3 3 2 2
11466 209/300 Read block 3126 len 20480
11467 210/300 Read block 4638 len 32768
11468 211/300 Write block 2406 len 12288 data: 3 3 2
11469 212/300 Write block 1557 len 32768 data: 3 3 3 4 3 4 2 2
11470 213/300 Read block 2848 len 12288
11471 214/300 Write block 3719 len 20480 data: 3 4 4 5 5
11472 215/300 Write block 3493 len 20480 data: 5 4 3 4 4
11473 216/300 Read block 4516 len 40960
11474 217/300 Read block 1572 len 32768
11475 218/300 Read block 2272 len 12288
11476 219/300 Write block 1639 len 4096 data: 2
11477 220/300 Write block 958 len 20480 data: 4 4 4 4 3
11478 221/300 Write block 1139 len 12288 data: 2 2 2
11479 222/300 Write block 3242 len 24576 data: 5 4 5 4 4 3
11480 223/300 Read block 3196 len 40960
11481 224/300 Write block 3627 len 40960 data: 2 2 3 4 3 3 4 4 4 4
11482 225/300 Read block 3838 len 4096
11483 226/300 Read block 1903 len 36864
11484 227/300 Read block 4023 len 32768
11485 228/300 Write block 3497 len 32768 data: 5 3 2 2 2 2 2 2
11486 229/300 Read block 2436 len 20480
11487 230/300 Flush
11488 231/300 Write block 3621 len 40960 data: 4 4 4 3 2 2 3 3 4 5
11489 232/300 Write block 3432 len 24576 data: 2 2 3 3 4 3
11490 233/300 Write block 335 len 40960 data: 5 5 3 3 3 2 2 2 2 2
11491 234/300 Read block 1720 len 36864
11492 235/300 Read block 4722 len 16384
11493 236/300 Read block 4394 len 40960
11494 237/300 Read block 940 len 8192
11495 238/300 Read block 2286 len 20480
11496 239/300 Write block 1649 len 12288 data: 2 2 2
11497 240/300 Read block 402 len 24576
11498 241/300 Read block 2355 len 32768
11499 242/300 Read block 1343 len 20480
11500 243/300 Read block 1992 len 4096
11501 244/300 Read block 1294 len 40960
11502 245/300 Read block 51 len 36864
11503 246/300 Read block 3745 len 12288
11504 247/300 Write block 2517 len 16384 data: 4 4 4 4
11505 248/300 Write block 4288 len 28672 data: 2 2 2 2 2 2 2
11506 249/300 Write block 2615 len 28672 data: 3 3 3 3 4 4 4
11507 250/300 Write block 3155 len 16384 data: 4 5 5 4
11508 251/300 Write block 307 len 8192 data: 4 4
11509 252/300 Read block 2612 len 8192
11510 253/300 Read block 4518 len 36864
11511 254/300 Write block 2769 len 24576 data: 4 5 4 4 4 4
11512 255/300 Write block 1198 len 24576 data: 4 4 4 4 5 4
11513 256/300 Read block 657 len 40960
11514 257/300 Write block 2696 len 12288 data: 2 2 2
11515 258/300 Read block 3948 len 12288
11516 259/300 Write block 2632 len 8192 data: 2 2
11517 260/300 Flush
11518 261/300 Read block 3724 len 16384
11519 262/300 Flush
11520 263/300 Flush
11521 264/300 Read block 3301 len 12288
11522 265/300 Read block 979 len 20480
11523 266/300 Write block 552 len 16384 data: 3 3 3 3
11524 267/300 Read block 3432 len 4096
11525 268/300 Read block 4727 len 4096
11526 269/300 Read block 256 len 4096
11527 270/300 Write block 189 len 32768 data: 4 4 4 3 3 4 3 4
11528 271/300 Read block 726 len 28672
11529 272/300 Write block 2665 len 16384 data: 2 2 2 2
11530 273/300 Write block 2325 len 4096 data: 3
11531 274/300 Write block 1598 len 28672 data: 4 4 5 3 3 4 4
11532 275/300 Write block 4635 len 24576 data: 2 2 2 2 2 2
11533 276/300 Write block 4254 len 4096 data: 2
11534 277/300 Write block 3836 len 24576 data: 3 3 2 2 2 3
11535 278/300 Read block 1814 len 16384
11536 279/300 Read block 2353 len 28672
11537 280/300 Write block 4060 len 4096 data: 3
11538 281/300 Read block 442 len 4096
11539 282/300 Read block 2194 len 4096
11540 283/300 Write block 3328 len 24576 data: 2 2 2 2 2 3
11541 284/300 Write block 3488 len 32768 data: 4 3 4 4 5 6 5 4
11542 285/300 Read block 523 len 28672
11543 286/300 Flush
11544 287/300 Write block 2866 len 28672 data: 2 4 5 5 5 5 4
11545 288/300 Read block 902 len 8192
11546 289/300 Write block 3915 len 36864 data: 2 2 2 2 2 2 2 2 2
11547 290/300 Read block 1695 len 4096
11548 291/300 Flush
11549 292/300 Flush
11550 293/300 Write block 3116 len 40960 data: 3 2 2 2 2 3 3 3 3 3
11551 294/300 Read block 291 len 36864
11552 295/300 Write block 1133 len 12288 data: 2 2 2
11553 296/300 Write block 705 len 12288 data: 5 5 4
11554 297/300 Read block 1924 len 24576
11555 298/300 Read block 2186 len 12288
11556 299/300 Write block 2157 len 32768 data: 3 3 4 4 4 4 4 4
11557 300/300 Read block 2528 len 24576
11558 301/300 Read block 2372 len 8192
11559 Main task has work to do, go find it
11560 got dsc Start(1)
11561 start 1
11562 Feb 25 20:49:11.159 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: c9a33823-f7a2-4f88-b608-1a84cb1a21e2, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
11563 Replay: started 1, returned:Ok(())
11564 ----------------------------------------------------------------
11565 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11566 States: Active Offline Active
11567 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11568 3717 Acked Read 10 Done Sent Done false
11569 3718 Acked Read 9 Done Sent Done false
11570 3719 Acked Write 5 Done Sent Done false
11571 3720 Acked Write 6 Done Sent Done false
11572 3721 Acked Read 5 Done Sent Done false
11573 3722 Acked Read 3 Done Sent Done false
11574 3723 Acked Read 3 Done Sent Done false
11575 3724 Acked Write 10 Done Sent Done false
11576 3725 Acked Write 5 Done Sent Done false
11577 3726 Acked Write 6 Done Sent Done false
11578 3727 Acked Write 4 Done Sent Done false
11579 3728 Acked Read 6 Done Sent Done false
11580 3729 Acked Write 8 Done Sent Done false
11581 3730 Acked Read 2 Done Sent Done false
11582 3731 Acked Read 6 Done Sent Done false
11583 3732 Acked Flush 0 Done Sent Done false
11584 3733 Acked Read 10 Done Sent Done false
11585 3734 Acked Write 4 Done Sent Done false
11586 3735 Acked Flush 0 Done Sent Done false
11587 3736 Acked Read 2 Done Sent Done false
11588 3737 Acked Read 8 Done Sent Done false
11589 3738 Acked Read 2 Done Sent Done false
11590 3739 Acked Write 6 Done Sent Done false
11591 3740 Acked Read 9 Done Sent Done false
11592 3741 Acked Write 10 Done Sent Done false
11593 3742 Acked Flush 0 Done Sent Done false
11594 3743 Acked Read 5 Done Sent Done false
11595 3744 Acked Read 5 Done Sent Done false
11596 3745 Acked Flush 0 Done Sent Done false
11597 3746 Acked Read 7 Done Sent Done false
11598 3747 Acked Write 10 Done Sent Done false
11599 3748 Acked Read 3 Done Sent Done false
11600 3749 Acked Read 2 Done Sent Done false
11601 3750 Acked Write 10 Done Sent Done false
11602 3751 Acked Write 10 Done Sent Done false
11603 3752 Acked Read 5 Done Sent Done false
11604 3753 Acked Write 6 Done Sent Done false
11605 3754 Acked Write 2 Done Sent Done false
11606 3755 Acked Flush 0 Done Sent Done false
11607 3756 Acked Read 9 Done Sent Done false
11608 3757 Acked Write 6 Done Sent Done false
11609 3758 Acked Read 3 Done Sent Done false
11610 3759 Acked Write 1 Done Sent Done false
11611 3760 Acked Read 6 Done Sent Done false
11612 3761 Acked Write 1 Done Sent Done false
11613 3762 Acked Write 9 Done Sent Done false
11614 3763 Acked Read 9 Done Sent Done false
11615 3764 Acked Read 2 Done Sent Done false
11616 3765 Acked Write 7 Done Sent Done false
11617 3766 Acked Read 5 Done Sent Done false
11618 3767 Acked Read 5 Done Sent Done false
11619 3768 Acked Read 5 Done Sent Done false
11620 3769 Acked Write 5 Done Sent Done false
11621 3770 Acked Write 10 Done Sent Done false
11622 3771 Acked Read 6 Done Sent Done false
11623 3772 Acked Write 10 Done Sent Done false
11624 3773 Acked Read 4 Done Sent Done false
11625 3774 Acked Read 9 Done Sent Done false
11626 3775 Acked Write 10 Done Sent Done false
11627 3776 Acked Write 9 Done Sent Done false
11628 3777 Acked Read 8 Done Sent Done false
11629 3778 Acked Read 10 Done Sent Done false
11630 3779 Acked Read 9 Done Sent Done false
11631 3780 Acked Read 5 Done Sent Done false
11632 3781 Acked Write 3 Done Sent Done false
11633 3782 Acked Read 1 Done Sent Done false
11634 3783 Acked Read 6 Done Sent Done false
11635 3784 Acked Read 10 Done Sent Done false
11636 3785 Acked Read 1 Done Sent Done false
11637 3786 Acked Write 1 Done Sent Done false
11638 3787 Acked Write 6 Done Sent Done false
11639 3788 Acked Flush 0 Done Sent Done false
11640 3789 Acked Read 4 Done Sent Done false
11641 3790 Acked Write 5 Done Sent Done false
11642 3791 Acked Write 10 Done Sent Done false
11643 3792 Acked Read 7 Done Sent Done false
11644 3793 Acked Read 7 Done Sent Done false
11645 3794 Acked Read 9 Done Sent Done false
11646 3795 Acked Flush 0 Done Sent Done false
11647 3796 Acked Read 10 Done Sent Done false
11648 3797 Acked Flush 0 Done Sent Done false
11649 3798 Acked Write 9 Done Sent Done false
11650 3799 Acked Write 6 Done Sent Done false
11651 3800 Acked Write 8 Done Sent Done false
11652 3801 Acked Flush 0 Done Sent Done false
11653 3802 Acked Write 1 Done Sent Done false
11654 3803 Acked Read 5 Done Sent Done false
11655 3804 Acked Write 9 Done Sent Done false
11656 3805 Acked Write 2 Done Sent Done false
11657 3806 Acked Write 4 Done Sent Done false
11658 3807 Acked Write 1 Done Sent Done false
11659 3808 Acked Read 1 Done Sent Done false
11660 3809 Acked Flush 0 Done Sent Done false
11661 3810 Acked Read 10 Done Sent Done false
11662 3811 Acked Read 2 Done Sent Done false
11663 3812 Acked Flush 0 Done Sent Done false
11664 3813 Acked Read 7 Done Sent Done false
11665 3814 Acked Read 5 Done Sent Done false
11666 3815 Acked Write 9 Done Sent Done false
11667 3816 Acked Write 6 Done Sent Done false
11668 3817 Acked Read 8 Done Sent Done false
11669 3818 Acked Write 6 Done Sent Done false
11670 3819 Acked Read 6 Done Sent Done false
11671 3820 Acked Write 2 Done Sent Done false
11672 3821 Acked Flush 0 Done Sent Done false
11673 3822 Acked Read 8 Done Sent Done false
11674 3823 Acked Flush 0 Done Sent Done false
11675 3824 Acked Read 7 Done Sent Done false
11676 3825 Acked Write 9 Done Sent Done false
11677 3826 Acked Write 6 Done Sent Done false
11678 3827 Acked Write 5 Done Sent Done false
11679 3828 Acked Write 2 Done Sent Done false
11680 3829 Acked Read 8 Done Sent Done false
11681 3830 Acked Read 10 Done Sent Done false
11682 3831 Acked Read 5 Done Sent Done false
11683 3832 Acked Write 10 Done Sent Done false
11684 3833 Acked Read 3 Done Sent Done false
11685 3834 Acked Write 7 Done Sent Done false
11686 3835 Acked Write 10 Done Sent Done false
11687 3836 Acked Write 8 Done Sent Done false
11688 3837 Acked Flush 0 Done Sent Done false
11689 3838 Acked Read 7 Done Sent Done false
11690 3839 Acked Write 10 Done Sent Done false
11691 3840 Acked Read 6 Done Sent Done false
11692 3841 Acked Flush 0 Done Sent Done false
11693 3842 Acked Write 3 Done Sent Done false
11694 3843 Acked Write 4 Done Sent Done false
11695 3844 Acked Read 7 Done Sent Done false
11696 3845 Acked Read 8 Done Sent Done false
11697 3846 Acked Write 5 Done Sent Done false
11698 3847 Acked Write 6 Done Sent Done false
11699 3848 Acked Write 6 Done Sent Done false
11700 3849 Acked Read 1 Done Sent Done false
11701 3850 Acked Flush 0 Done Sent Done false
11702 3851 Acked Flush 0 Done Sent Done false
11703 3852 Acked Read 3 Done Sent Done false
11704 3853 Acked Write 6 Done Sent Done false
11705 3854 Acked Write 2 Done Sent Done false
11706 3855 Acked Read 10 Done Sent Done false
11707 3856 Acked Read 2 Done Sent Done false
11708 3857 Acked Write 5 Done Sent Done false
11709 3858 Acked Read 7 Done Sent Done false
11710 3859 Acked Write 2 Done Sent Done false
11711 3860 Acked Read 10 Done Sent Done false
11712 3861 Acked Read 6 Done Sent Done false
11713 3862 Acked Write 10 Done Sent Done false
11714 3863 Acked Write 2 Done Sent Done false
11715 3864 Acked Write 2 Done Sent Done false
11716 3865 Acked Write 3 Done Sent Done false
11717 3866 Acked Write 4 Done Sent Done false
11718 3867 Acked Write 9 Done Sent Done false
11719 3868 Acked Read 7 Done Sent Done false
11720 3869 Acked Read 1 Done Sent Done false
11721 3870 Acked Read 1 Done Sent Done false
11722 3871 Acked Read 5 Done Sent Done false
11723 3872 Acked Flush 0 Done Sent Done false
11724 3873 Acked Write 9 Done Sent Done false
11725 3874 Acked Write 3 Done Sent Done false
11726 3875 Acked Write 3 Done Sent Done false
11727 3876 Acked Read 9 Done Sent Done false
11728 3877 Acked Write 7 Done Sent Done false
11729 3878 Acked Read 6 Done Sent Done false
11730 3879 Acked Read 3 Done Sent Done false
11731 3880 Acked Flush 0 Done Sent Done false
11732 3881 Acked Flush 0 Done Sent Done false
11733 3882 Acked Read 4 Done Sent Done false
11734 3883 Acked Read 3 Done Sent Done false
11735 3884 Acked Read 9 Done Sent Done false
11736 3885 Acked Write 9 Done Sent Done false
11737 3886 Acked Read 2 Done Sent Done false
11738 3887 Acked Read 6 Done Sent Done false
11739 3888 Acked Read 3 Done Sent Done false
11740 3889 Acked Flush 0 Done Sent Done false
11741 3890 Acked Read 4 Done Sent Done false
11742 3891 Acked Write 4 Done Sent Done false
11743 3892 Acked Write 4 Done Sent Done false
11744 3893 Acked Read 2 Done Sent Done false
11745 3894 Acked Read 5 Done Sent Done false
11746 3895 Acked Read 8 Done Sent Done false
11747 3896 Acked Read 2 Done Sent Done false
11748 3897 Acked Read 7 Done Sent Done false
11749 3898 Acked Flush 0 Done Sent Done false
11750 3899 Acked Flush 0 Done Sent Done false
11751 3900 Acked Write 4 Done Sent Done false
11752 3901 Acked Write 2 Done Sent Done false
11753 3902 Acked Write 9 Done Sent Done false
11754 3903 Acked Read 9 Done Sent Done false
11755 3904 Acked Write 5 Done Sent Done false
11756 3905 Acked Read 9 Done Sent Done false
11757 3906 Acked Read 5 Done Sent Done false
11758 3907 Acked Write 3 Done Sent Done false
11759 3908 Acked Read 4 Done Sent Done false
11760 3909 Acked Write 9 Done Sent Done false
11761 3910 Acked Read 9 Done Sent Done false
11762 3911 Acked Read 9 Done Sent Done false
11763 3912 Acked Flush 0 Done Sent Done false
11764 3913 Acked Read 5 Done Sent Done false
11765 3914 Acked Read 1 Done Sent Done false
11766 3915 Acked Read 7 Done Sent Done false
11767 3916 Acked Read 7 Done Sent Done false
11768 3917 Acked Read 9 Done Sent Done false
11769 3918 Acked Read 9 Done Sent Done false
11770 3919 Acked Flush 0 Done Sent Done false
11771 3920 Acked Read 7 Done Sent Done false
11772 3921 Acked Read 4 Done Sent Done false
11773 3922 Acked Write 10 Done Sent Done false
11774 3923 Acked Read 6 Done Sent Done false
11775 3924 Acked Write 8 Done Sent Done false
11776 3925 Acked Read 5 Done Sent Done false
11777 3926 Acked Read 8 Done Sent Done false
11778 3927 Acked Write 3 Done Sent Done false
11779 3928 Acked Write 8 Done Sent Done false
11780 3929 Acked Read 3 Done Sent Done false
11781 3930 Acked Write 5 Done Sent Done false
11782 3931 Acked Write 5 Done Sent Done false
11783 3932 Acked Read 10 Done Sent Done false
11784 3933 Acked Read 8 Done Sent Done false
11785 3934 Acked Read 3 Done Sent Done false
11786 3935 Acked Write 1 Done Sent Done false
11787 3936 Acked Write 5 Done Sent Done false
11788 3937 Acked Write 3 Done Sent Done false
11789 3938 Acked Write 6 Done Sent Done false
11790 3939 Acked Read 10 Done Sent Done false
11791 3940 Acked Write 10 Done Sent Done false
11792 3941 Acked Read 1 Done Sent Done false
11793 3942 Acked Read 9 Done Sent Done false
11794 3943 Acked Read 8 Done Sent Done false
11795 3944 Acked Write 8 Done Sent Done false
11796 3945 Acked Read 5 Done Sent Done false
11797 3946 Acked Flush 0 Done Sent Done false
11798 3947 Acked Write 10 Done Sent Done false
11799 3948 Acked Write 6 Done Sent Done false
11800 3949 Acked Write 10 Done Sent Done false
11801 3950 Acked Read 9 Done Sent Done false
11802 3951 Acked Read 4 Done Sent Done false
11803 3952 Acked Read 10 Done Sent Done false
11804 3953 Acked Read 2 Done Sent Done false
11805 3954 Acked Read 5 Done Sent Done false
11806 3955 Acked Write 3 Done Sent Done false
11807 3956 Acked Read 6 Done Sent Done false
11808 3957 Acked Read 8 Done Sent Done false
11809 3958 Acked Read 5 Done Sent Done false
11810 3959 Acked Read 1 Done Sent Done false
11811 3960 Acked Read 10 Done Sent Done false
11812 3961 Acked Read 9 Done Sent Done false
11813 3962 Acked Read 3 Done Sent Done false
11814 3963 Acked Write 4 Done Sent Done false
11815 3964 Acked Write 7 Done Sent Done false
11816 3965 Acked Write 7 Done Sent Done false
11817 3966 Acked Write 4 Done Sent Done false
11818 3967 Acked Write 2 Done Sent Done false
11819 3968 Acked Read 2 Done Sent Done false
11820 3969 Acked Read 9 Done Sent Done false
11821 3970 Acked Write 6 Done Sent Done false
11822 3971 Acked Write 6 Done Sent Done false
11823 3972 Acked Read 10 Done Sent Done false
11824 3973 Acked Write 3 Done Sent Done false
11825 3974 Acked Read 3 Done Sent Done false
11826 3975 Acked Write 2 Done Sent Done false
11827 3976 Acked Flush 0 Done Sent Done false
11828 3977 Acked Read 4 Done Sent Done false
11829 3978 Acked Flush 0 Done Sent Done false
11830 3979 Acked Flush 0 Done Sent Done false
11831 3980 Acked Read 3 Done Sent Done false
11832 3981 Acked Read 5 Done Sent Done false
11833 3982 Acked Write 4 Done Sent Done false
11834 3983 Acked Read 1 Done Sent Done false
11835 3984 Acked Read 1 Done Sent Done false
11836 3985 Acked Read 1 Done Sent Done false
11837 3986 Acked Write 8 Done Sent Done false
11838 3987 Acked Read 7 Done Sent Done false
11839 3988 Acked Write 4 Done Sent Done false
11840 3989 Acked Write 1 Done Sent Done false
11841 3990 Acked Write 7 Done Sent Done false
11842 3991 Acked Write 6 Done Sent Done false
11843 3992 Acked Write 1 Done Sent Done false
11844 3993 Acked Write 6 Done Sent Done false
11845 3994 Acked Read 4 Done Sent Done false
11846 3995 Acked Read 7 Done Sent Done false
11847 3996 Acked Write 1 Done Sent Done false
11848 3997 Acked Read 1 Done Sent Done false
11849 3998 Acked Read 1 Done Sent Done false
11850 3999 Acked Write 6 Done Sent Done false
11851 4000 Acked Write 8 Done Sent Done false
11852 4001 Acked Read 7 Done Sent Done false
11853 4002 Acked Flush 0 Done Sent Done false
11854 4003 Acked Write 7 Done Sent Done false
11855 4004 Acked Read 2 Done Sent Done false
11856 4005 Acked Write 9 Done Sent Done false
11857 4006 Acked Read 1 Done Sent Done false
11858 4007 Acked Flush 0 Done Sent Done false
11859 4008 Acked Flush 0 Done Sent Done false
11860 4009 Acked Write 10 Done Sent Done false
11861 4010 Acked Read 9 Done Sent Done false
11862 4011 Acked Write 3 Done Sent Done false
11863 4012 Acked Write 3 Done Sent Done false
11864 4013 Acked Read 6 Done Sent Done false
11865 4014 Acked Read 3 Done Sent Done false
11866 4015 Acked Write 8 Done Sent Done false
11867 4016 Acked Read 6 Done Sent Done false
11868 4017 Acked Read 2 Done Sent Done false
11869 STATES DS:0 DS:1 DS:2 TOTAL
11870 Sent 0 301 0 301
11871 Done 301 0 301 602
11872 Skipped 0 0 0 0
11873 Error 0 0 0 0
11874 Last Flush: Some(JobId(4008)) Some(JobId(3716)) Some(JobId(4008))
11875 Downstairs last five completed: 3716 3715 3714 3713 3712
11876 Upstairs last five completed: 4017 4016 4015 4014 4013
11877 CLIENT: Up:0 ds:301 act:2
118782025-02-25T20:49:11.661ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
118792025-02-25T20:49:12.305ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
118802025-02-25T20:49:13.307ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
118812025-02-25T20:49:14.309ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
11882 ----------------------------------------------------------------
11883 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11884 States: Active Offline Active
11885 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11886 3717 Acked Read 10 Done Sent Done false
11887 3718 Acked Read 9 Done Sent Done false
11888 3719 Acked Write 5 Done Sent Done false
11889 3720 Acked Write 6 Done Sent Done false
11890 3721 Acked Read 5 Done Sent Done false
11891 3722 Acked Read 3 Done Sent Done false
11892 3723 Acked Read 3 Done Sent Done false
11893 3724 Acked Write 10 Done Sent Done false
11894 3725 Acked Write 5 Done Sent Done false
11895 3726 Acked Write 6 Done Sent Done false
11896 3727 Acked Write 4 Done Sent Done false
11897 3728 Acked Read 6 Done Sent Done false
11898 3729 Acked Write 8 Done Sent Done false
11899 3730 Acked Read 2 Done Sent Done false
11900 3731 Acked Read 6 Done Sent Done false
11901 3732 Acked Flush 0 Done Sent Done false
11902 3733 Acked Read 10 Done Sent Done false
11903 3734 Acked Write 4 Done Sent Done false
11904 3735 Acked Flush 0 Done Sent Done false
11905 3736 Acked Read 2 Done Sent Done false
11906 3737 Acked Read 8 Done Sent Done false
11907 3738 Acked Read 2 Done Sent Done false
11908 3739 Acked Write 6 Done Sent Done false
11909 3740 Acked Read 9 Done Sent Done false
11910 3741 Acked Write 10 Done Sent Done false
11911 3742 Acked Flush 0 Done Sent Done false
11912 3743 Acked Read 5 Done Sent Done false
11913 3744 Acked Read 5 Done Sent Done false
11914 3745 Acked Flush 0 Done Sent Done false
11915 3746 Acked Read 7 Done Sent Done false
11916 3747 Acked Write 10 Done Sent Done false
11917 3748 Acked Read 3 Done Sent Done false
11918 3749 Acked Read 2 Done Sent Done false
11919 3750 Acked Write 10 Done Sent Done false
11920 3751 Acked Write 10 Done Sent Done false
11921 3752 Acked Read 5 Done Sent Done false
11922 3753 Acked Write 6 Done Sent Done false
11923 3754 Acked Write 2 Done Sent Done false
11924 3755 Acked Flush 0 Done Sent Done false
11925 3756 Acked Read 9 Done Sent Done false
11926 3757 Acked Write 6 Done Sent Done false
11927 3758 Acked Read 3 Done Sent Done false
11928 3759 Acked Write 1 Done Sent Done false
11929 3760 Acked Read 6 Done Sent Done false
11930 3761 Acked Write 1 Done Sent Done false
11931 3762 Acked Write 9 Done Sent Done false
11932 3763 Acked Read 9 Done Sent Done false
11933 3764 Acked Read 2 Done Sent Done false
11934 3765 Acked Write 7 Done Sent Done false
11935 3766 Acked Read 5 Done Sent Done false
11936 3767 Acked Read 5 Done Sent Done false
11937 3768 Acked Read 5 Done Sent Done false
11938 3769 Acked Write 5 Done Sent Done false
11939 3770 Acked Write 10 Done Sent Done false
11940 3771 Acked Read 6 Done Sent Done false
11941 3772 Acked Write 10 Done Sent Done false
11942 3773 Acked Read 4 Done Sent Done false
11943 3774 Acked Read 9 Done Sent Done false
11944 3775 Acked Write 10 Done Sent Done false
11945 3776 Acked Write 9 Done Sent Done false
11946 3777 Acked Read 8 Done Sent Done false
11947 3778 Acked Read 10 Done Sent Done false
11948 3779 Acked Read 9 Done Sent Done false
11949 3780 Acked Read 5 Done Sent Done false
11950 3781 Acked Write 3 Done Sent Done false
11951 3782 Acked Read 1 Done Sent Done false
11952 3783 Acked Read 6 Done Sent Done false
11953 3784 Acked Read 10 Done Sent Done false
11954 3785 Acked Read 1 Done Sent Done false
11955 3786 Acked Write 1 Done Sent Done false
11956 3787 Acked Write 6 Done Sent Done false
11957 3788 Acked Flush 0 Done Sent Done false
11958 3789 Acked Read 4 Done Sent Done false
11959 3790 Acked Write 5 Done Sent Done false
11960 3791 Acked Write 10 Done Sent Done false
11961 3792 Acked Read 7 Done Sent Done false
11962 3793 Acked Read 7 Done Sent Done false
11963 3794 Acked Read 9 Done Sent Done false
11964 3795 Acked Flush 0 Done Sent Done false
11965 3796 Acked Read 10 Done Sent Done false
11966 3797 Acked Flush 0 Done Sent Done false
11967 3798 Acked Write 9 Done Sent Done false
11968 3799 Acked Write 6 Done Sent Done false
11969 3800 Acked Write 8 Done Sent Done false
11970 3801 Acked Flush 0 Done Sent Done false
11971 3802 Acked Write 1 Done Sent Done false
11972 3803 Acked Read 5 Done Sent Done false
11973 3804 Acked Write 9 Done Sent Done false
11974 3805 Acked Write 2 Done Sent Done false
11975 3806 Acked Write 4 Done Sent Done false
11976 3807 Acked Write 1 Done Sent Done false
11977 3808 Acked Read 1 Done Sent Done false
11978 3809 Acked Flush 0 Done Sent Done false
11979 3810 Acked Read 10 Done Sent Done false
11980 3811 Acked Read 2 Done Sent Done false
11981 3812 Acked Flush 0 Done Sent Done false
11982 3813 Acked Read 7 Done Sent Done false
11983 3814 Acked Read 5 Done Sent Done false
11984 3815 Acked Write 9 Done Sent Done false
11985 3816 Acked Write 6 Done Sent Done false
11986 3817 Acked Read 8 Done Sent Done false
11987 3818 Acked Write 6 Done Sent Done false
11988 3819 Acked Read 6 Done Sent Done false
11989 3820 Acked Write 2 Done Sent Done false
11990 3821 Acked Flush 0 Done Sent Done false
11991 3822 Acked Read 8 Done Sent Done false
11992 3823 Acked Flush 0 Done Sent Done false
11993 3824 Acked Read 7 Done Sent Done false
11994 3825 Acked Write 9 Done Sent Done false
11995 3826 Acked Write 6 Done Sent Done false
11996 3827 Acked Write 5 Done Sent Done false
11997 3828 Acked Write 2 Done Sent Done false
11998 3829 Acked Read 8 Done Sent Done false
11999 3830 Acked Read 10 Done Sent Done false
12000 3831 Acked Read 5 Done Sent Done false
12001 3832 Acked Write 10 Done Sent Done false
12002 3833 Acked Read 3 Done Sent Done false
12003 3834 Acked Write 7 Done Sent Done false
12004 3835 Acked Write 10 Done Sent Done false
12005 3836 Acked Write 8 Done Sent Done false
12006 3837 Acked Flush 0 Done Sent Done false
12007 3838 Acked Read 7 Done Sent Done false
12008 3839 Acked Write 10 Done Sent Done false
12009 3840 Acked Read 6 Done Sent Done false
12010 3841 Acked Flush 0 Done Sent Done false
12011 3842 Acked Write 3 Done Sent Done false
12012 3843 Acked Write 4 Done Sent Done false
12013 3844 Acked Read 7 Done Sent Done false
12014 3845 Acked Read 8 Done Sent Done false
12015 3846 Acked Write 5 Done Sent Done false
12016 3847 Acked Write 6 Done Sent Done false
12017 3848 Acked Write 6 Done Sent Done false
12018 3849 Acked Read 1 Done Sent Done false
12019 3850 Acked Flush 0 Done Sent Done false
12020 3851 Acked Flush 0 Done Sent Done false
12021 3852 Acked Read 3 Done Sent Done false
12022 3853 Acked Write 6 Done Sent Done false
12023 3854 Acked Write 2 Done Sent Done false
12024 3855 Acked Read 10 Done Sent Done false
12025 3856 Acked Read 2 Done Sent Done false
12026 3857 Acked Write 5 Done Sent Done false
12027 3858 Acked Read 7 Done Sent Done false
12028 3859 Acked Write 2 Done Sent Done false
12029 3860 Acked Read 10 Done Sent Done false
12030 3861 Acked Read 6 Done Sent Done false
12031 3862 Acked Write 10 Done Sent Done false
12032 3863 Acked Write 2 Done Sent Done false
12033 3864 Acked Write 2 Done Sent Done false
12034 3865 Acked Write 3 Done Sent Done false
12035 3866 Acked Write 4 Done Sent Done false
12036 3867 Acked Write 9 Done Sent Done false
12037 3868 Acked Read 7 Done Sent Done false
12038 3869 Acked Read 1 Done Sent Done false
12039 3870 Acked Read 1 Done Sent Done false
12040 3871 Acked Read 5 Done Sent Done false
12041 3872 Acked Flush 0 Done Sent Done false
12042 3873 Acked Write 9 Done Sent Done false
12043 3874 Acked Write 3 Done Sent Done false
12044 3875 Acked Write 3 Done Sent Done false
12045 3876 Acked Read 9 Done Sent Done false
12046 3877 Acked Write 7 Done Sent Done false
12047 3878 Acked Read 6 Done Sent Done false
12048 3879 Acked Read 3 Done Sent Done false
12049 3880 Acked Flush 0 Done Sent Done false
12050 3881 Acked Flush 0 Done Sent Done false
12051 3882 Acked Read 4 Done Sent Done false
12052 3883 Acked Read 3 Done Sent Done false
12053 3884 Acked Read 9 Done Sent Done false
12054 3885 Acked Write 9 Done Sent Done false
12055 3886 Acked Read 2 Done Sent Done false
12056 3887 Acked Read 6 Done Sent Done false
12057 3888 Acked Read 3 Done Sent Done false
12058 3889 Acked Flush 0 Done Sent Done false
12059 3890 Acked Read 4 Done Sent Done false
12060 3891 Acked Write 4 Done Sent Done false
12061 3892 Acked Write 4 Done Sent Done false
12062 3893 Acked Read 2 Done Sent Done false
12063 3894 Acked Read 5 Done Sent Done false
12064 3895 Acked Read 8 Done Sent Done false
12065 3896 Acked Read 2 Done Sent Done false
12066 3897 Acked Read 7 Done Sent Done false
12067 3898 Acked Flush 0 Done Sent Done false
12068 3899 Acked Flush 0 Done Sent Done false
12069 3900 Acked Write 4 Done Sent Done false
12070 3901 Acked Write 2 Done Sent Done false
12071 3902 Acked Write 9 Done Sent Done false
12072 3903 Acked Read 9 Done Sent Done false
12073 3904 Acked Write 5 Done Sent Done false
12074 3905 Acked Read 9 Done Sent Done false
12075 3906 Acked Read 5 Done Sent Done false
12076 3907 Acked Write 3 Done Sent Done false
12077 3908 Acked Read 4 Done Sent Done false
12078 3909 Acked Write 9 Done Sent Done false
12079 3910 Acked Read 9 Done Sent Done false
12080 3911 Acked Read 9 Done Sent Done false
12081 3912 Acked Flush 0 Done Sent Done false
12082 3913 Acked Read 5 Done Sent Done false
12083 3914 Acked Read 1 Done Sent Done false
12084 3915 Acked Read 7 Done Sent Done false
12085 3916 Acked Read 7 Done Sent Done false
12086 3917 Acked Read 9 Done Sent Done false
12087 3918 Acked Read 9 Done Sent Done false
12088 3919 Acked Flush 0 Done Sent Done false
12089 3920 Acked Read 7 Done Sent Done false
12090 3921 Acked Read 4 Done Sent Done false
12091 3922 Acked Write 10 Done Sent Done false
12092 3923 Acked Read 6 Done Sent Done false
12093 3924 Acked Write 8 Done Sent Done false
12094 3925 Acked Read 5 Done Sent Done false
12095 3926 Acked Read 8 Done Sent Done false
12096 3927 Acked Write 3 Done Sent Done false
12097 3928 Acked Write 8 Done Sent Done false
12098 3929 Acked Read 3 Done Sent Done false
12099 3930 Acked Write 5 Done Sent Done false
12100 3931 Acked Write 5 Done Sent Done false
12101 3932 Acked Read 10 Done Sent Done false
12102 3933 Acked Read 8 Done Sent Done false
12103 3934 Acked Read 3 Done Sent Done false
12104 3935 Acked Write 1 Done Sent Done false
12105 3936 Acked Write 5 Done Sent Done false
12106 3937 Acked Write 3 Done Sent Done false
12107 3938 Acked Write 6 Done Sent Done false
12108 3939 Acked Read 10 Done Sent Done false
12109 3940 Acked Write 10 Done Sent Done false
12110 3941 Acked Read 1 Done Sent Done false
12111 3942 Acked Read 9 Done Sent Done false
12112 3943 Acked Read 8 Done Sent Done false
12113 3944 Acked Write 8 Done Sent Done false
12114 3945 Acked Read 5 Done Sent Done false
12115 3946 Acked Flush 0 Done Sent Done false
12116 3947 Acked Write 10 Done Sent Done false
12117 3948 Acked Write 6 Done Sent Done false
12118 3949 Acked Write 10 Done Sent Done false
12119 3950 Acked Read 9 Done Sent Done false
12120 3951 Acked Read 4 Done Sent Done false
12121 3952 Acked Read 10 Done Sent Done false
12122 3953 Acked Read 2 Done Sent Done false
12123 3954 Acked Read 5 Done Sent Done false
12124 3955 Acked Write 3 Done Sent Done false
12125 3956 Acked Read 6 Done Sent Done false
12126 3957 Acked Read 8 Done Sent Done false
12127 3958 Acked Read 5 Done Sent Done false
12128 3959 Acked Read 1 Done Sent Done false
12129 3960 Acked Read 10 Done Sent Done false
12130 3961 Acked Read 9 Done Sent Done false
12131 3962 Acked Read 3 Done Sent Done false
12132 3963 Acked Write 4 Done Sent Done false
12133 3964 Acked Write 7 Done Sent Done false
12134 3965 Acked Write 7 Done Sent Done false
12135 3966 Acked Write 4 Done Sent Done false
12136 3967 Acked Write 2 Done Sent Done false
12137 3968 Acked Read 2 Done Sent Done false
12138 3969 Acked Read 9 Done Sent Done false
12139 3970 Acked Write 6 Done Sent Done false
12140 3971 Acked Write 6 Done Sent Done false
12141 3972 Acked Read 10 Done Sent Done false
12142 3973 Acked Write 3 Done Sent Done false
12143 3974 Acked Read 3 Done Sent Done false
12144 3975 Acked Write 2 Done Sent Done false
12145 3976 Acked Flush 0 Done Sent Done false
12146 3977 Acked Read 4 Done Sent Done false
12147 3978 Acked Flush 0 Done Sent Done false
12148 3979 Acked Flush 0 Done Sent Done false
12149 3980 Acked Read 3 Done Sent Done false
12150 3981 Acked Read 5 Done Sent Done false
12151 3982 Acked Write 4 Done Sent Done false
12152 3983 Acked Read 1 Done Sent Done false
12153 3984 Acked Read 1 Done Sent Done false
12154 3985 Acked Read 1 Done Sent Done false
12155 3986 Acked Write 8 Done Sent Done false
12156 3987 Acked Read 7 Done Sent Done false
12157 3988 Acked Write 4 Done Sent Done false
12158 3989 Acked Write 1 Done Sent Done false
12159 3990 Acked Write 7 Done Sent Done false
12160 3991 Acked Write 6 Done Sent Done false
12161 3992 Acked Write 1 Done Sent Done false
12162 3993 Acked Write 6 Done Sent Done false
12163 3994 Acked Read 4 Done Sent Done false
12164 3995 Acked Read 7 Done Sent Done false
12165 3996 Acked Write 1 Done Sent Done false
12166 3997 Acked Read 1 Done Sent Done false
12167 3998 Acked Read 1 Done Sent Done false
12168 3999 Acked Write 6 Done Sent Done false
12169 4000 Acked Write 8 Done Sent Done false
12170 4001 Acked Read 7 Done Sent Done false
12171 4002 Acked Flush 0 Done Sent Done false
12172 4003 Acked Write 7 Done Sent Done false
12173 4004 Acked Read 2 Done Sent Done false
12174 4005 Acked Write 9 Done Sent Done false
12175 4006 Acked Read 1 Done Sent Done false
12176 4007 Acked Flush 0 Done Sent Done false
12177 4008 Acked Flush 0 Done Sent Done false
12178 4009 Acked Write 10 Done Sent Done false
12179 4010 Acked Read 9 Done Sent Done false
12180 4011 Acked Write 3 Done Sent Done false
12181 4012 Acked Write 3 Done Sent Done false
12182 4013 Acked Read 6 Done Sent Done false
12183 4014 Acked Read 3 Done Sent Done false
12184 4015 Acked Write 8 Done Sent Done false
12185 4016 Acked Read 6 Done Sent Done false
12186 4017 Acked Read 2 Done Sent Done false
12187 STATES DS:0 DS:1 DS:2 TOTAL
12188 Sent 0 301 0 301
12189 Done 301 0 301 602
12190 Skipped 0 0 0 0
12191 Error 0 0 0 0
12192 Last Flush: Some(JobId(4008)) Some(JobId(3716)) Some(JobId(4008))
12193 Downstairs last five completed: 3716 3715 3714 3713 3712
12194 Upstairs last five completed: 4017 4016 4015 4014 4013
12195 CLIENT: Up:0 ds:301 act:2
121962025-02-25T20:49:16.312ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
121972025-02-25T20:49:16.910ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
121982025-02-25T20:49:16.910ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
121992025-02-25T20:49:16.910ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
122002025-02-25T20:49:16.910ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
122012025-02-25T20:49:16.910ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
122022025-02-25T20:49:17.411ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
122032025-02-25T20:49:18.317ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
12204 ----------------------------------------------------------------
12205 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12206 States: Active Offline Active
12207 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12208 3717 Acked Read 10 Done Sent Done false
12209 3718 Acked Read 9 Done Sent Done false
12210 3719 Acked Write 5 Done Sent Done false
12211 3720 Acked Write 6 Done Sent Done false
12212 3721 Acked Read 5 Done Sent Done false
12213 3722 Acked Read 3 Done Sent Done false
12214 3723 Acked Read 3 Done Sent Done false
12215 3724 Acked Write 10 Done Sent Done false
12216 3725 Acked Write 5 Done Sent Done false
12217 3726 Acked Write 6 Done Sent Done false
12218 3727 Acked Write 4 Done Sent Done false
12219 3728 Acked Read 6 Done Sent Done false
12220 3729 Acked Write 8 Done Sent Done false
12221 3730 Acked Read 2 Done Sent Done false
12222 3731 Acked Read 6 Done Sent Done false
12223 3732 Acked Flush 0 Done Sent Done false
12224 3733 Acked Read 10 Done Sent Done false
12225 3734 Acked Write 4 Done Sent Done false
12226 3735 Acked Flush 0 Done Sent Done false
12227 3736 Acked Read 2 Done Sent Done false
12228 3737 Acked Read 8 Done Sent Done false
12229 3738 Acked Read 2 Done Sent Done false
12230 3739 Acked Write 6 Done Sent Done false
12231 3740 Acked Read 9 Done Sent Done false
12232 3741 Acked Write 10 Done Sent Done false
12233 3742 Acked Flush 0 Done Sent Done false
12234 3743 Acked Read 5 Done Sent Done false
12235 3744 Acked Read 5 Done Sent Done false
12236 3745 Acked Flush 0 Done Sent Done false
12237 3746 Acked Read 7 Done Sent Done false
12238 3747 Acked Write 10 Done Sent Done false
12239 3748 Acked Read 3 Done Sent Done false
12240 3749 Acked Read 2 Done Sent Done false
12241 3750 Acked Write 10 Done Sent Done false
12242 3751 Acked Write 10 Done Sent Done false
12243 3752 Acked Read 5 Done Sent Done false
12244 3753 Acked Write 6 Done Sent Done false
12245 3754 Acked Write 2 Done Sent Done false
12246 3755 Acked Flush 0 Done Sent Done false
12247 3756 Acked Read 9 Done Sent Done false
12248 3757 Acked Write 6 Done Sent Done false
12249 3758 Acked Read 3 Done Sent Done false
12250 3759 Acked Write 1 Done Sent Done false
12251 3760 Acked Read 6 Done Sent Done false
12252 3761 Acked Write 1 Done Sent Done false
12253 3762 Acked Write 9 Done Sent Done false
12254 3763 Acked Read 9 Done Sent Done false
12255 3764 Acked Read 2 Done Sent Done false
12256 3765 Acked Write 7 Done Sent Done false
12257 3766 Acked Read 5 Done Sent Done false
12258 3767 Acked Read 5 Done Sent Done false
12259 3768 Acked Read 5 Done Sent Done false
12260 3769 Acked Write 5 Done Sent Done false
12261 3770 Acked Write 10 Done Sent Done false
12262 3771 Acked Read 6 Done Sent Done false
12263 3772 Acked Write 10 Done Sent Done false
12264 3773 Acked Read 4 Done Sent Done false
12265 3774 Acked Read 9 Done Sent Done false
12266 3775 Acked Write 10 Done Sent Done false
12267 3776 Acked Write 9 Done Sent Done false
12268 3777 Acked Read 8 Done Sent Done false
12269 3778 Acked Read 10 Done Sent Done false
12270 3779 Acked Read 9 Done Sent Done false
12271 3780 Acked Read 5 Done Sent Done false
12272 3781 Acked Write 3 Done Sent Done false
12273 3782 Acked Read 1 Done Sent Done false
12274 3783 Acked Read 6 Done Sent Done false
12275 3784 Acked Read 10 Done Sent Done false
12276 3785 Acked Read 1 Done Sent Done false
12277 3786 Acked Write 1 Done Sent Done false
12278 3787 Acked Write 6 Done Sent Done false
12279 3788 Acked Flush 0 Done Sent Done false
12280 3789 Acked Read 4 Done Sent Done false
12281 3790 Acked Write 5 Done Sent Done false
12282 3791 Acked Write 10 Done Sent Done false
12283 3792 Acked Read 7 Done Sent Done false
12284 3793 Acked Read 7 Done Sent Done false
12285 3794 Acked Read 9 Done Sent Done false
12286 3795 Acked Flush 0 Done Sent Done false
12287 3796 Acked Read 10 Done Sent Done false
12288 3797 Acked Flush 0 Done Sent Done false
12289 3798 Acked Write 9 Done Sent Done false
12290 3799 Acked Write 6 Done Sent Done false
12291 3800 Acked Write 8 Done Sent Done false
12292 3801 Acked Flush 0 Done Sent Done false
12293 3802 Acked Write 1 Done Sent Done false
12294 3803 Acked Read 5 Done Sent Done false
12295 3804 Acked Write 9 Done Sent Done false
12296 3805 Acked Write 2 Done Sent Done false
12297 3806 Acked Write 4 Done Sent Done false
12298 3807 Acked Write 1 Done Sent Done false
12299 3808 Acked Read 1 Done Sent Done false
12300 3809 Acked Flush 0 Done Sent Done false
12301 3810 Acked Read 10 Done Sent Done false
12302 3811 Acked Read 2 Done Sent Done false
12303 3812 Acked Flush 0 Done Sent Done false
12304 3813 Acked Read 7 Done Sent Done false
12305 3814 Acked Read 5 Done Sent Done false
12306 3815 Acked Write 9 Done Sent Done false
12307 3816 Acked Write 6 Done Sent Done false
12308 3817 Acked Read 8 Done Sent Done false
12309 3818 Acked Write 6 Done Sent Done false
12310 3819 Acked Read 6 Done Sent Done false
12311 3820 Acked Write 2 Done Sent Done false
12312 3821 Acked Flush 0 Done Sent Done false
12313 3822 Acked Read 8 Done Sent Done false
12314 3823 Acked Flush 0 Done Sent Done false
12315 3824 Acked Read 7 Done Sent Done false
12316 3825 Acked Write 9 Done Sent Done false
12317 3826 Acked Write 6 Done Sent Done false
12318 3827 Acked Write 5 Done Sent Done false
12319 3828 Acked Write 2 Done Sent Done false
12320 3829 Acked Read 8 Done Sent Done false
12321 3830 Acked Read 10 Done Sent Done false
12322 3831 Acked Read 5 Done Sent Done false
12323 3832 Acked Write 10 Done Sent Done false
12324 3833 Acked Read 3 Done Sent Done false
12325 3834 Acked Write 7 Done Sent Done false
12326 3835 Acked Write 10 Done Sent Done false
12327 3836 Acked Write 8 Done Sent Done false
12328 3837 Acked Flush 0 Done Sent Done false
12329 3838 Acked Read 7 Done Sent Done false
12330 3839 Acked Write 10 Done Sent Done false
12331 3840 Acked Read 6 Done Sent Done false
12332 3841 Acked Flush 0 Done Sent Done false
12333 3842 Acked Write 3 Done Sent Done false
12334 3843 Acked Write 4 Done Sent Done false
12335 3844 Acked Read 7 Done Sent Done false
12336 3845 Acked Read 8 Done Sent Done false
12337 3846 Acked Write 5 Done Sent Done false
12338 3847 Acked Write 6 Done Sent Done false
12339 3848 Acked Write 6 Done Sent Done false
12340 3849 Acked Read 1 Done Sent Done false
12341 3850 Acked Flush 0 Done Sent Done false
12342 3851 Acked Flush 0 Done Sent Done false
12343 3852 Acked Read 3 Done Sent Done false
12344 3853 Acked Write 6 Done Sent Done false
12345 3854 Acked Write 2 Done Sent Done false
12346 3855 Acked Read 10 Done Sent Done false
12347 3856 Acked Read 2 Done Sent Done false
12348 3857 Acked Write 5 Done Sent Done false
12349 3858 Acked Read 7 Done Sent Done false
12350 3859 Acked Write 2 Done Sent Done false
12351 3860 Acked Read 10 Done Sent Done false
12352 3861 Acked Read 6 Done Sent Done false
12353 3862 Acked Write 10 Done Sent Done false
12354 3863 Acked Write 2 Done Sent Done false
12355 3864 Acked Write 2 Done Sent Done false
12356 3865 Acked Write 3 Done Sent Done false
12357 3866 Acked Write 4 Done Sent Done false
12358 3867 Acked Write 9 Done Sent Done false
12359 3868 Acked Read 7 Done Sent Done false
12360 3869 Acked Read 1 Done Sent Done false
12361 3870 Acked Read 1 Done Sent Done false
12362 3871 Acked Read 5 Done Sent Done false
12363 3872 Acked Flush 0 Done Sent Done false
12364 3873 Acked Write 9 Done Sent Done false
12365 3874 Acked Write 3 Done Sent Done false
12366 3875 Acked Write 3 Done Sent Done false
12367 3876 Acked Read 9 Done Sent Done false
12368 3877 Acked Write 7 Done Sent Done false
12369 3878 Acked Read 6 Done Sent Done false
12370 3879 Acked Read 3 Done Sent Done false
12371 3880 Acked Flush 0 Done Sent Done false
12372 3881 Acked Flush 0 Done Sent Done false
12373 3882 Acked Read 4 Done Sent Done false
12374 3883 Acked Read 3 Done Sent Done false
12375 3884 Acked Read 9 Done Sent Done false
12376 3885 Acked Write 9 Done Sent Done false
12377 3886 Acked Read 2 Done Sent Done false
12378 3887 Acked Read 6 Done Sent Done false
12379 3888 Acked Read 3 Done Sent Done false
12380 3889 Acked Flush 0 Done Sent Done false
12381 3890 Acked Read 4 Done Sent Done false
12382 3891 Acked Write 4 Done Sent Done false
12383 3892 Acked Write 4 Done Sent Done false
12384 3893 Acked Read 2 Done Sent Done false
12385 3894 Acked Read 5 Done Sent Done false
12386 3895 Acked Read 8 Done Sent Done false
12387 3896 Acked Read 2 Done Sent Done false
12388 3897 Acked Read 7 Done Sent Done false
12389 3898 Acked Flush 0 Done Sent Done false
12390 3899 Acked Flush 0 Done Sent Done false
12391 3900 Acked Write 4 Done Sent Done false
12392 3901 Acked Write 2 Done Sent Done false
12393 3902 Acked Write 9 Done Sent Done false
12394 3903 Acked Read 9 Done Sent Done false
12395 3904 Acked Write 5 Done Sent Done false
12396 3905 Acked Read 9 Done Sent Done false
12397 3906 Acked Read 5 Done Sent Done false
12398 3907 Acked Write 3 Done Sent Done false
12399 3908 Acked Read 4 Done Sent Done false
12400 3909 Acked Write 9 Done Sent Done false
12401 3910 Acked Read 9 Done Sent Done false
12402 3911 Acked Read 9 Done Sent Done false
12403 3912 Acked Flush 0 Done Sent Done false
12404 3913 Acked Read 5 Done Sent Done false
12405 3914 Acked Read 1 Done Sent Done false
12406 3915 Acked Read 7 Done Sent Done false
12407 3916 Acked Read 7 Done Sent Done false
12408 3917 Acked Read 9 Done Sent Done false
12409 3918 Acked Read 9 Done Sent Done false
12410 3919 Acked Flush 0 Done Sent Done false
12411 3920 Acked Read 7 Done Sent Done false
12412 3921 Acked Read 4 Done Sent Done false
12413 3922 Acked Write 10 Done Sent Done false
12414 3923 Acked Read 6 Done Sent Done false
12415 3924 Acked Write 8 Done Sent Done false
12416 3925 Acked Read 5 Done Sent Done false
12417 3926 Acked Read 8 Done Sent Done false
12418 3927 Acked Write 3 Done Sent Done false
12419 3928 Acked Write 8 Done Sent Done false
12420 3929 Acked Read 3 Done Sent Done false
12421 3930 Acked Write 5 Done Sent Done false
12422 3931 Acked Write 5 Done Sent Done false
12423 3932 Acked Read 10 Done Sent Done false
12424 3933 Acked Read 8 Done Sent Done false
12425 3934 Acked Read 3 Done Sent Done false
12426 3935 Acked Write 1 Done Sent Done false
12427 3936 Acked Write 5 Done Sent Done false
12428 3937 Acked Write 3 Done Sent Done false
12429 3938 Acked Write 6 Done Sent Done false
12430 3939 Acked Read 10 Done Sent Done false
12431 3940 Acked Write 10 Done Sent Done false
12432 3941 Acked Read 1 Done Sent Done false
12433 3942 Acked Read 9 Done Sent Done false
12434 3943 Acked Read 8 Done Sent Done false
12435 3944 Acked Write 8 Done Sent Done false
12436 3945 Acked Read 5 Done Sent Done false
12437 3946 Acked Flush 0 Done Sent Done false
12438 3947 Acked Write 10 Done Sent Done false
12439 3948 Acked Write 6 Done Sent Done false
12440 3949 Acked Write 10 Done Sent Done false
12441 3950 Acked Read 9 Done Sent Done false
12442 3951 Acked Read 4 Done Sent Done false
12443 3952 Acked Read 10 Done Sent Done false
12444 3953 Acked Read 2 Done Sent Done false
12445 3954 Acked Read 5 Done Sent Done false
12446 3955 Acked Write 3 Done Sent Done false
12447 3956 Acked Read 6 Done Sent Done false
12448 3957 Acked Read 8 Done Sent Done false
12449 3958 Acked Read 5 Done Sent Done false
12450 3959 Acked Read 1 Done Sent Done false
12451 3960 Acked Read 10 Done Sent Done false
12452 3961 Acked Read 9 Done Sent Done false
12453 3962 Acked Read 3 Done Sent Done false
12454 3963 Acked Write 4 Done Sent Done false
12455 3964 Acked Write 7 Done Sent Done false
12456 3965 Acked Write 7 Done Sent Done false
12457 3966 Acked Write 4 Done Sent Done false
12458 3967 Acked Write 2 Done Sent Done false
12459 3968 Acked Read 2 Done Sent Done false
12460 3969 Acked Read 9 Done Sent Done false
12461 3970 Acked Write 6 Done Sent Done false
12462 3971 Acked Write 6 Done Sent Done false
12463 3972 Acked Read 10 Done Sent Done false
12464 3973 Acked Write 3 Done Sent Done false
12465 3974 Acked Read 3 Done Sent Done false
12466 3975 Acked Write 2 Done Sent Done false
12467 3976 Acked Flush 0 Done Sent Done false
12468 3977 Acked Read 4 Done Sent Done false
12469 3978 Acked Flush 0 Done Sent Done false
12470 3979 Acked Flush 0 Done Sent Done false
12471 3980 Acked Read 3 Done Sent Done false
12472 3981 Acked Read 5 Done Sent Done false
12473 3982 Acked Write 4 Done Sent Done false
12474 3983 Acked Read 1 Done Sent Done false
12475 3984 Acked Read 1 Done Sent Done false
12476 3985 Acked Read 1 Done Sent Done false
12477 3986 Acked Write 8 Done Sent Done false
12478 3987 Acked Read 7 Done Sent Done false
12479 3988 Acked Write 4 Done Sent Done false
12480 3989 Acked Write 1 Done Sent Done false
12481 3990 Acked Write 7 Done Sent Done false
12482 3991 Acked Write 6 Done Sent Done false
12483 3992 Acked Write 1 Done Sent Done false
12484 3993 Acked Write 6 Done Sent Done false
12485 3994 Acked Read 4 Done Sent Done false
12486 3995 Acked Read 7 Done Sent Done false
12487 3996 Acked Write 1 Done Sent Done false
12488 3997 Acked Read 1 Done Sent Done false
12489 3998 Acked Read 1 Done Sent Done false
12490 3999 Acked Write 6 Done Sent Done false
12491 4000 Acked Write 8 Done Sent Done false
12492 4001 Acked Read 7 Done Sent Done false
12493 4002 Acked Flush 0 Done Sent Done false
12494 4003 Acked Write 7 Done Sent Done false
12495 4004 Acked Read 2 Done Sent Done false
12496 4005 Acked Write 9 Done Sent Done false
12497 4006 Acked Read 1 Done Sent Done false
12498 4007 Acked Flush 0 Done Sent Done false
12499 4008 Acked Flush 0 Done Sent Done false
12500 4009 Acked Write 10 Done Sent Done false
12501 4010 Acked Read 9 Done Sent Done false
12502 4011 Acked Write 3 Done Sent Done false
12503 4012 Acked Write 3 Done Sent Done false
12504 4013 Acked Read 6 Done Sent Done false
12505 4014 Acked Read 3 Done Sent Done false
12506 4015 Acked Write 8 Done Sent Done false
12507 4016 Acked Read 6 Done Sent Done false
12508 4017 Acked Read 2 Done Sent Done false
12509 STATES DS:0 DS:1 DS:2 TOTAL
12510 Sent 0 301 0 301
12511 Done 301 0 301 602
12512 Skipped 0 0 0 0
12513 Error 0 0 0 0
12514 Last Flush: Some(JobId(4008)) Some(JobId(3716)) Some(JobId(4008))
12515 Downstairs last five completed: 3716 3715 3714 3713 3712
12516 Upstairs last five completed: 4017 4016 4015 4014 4013
12517 CLIENT: Up:0 ds:301 act:2
125182025-02-25T20:49:20.320ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
125192025-02-25T20:49:21.321ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
125202025-02-25T20:49:22.322ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
12521 [8820] got start action
12522 [8820] I am going to restart
12523 Starting downstairs at port 8820
12524 Make output file at "/tmp/dsc/downstairs-8820.txt"
12525 [8820][1] reports Starting
12526 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1216)
12527 [8820][1] reports Running
12528 ----------------------------------------------------------------
12529 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12530 States: Active Offline Active
12531 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12532 3717 Acked Read 10 Done Sent Done false
12533 3718 Acked Read 9 Done Sent Done false
12534 3719 Acked Write 5 Done Sent Done false
12535 3720 Acked Write 6 Done Sent Done false
12536 3721 Acked Read 5 Done Sent Done false
12537 3722 Acked Read 3 Done Sent Done false
12538 3723 Acked Read 3 Done Sent Done false
12539 3724 Acked Write 10 Done Sent Done false
12540 3725 Acked Write 5 Done Sent Done false
12541 3726 Acked Write 6 Done Sent Done false
12542 3727 Acked Write 4 Done Sent Done false
12543 3728 Acked Read 6 Done Sent Done false
12544 3729 Acked Write 8 Done Sent Done false
12545 3730 Acked Read 2 Done Sent Done false
12546 3731 Acked Read 6 Done Sent Done false
12547 3732 Acked Flush 0 Done Sent Done false
12548 3733 Acked Read 10 Done Sent Done false
12549 3734 Acked Write 4 Done Sent Done false
12550 3735 Acked Flush 0 Done Sent Done false
12551 3736 Acked Read 2 Done Sent Done false
12552 3737 Acked Read 8 Done Sent Done false
12553 3738 Acked Read 2 Done Sent Done false
12554 3739 Acked Write 6 Done Sent Done false
12555 3740 Acked Read 9 Done Sent Done false
12556 3741 Acked Write 10 Done Sent Done false
12557 3742 Acked Flush 0 Done Sent Done false
12558 3743 Acked Read 5 Done Sent Done false
12559 3744 Acked Read 5 Done Sent Done false
12560 3745 Acked Flush 0 Done Sent Done false
12561 3746 Acked Read 7 Done Sent Done false
12562 3747 Acked Write 10 Done Sent Done false
12563 3748 Acked Read 3 Done Sent Done false
12564 3749 Acked Read 2 Done Sent Done false
12565 3750 Acked Write 10 Done Sent Done false
12566 3751 Acked Write 10 Done Sent Done false
12567 3752 Acked Read 5 Done Sent Done false
12568 3753 Acked Write 6 Done Sent Done false
12569 3754 Acked Write 2 Done Sent Done false
12570 3755 Acked Flush 0 Done Sent Done false
12571 3756 Acked Read 9 Done Sent Done false
12572 3757 Acked Write 6 Done Sent Done false
12573 3758 Acked Read 3 Done Sent Done false
12574 3759 Acked Write 1 Done Sent Done false
12575 3760 Acked Read 6 Done Sent Done false
12576 3761 Acked Write 1 Done Sent Done false
12577 3762 Acked Write 9 Done Sent Done false
12578 3763 Acked Read 9 Done Sent Done false
12579 3764 Acked Read 2 Done Sent Done false
12580 3765 Acked Write 7 Done Sent Done false
12581 3766 Acked Read 5 Done Sent Done false
12582 3767 Acked Read 5 Done Sent Done false
12583 3768 Acked Read 5 Done Sent Done false
12584 3769 Acked Write 5 Done Sent Done false
12585 3770 Acked Write 10 Done Sent Done false
12586 3771 Acked Read 6 Done Sent Done false
12587 3772 Acked Write 10 Done Sent Done false
12588 3773 Acked Read 4 Done Sent Done false
12589 3774 Acked Read 9 Done Sent Done false
12590 3775 Acked Write 10 Done Sent Done false
12591 3776 Acked Write 9 Done Sent Done false
12592 3777 Acked Read 8 Done Sent Done false
12593 3778 Acked Read 10 Done Sent Done false
12594 3779 Acked Read 9 Done Sent Done false
12595 3780 Acked Read 5 Done Sent Done false
12596 3781 Acked Write 3 Done Sent Done false
12597 3782 Acked Read 1 Done Sent Done false
12598 3783 Acked Read 6 Done Sent Done false
12599 3784 Acked Read 10 Done Sent Done false
12600 3785 Acked Read 1 Done Sent Done false
12601 3786 Acked Write 1 Done Sent Done false
12602 3787 Acked Write 6 Done Sent Done false
12603 3788 Acked Flush 0 Done Sent Done false
12604 3789 Acked Read 4 Done Sent Done false
12605 3790 Acked Write 5 Done Sent Done false
12606 3791 Acked Write 10 Done Sent Done false
12607 3792 Acked Read 7 Done Sent Done false
12608 3793 Acked Read 7 Done Sent Done false
12609 3794 Acked Read 9 Done Sent Done false
12610 3795 Acked Flush 0 Done Sent Done false
12611 3796 Acked Read 10 Done Sent Done false
12612 3797 Acked Flush 0 Done Sent Done false
12613 3798 Acked Write 9 Done Sent Done false
12614 3799 Acked Write 6 Done Sent Done false
12615 3800 Acked Write 8 Done Sent Done false
12616 3801 Acked Flush 0 Done Sent Done false
12617 3802 Acked Write 1 Done Sent Done false
12618 3803 Acked Read 5 Done Sent Done false
12619 3804 Acked Write 9 Done Sent Done false
12620 3805 Acked Write 2 Done Sent Done false
12621 3806 Acked Write 4 Done Sent Done false
12622 3807 Acked Write 1 Done Sent Done false
12623 3808 Acked Read 1 Done Sent Done false
12624 3809 Acked Flush 0 Done Sent Done false
12625 3810 Acked Read 10 Done Sent Done false
12626 3811 Acked Read 2 Done Sent Done false
12627 3812 Acked Flush 0 Done Sent Done false
12628 3813 Acked Read 7 Done Sent Done false
12629 3814 Acked Read 5 Done Sent Done false
12630 3815 Acked Write 9 Done Sent Done false
12631 3816 Acked Write 6 Done Sent Done false
12632 3817 Acked Read 8 Done Sent Done false
12633 3818 Acked Write 6 Done Sent Done false
12634 3819 Acked Read 6 Done Sent Done false
12635 3820 Acked Write 2 Done Sent Done false
12636 3821 Acked Flush 0 Done Sent Done false
12637 3822 Acked Read 8 Done Sent Done false
12638 3823 Acked Flush 0 Done Sent Done false
12639 3824 Acked Read 7 Done Sent Done false
12640 3825 Acked Write 9 Done Sent Done false
12641 3826 Acked Write 6 Done Sent Done false
12642 3827 Acked Write 5 Done Sent Done false
12643 3828 Acked Write 2 Done Sent Done false
12644 3829 Acked Read 8 Done Sent Done false
12645 3830 Acked Read 10 Done Sent Done false
12646 3831 Acked Read 5 Done Sent Done false
12647 3832 Acked Write 10 Done Sent Done false
12648 3833 Acked Read 3 Done Sent Done false
12649 3834 Acked Write 7 Done Sent Done false
12650 3835 Acked Write 10 Done Sent Done false
12651 3836 Acked Write 8 Done Sent Done false
12652 3837 Acked Flush 0 Done Sent Done false
12653 3838 Acked Read 7 Done Sent Done false
12654 3839 Acked Write 10 Done Sent Done false
12655 3840 Acked Read 6 Done Sent Done false
12656 3841 Acked Flush 0 Done Sent Done false
12657 3842 Acked Write 3 Done Sent Done false
12658 3843 Acked Write 4 Done Sent Done false
12659 3844 Acked Read 7 Done Sent Done false
12660 3845 Acked Read 8 Done Sent Done false
12661 3846 Acked Write 5 Done Sent Done false
12662 3847 Acked Write 6 Done Sent Done false
12663 3848 Acked Write 6 Done Sent Done false
12664 3849 Acked Read 1 Done Sent Done false
12665 3850 Acked Flush 0 Done Sent Done false
12666 3851 Acked Flush 0 Done Sent Done false
12667 3852 Acked Read 3 Done Sent Done false
12668 3853 Acked Write 6 Done Sent Done false
12669 3854 Acked Write 2 Done Sent Done false
12670 3855 Acked Read 10 Done Sent Done false
12671 3856 Acked Read 2 Done Sent Done false
12672 3857 Acked Write 5 Done Sent Done false
12673 3858 Acked Read 7 Done Sent Done false
12674 3859 Acked Write 2 Done Sent Done false
12675 3860 Acked Read 10 Done Sent Done false
12676 3861 Acked Read 6 Done Sent Done false
12677 3862 Acked Write 10 Done Sent Done false
12678 3863 Acked Write 2 Done Sent Done false
12679 3864 Acked Write 2 Done Sent Done false
12680 3865 Acked Write 3 Done Sent Done false
12681 3866 Acked Write 4 Done Sent Done false
12682 3867 Acked Write 9 Done Sent Done false
12683 3868 Acked Read 7 Done Sent Done false
12684 3869 Acked Read 1 Done Sent Done false
12685 3870 Acked Read 1 Done Sent Done false
12686 3871 Acked Read 5 Done Sent Done false
12687 3872 Acked Flush 0 Done Sent Done false
12688 3873 Acked Write 9 Done Sent Done false
12689 3874 Acked Write 3 Done Sent Done false
12690 3875 Acked Write 3 Done Sent Done false
12691 3876 Acked Read 9 Done Sent Done false
12692 3877 Acked Write 7 Done Sent Done false
12693 3878 Acked Read 6 Done Sent Done false
12694 3879 Acked Read 3 Done Sent Done false
12695 3880 Acked Flush 0 Done Sent Done false
12696 3881 Acked Flush 0 Done Sent Done false
12697 3882 Acked Read 4 Done Sent Done false
12698 3883 Acked Read 3 Done Sent Done false
12699 3884 Acked Read 9 Done Sent Done false
12700 3885 Acked Write 9 Done Sent Done false
12701 3886 Acked Read 2 Done Sent Done false
12702 3887 Acked Read 6 Done Sent Done false
12703 3888 Acked Read 3 Done Sent Done false
12704 3889 Acked Flush 0 Done Sent Done false
12705 3890 Acked Read 4 Done Sent Done false
12706 3891 Acked Write 4 Done Sent Done false
12707 3892 Acked Write 4 Done Sent Done false
12708 3893 Acked Read 2 Done Sent Done false
12709 3894 Acked Read 5 Done Sent Done false
12710 3895 Acked Read 8 Done Sent Done false
12711 3896 Acked Read 2 Done Sent Done false
12712 3897 Acked Read 7 Done Sent Done false
12713 3898 Acked Flush 0 Done Sent Done false
12714 3899 Acked Flush 0 Done Sent Done false
12715 3900 Acked Write 4 Done Sent Done false
12716 3901 Acked Write 2 Done Sent Done false
12717 3902 Acked Write 9 Done Sent Done false
12718 3903 Acked Read 9 Done Sent Done false
12719 3904 Acked Write 5 Done Sent Done false
12720 3905 Acked Read 9 Done Sent Done false
12721 3906 Acked Read 5 Done Sent Done false
12722 3907 Acked Write 3 Done Sent Done false
12723 3908 Acked Read 4 Done Sent Done false
12724 3909 Acked Write 9 Done Sent Done false
12725 3910 Acked Read 9 Done Sent Done false
12726 3911 Acked Read 9 Done Sent Done false
12727 3912 Acked Flush 0 Done Sent Done false
12728 3913 Acked Read 5 Done Sent Done false
12729 3914 Acked Read 1 Done Sent Done false
12730 3915 Acked Read 7 Done Sent Done false
12731 3916 Acked Read 7 Done Sent Done false
12732 3917 Acked Read 9 Done Sent Done false
12733 3918 Acked Read 9 Done Sent Done false
12734 3919 Acked Flush 0 Done Sent Done false
12735 3920 Acked Read 7 Done Sent Done false
12736 3921 Acked Read 4 Done Sent Done false
12737 3922 Acked Write 10 Done Sent Done false
12738 3923 Acked Read 6 Done Sent Done false
12739 3924 Acked Write 8 Done Sent Done false
12740 3925 Acked Read 5 Done Sent Done false
12741 3926 Acked Read 8 Done Sent Done false
12742 3927 Acked Write 3 Done Sent Done false
12743 3928 Acked Write 8 Done Sent Done false
12744 3929 Acked Read 3 Done Sent Done false
12745 3930 Acked Write 5 Done Sent Done false
12746 3931 Acked Write 5 Done Sent Done false
12747 3932 Acked Read 10 Done Sent Done false
12748 3933 Acked Read 8 Done Sent Done false
12749 3934 Acked Read 3 Done Sent Done false
12750 3935 Acked Write 1 Done Sent Done false
12751 3936 Acked Write 5 Done Sent Done false
12752 3937 Acked Write 3 Done Sent Done false
12753 3938 Acked Write 6 Done Sent Done false
12754 3939 Acked Read 10 Done Sent Done false
12755 3940 Acked Write 10 Done Sent Done false
12756 3941 Acked Read 1 Done Sent Done false
12757 3942 Acked Read 9 Done Sent Done false
12758 3943 Acked Read 8 Done Sent Done false
12759 3944 Acked Write 8 Done Sent Done false
12760 3945 Acked Read 5 Done Sent Done false
12761 3946 Acked Flush 0 Done Sent Done false
12762 3947 Acked Write 10 Done Sent Done false
12763 3948 Acked Write 6 Done Sent Done false
12764 3949 Acked Write 10 Done Sent Done false
12765 3950 Acked Read 9 Done Sent Done false
12766 3951 Acked Read 4 Done Sent Done false
12767 3952 Acked Read 10 Done Sent Done false
12768 3953 Acked Read 2 Done Sent Done false
12769 3954 Acked Read 5 Done Sent Done false
12770 3955 Acked Write 3 Done Sent Done false
12771 3956 Acked Read 6 Done Sent Done false
12772 3957 Acked Read 8 Done Sent Done false
12773 3958 Acked Read 5 Done Sent Done false
12774 3959 Acked Read 1 Done Sent Done false
12775 3960 Acked Read 10 Done Sent Done false
12776 3961 Acked Read 9 Done Sent Done false
12777 3962 Acked Read 3 Done Sent Done false
12778 3963 Acked Write 4 Done Sent Done false
12779 3964 Acked Write 7 Done Sent Done false
12780 3965 Acked Write 7 Done Sent Done false
12781 3966 Acked Write 4 Done Sent Done false
12782 3967 Acked Write 2 Done Sent Done false
12783 3968 Acked Read 2 Done Sent Done false
12784 3969 Acked Read 9 Done Sent Done false
12785 3970 Acked Write 6 Done Sent Done false
12786 3971 Acked Write 6 Done Sent Done false
12787 3972 Acked Read 10 Done Sent Done false
12788 3973 Acked Write 3 Done Sent Done false
12789 3974 Acked Read 3 Done Sent Done false
12790 3975 Acked Write 2 Done Sent Done false
12791 3976 Acked Flush 0 Done Sent Done false
12792 3977 Acked Read 4 Done Sent Done false
12793 3978 Acked Flush 0 Done Sent Done false
12794 3979 Acked Flush 0 Done Sent Done false
12795 3980 Acked Read 3 Done Sent Done false
12796 3981 Acked Read 5 Done Sent Done false
12797 3982 Acked Write 4 Done Sent Done false
12798 3983 Acked Read 1 Done Sent Done false
12799 3984 Acked Read 1 Done Sent Done false
12800 3985 Acked Read 1 Done Sent Done false
12801 3986 Acked Write 8 Done Sent Done false
12802 3987 Acked Read 7 Done Sent Done false
12803 3988 Acked Write 4 Done Sent Done false
12804 3989 Acked Write 1 Done Sent Done false
12805 3990 Acked Write 7 Done Sent Done false
12806 3991 Acked Write 6 Done Sent Done false
12807 3992 Acked Write 1 Done Sent Done false
12808 3993 Acked Write 6 Done Sent Done false
12809 3994 Acked Read 4 Done Sent Done false
12810 3995 Acked Read 7 Done Sent Done false
12811 3996 Acked Write 1 Done Sent Done false
12812 3997 Acked Read 1 Done Sent Done false
12813 3998 Acked Read 1 Done Sent Done false
12814 3999 Acked Write 6 Done Sent Done false
12815 4000 Acked Write 8 Done Sent Done false
12816 4001 Acked Read 7 Done Sent Done false
12817 4002 Acked Flush 0 Done Sent Done false
12818 4003 Acked Write 7 Done Sent Done false
12819 4004 Acked Read 2 Done Sent Done false
12820 4005 Acked Write 9 Done Sent Done false
12821 4006 Acked Read 1 Done Sent Done false
12822 4007 Acked Flush 0 Done Sent Done false
12823 4008 Acked Flush 0 Done Sent Done false
12824 4009 Acked Write 10 Done Sent Done false
12825 4010 Acked Read 9 Done Sent Done false
12826 4011 Acked Write 3 Done Sent Done false
12827 4012 Acked Write 3 Done Sent Done false
12828 4013 Acked Read 6 Done Sent Done false
12829 4014 Acked Read 3 Done Sent Done false
12830 4015 Acked Write 8 Done Sent Done false
12831 4016 Acked Read 6 Done Sent Done false
12832 4017 Acked Read 2 Done Sent Done false
12833 STATES DS:0 DS:1 DS:2 TOTAL
12834 Sent 0 301 0 301
12835 Done 301 0 301 602
12836 Skipped 0 0 0 0
12837 Error 0 0 0 0
12838 Last Flush: Some(JobId(4008)) Some(JobId(3716)) Some(JobId(4008))
12839 Downstairs last five completed: 3716 3715 3714 3713 3712
12840 Upstairs last five completed: 4017 4016 4015 4014 4013
12841 CLIENT: Up:0 ds:301 act:2
128422025-02-25T20:49:23.668ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128432025-02-25T20:49:24.325ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128442025-02-25T20:49:25.325ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128452025-02-25T20:49:26.327ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128462025-02-25T20:49:26.828ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128472025-02-25T20:49:26.912ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128482025-02-25T20:49:26.912ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41908) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128492025-02-25T20:49:26.913ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128502025-02-25T20:49:26.914ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128512025-02-25T20:49:26.914ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128522025-02-25T20:49:26.914ZINFOcrucible: send last flush ID to this DS: Some(JobId(3716)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128532025-02-25T20:49:26.914ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128542025-02-25T20:49:26.914ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3716)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128552025-02-25T20:49:26.914ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3716)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
12856 ----------------------------------------------------------------
12857 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
12858 States: Active Active Active
12859 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12860 4009 Acked Write 10 Done Done Done true
12861 4010 Acked Read 9 Done Done Done true
12862 4011 Acked Write 3 Done Done Done true
12863 4012 Acked Write 3 Done Done Done true
12864 4013 Acked Read 6 Done Done Done true
12865 4014 Acked Read 3 Done Done Done true
12866 4015 Acked Write 8 Done Done Done true
12867 4016 Acked Read 6 Done Done Done true
12868 4017 Acked Read 2 Done Done Done true
12869 STATES DS:0 DS:1 DS:2 TOTAL
12870 Sent 0 0 0 0
12871 Done 9 9 9 27
12872 Skipped 0 0 0 0
12873 Error 0 0 0 0
12874 Last Flush: Some(JobId(4008)) Some(JobId(4008)) Some(JobId(4008))
12875 Downstairs last five completed: 4008 4007 4006 4005 4004
12876 Upstairs last five completed: 4017 4016 4015 4014 4013
12877 CLIENT: Up:0 ds:9 act:3
12878 ----------------------------------------------------------------
12879 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12880 Downstairs last five completed: 4018 4017 4016 4015 4014
12881 Upstairs last five completed: 4018 4017 4016 4015 4014
12882 CLIENT: Up:0 ds:0 act:3
12883 Replay: All jobs finished
12884 Main task has work to do, go find it
12885 got dsc Stop(1)
12886 stop 1
12887 Feb 25 20:49:31.171 INFO request completed, latency_us: 362, response_code: [8820] Got stop action so:false kr:false
12888 204, uri: /stop/cid/1, method: POST, req_id: 9f838466-12af-4a86-893b-c6fe841e4d70, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
128892025-02-25T20:49:31.172ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128902025-02-25T20:49:31.172ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128912025-02-25T20:49:31.172ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
128922025-02-25T20:49:31.172ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
12893 Feb 25 20:49:31.172 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4ec4f65c-c052-42c6-bff1-623b8c205594, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
12894 [8820] Exited with: ExitStatus(unix_wait_status(9))
12895 [8820][1] reports Exit
12896 Feb 25 20:49:35.174 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: a09998d2-4fde-4708-bd90-0cab208e634e, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
12897 001/300 Read block 290 len 12288
12898 002/300 Write block 56 len 40960 data: 3 3 3 3 3 3 3 2 2 2
12899 003/300 Flush
12900 004/300 Read block 2873 len 8192
12901 005/300 Read block 4046 len 8192
12902 006/300 Write block 1797 len 40960 data: 5 5 3 2 2 2 2 3 3 4
12903 007/300 Read block 1459 len 20480
12904 008/300 Write block 1721 len 40960 data: 4 4 4 3 3 3 4 3 3 4
12905 009/300 Write block 3656 len 20480 data: 4 4 5 5 5
12906 010/300 Write block 661 len 32768 data: 4 4 4 4 4 3 3 3
12907 011/300 Write block 4721 len 12288 data: 4 4 4
12908 012/300 Write block 4831 len 24576 data: 3 3 3 3 3 3
12909 013/300 Read block 1771 len 24576
12910 014/300 Flush
12911 015/300 Read block 790 len 8192
12912 016/300 Write block 3658 len 4096 data: 6
12913 017/300 Read block 3878 len 40960
12914 018/300 Write block 4159 len 28672 data: 4 5 7 7 7 7 8
12915 019/300 Read block 1174 len 24576
12916 020/300 Write block 2159 len 8192 data: 5 5
12917 021/300 Read block 3426 len 4096
12918 022/300 Read block 467 len 8192
12919 023/300 Write block 3948 len 16384 data: 5 5 6 5
12920 024/300 Read block 1496 len 4096
12921 025/300 Write block 2822 len 20480 data: 3 3 4 4 4
12922 026/300 Read block 2781 len 8192
12923 027/300 Read block 3195 len 8192
12924 028/300 Read block 4733 len 40960
12925 029/300 Flush
12926 030/300 Write block 890 len 16384 data: 3 3 2 2
12927 031/300 Read block 4655 len 4096
12928 032/300 Read block 1904 len 28672
12929 033/300 Write block 3177 len 4096 data: 3
12930 034/300 Write block 1570 len 4096 data: 3
12931 035/300 Read block 4224 len 40960
12932 036/300 Read block 902 len 4096
12933 037/300 Flush
12934 038/300 Read block 4677 len 36864
12935 039/300 Read block 1341 len 32768
12936 040/300 Read block 1093 len 40960
12937 041/300 Read block 1157 len 36864
12938 042/300 Write block 2254 len 40960 data: 3 4 5 4 5 6 5 5 2 2
12939 043/300 Read block 2069 len 28672
12940 044/300 Read block 4601 len 32768
12941 045/300 Write block 4746 len 16384 data: 3 3 3 4
12942 046/300 Read block 1655 len 40960
12943 047/300 Read block 774 len 32768
12944 048/300 Read block 2639 len 40960
12945 049/300 Read block 3599 len 40960
12946 050/300 Read block 3150 len 8192
12947 051/300 Read block 363 len 16384
12948 052/300 Read block 656 len 32768
12949 053/300 Read block 3938 len 40960
12950 054/300 Write block 4817 len 16384 data: 3 3 3 3
12951 055/300 Flush
12952 056/300 Read block 813 len 12288
12953 057/300 Read block 4589 len 16384
12954 058/300 Read block 3244 len 28672
12955 059/300 Read block 3069 len 12288
12956 060/300 Flush
12957 061/300 Write block 3841 len 8192 data: 4 3
12958 062/300 Read block 3207 len 16384
12959 063/300 Flush
12960 064/300 Read block 694 len 4096
12961 065/300 Read block 579 len 36864
12962 066/300 Write block 406 len 28672 data: 3 3 3 4 4 4 5
12963 067/300 Write block 1837 len 4096 data: 2
12964 068/300 Read block 1538 len 20480
12965 069/300 Read block 1712 len 4096
12966 070/300 Write block 1473 len 24576 data: 2 2 2 4 4 5
12967 071/300 Write block 2096 len 8192 data: 2 3
12968 072/300 Write block 3035 len 4096 data: 2
12969 073/300 Read block 4652 len 24576
12970 074/300 Read block 4557 len 16384
12971 075/300 Write block 3294 len 36864 data: 3 3 3 3 4 6 6 6 7
12972 076/300 Write block 720 len 40960 data: 3 4 4 3 3 4 4 4 4 5
12973 077/300 Read block 3550 len 20480
12974 078/300 Write block 2500 len 12288 data: 3 3 3
12975 079/300 Write block 332 len 12288 data: 6 5 5
12976 080/300 Read block 3603 len 20480
12977 081/300 Read block 2775 len 40960
12978 082/300 Write block 2552 len 32768 data: 4 4 3 3 3 3 3 2
12979 083/300 Read block 1031 len 32768
12980 084/300 Read block 4109 len 40960
12981 085/300 Read block 3907 len 20480
12982 086/300 Write block 2026 len 32768 data: 3 3 3 3 3 4 4 4
12983 087/300 Read block 2100 len 8192
12984 088/300 Flush
12985 089/300 Read block 2506 len 28672
12986 090/300 Write block 4634 len 16384 data: 2 3 3 3
12987 091/300 Read block 2766 len 36864
12988 092/300 Write block 3162 len 16384 data: 3 3 4 5
12989 093/300 Write block 4570 len 32768 data: 4 4 4 3 3 3 2 3
12990 094/300 Read block 1941 len 36864
12991 095/300 Read block 2958 len 36864
12992 096/300 Read block 1065 len 28672
12993 097/300 Read block 509 len 28672
12994 098/300 Read block 424 len 40960
12995 099/300 Write block 3796 len 20480 data: 2 3 4 4 4
12996 100/300 Write block 3956 len 36864 data: 3 3 4 3 4 4 4 3 2
12997 101/300 Read block 2952 len 24576
12998 102/300 Read block 4227 len 12288
12999 103/300 Read block 2531 len 32768
13000 104/300 Flush
13001 105/300 Read block 666 len 8192
13002 106/300 Read block 4864 len 36864
13003 107/300 Write block 4019 len 32768 data: 2 2 3 4 5 4 3 3
13004 108/300 Write block 3237 len 8192 data: 3 3
13005 109/300 Read block 772 len 4096
13006 110/300 Write block 3991 len 32768 data: 3 3 3 3 4 3 3 3
13007 111/300 Read block 1308 len 4096
13008 112/300 Write block 1796 len 12288 data: 5 6 6
13009 113/300 Read block 2381 len 24576
13010 114/300 Flush
13011 115/300 Write block 70 len 36864 data: 3 3 3 3 4 4 3 3 3
13012 116/300 Read block 1604 len 24576
13013 117/300 Read block 2873 len 28672
13014 118/300 Read block 4494 len 36864
13015 119/300 Flush
13016 120/300 Read block 2978 len 28672
13017 121/300 Write block 2187 len 12288 data: 4 4 4
13018 122/300 Write block 1577 len 40960 data: 4 4 4 5 4 6 7 6 5 4
13019 123/300 Read block 714 len 40960
13020 124/300 Read block 616 len 16384
13021 125/300 Write block 3819 len 24576 data: 3 4 3 3 3 3
13022 126/300 Write block 758 len 32768 data: 2 2 2 2 2 2 2 2
13023 127/300 Flush
13024 128/300 Write block 1779 len 8192 data: 5 3
13025 129/300 Read block 2071 len 28672
13026 130/300 Write block 2915 len 36864 data: 4 4 4 4 5 5 5 5 4
13027 131/300 Write block 899 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13028 132/300 Write block 2733 len 12288 data: 2 2 2
13029 133/300 Write block 163 len 12288 data: 4 3 4
13030 134/300 Write block 128 len 24576 data: 5 5 5 4 4 3
13031 135/300 Write block 1634 len 20480 data: 3 3 2 2 2
13032 136/300 Read block 1010 len 4096
13033 137/300 Read block 2876 len 24576
13034 138/300 Read block 2134 len 16384
13035 139/300 Read block 4070 len 12288
13036 140/300 Read block 862 len 32768
13037 141/300 Read block 4911 len 20480
13038 142/300 Read block 1030 len 28672
13039 143/300 Read block 2755 len 8192
13040 144/300 Write block 3429 len 36864 data: 3 2 2 3 3 4 4 5 4
13041 145/300 Write block 861 len 12288 data: 4 4 5
13042 146/300 Read block 1648 len 16384
13043 147/300 Read block 4483 len 16384
13044 148/300 Write block 541 len 24576 data: 2 2 2 2 2 2
13045 149/300 Read block 3706 len 28672
13046 150/300 Write block 3531 len 12288 data: 3 3 3
13047 151/300 Write block 2373 len 4096 data: 2
13048 152/300 Write block 2356 len 8192 data: 5 5
13049 153/300 Write block 4976 len 24576 data: 3 2 2 2 2 2
13050 154/300 Write block 1583 len 32768 data: 8 7 6 5 4 3 2 2
13051 155/300 Read block 3075 len 20480
13052 156/300 Write block 884 len 20480 data: 5 5 4 5 4
13053 157/300 Write block 3817 len 16384 data: 3 3 4 5
13054 158/300 Write block 4300 len 4096 data: 4
13055 159/300 Write block 3582 len 4096 data: 2
13056 160/300 Read block 759 len 12288
13057 161/300 Read block 4802 len 4096
13058 162/300 Write block 2028 len 20480 data: 4 4 4 5 5
13059 163/300 Write block 4870 len 40960 data: 2 3 4 3 5 4 5 5 4 4
13060 164/300 Flush
13061 165/300 Read block 2459 len 12288
13062 166/300 Read block 3119 len 40960
13063 167/300 Read block 3543 len 32768
13064 168/300 Read block 4000 len 32768
13065 169/300 Read block 1912 len 20480
13066 170/300 Write block 215 len 8192 data: 5 5
13067 171/300 Write block 3625 len 20480 data: 3 3 4 4 5
13068 172/300 Read block 2563 len 36864
13069 173/300 Read block 1583 len 24576
13070 174/300 Write block 2007 len 20480 data: 4 5 6 6 4
13071 175/300 Read block 2339 len 28672
13072 176/300 Flush
13073 177/300 Write block 2772 len 16384 data: 5 5 5 3
13074 178/300 Write block 4325 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13075 179/300 Write block 4028 len 32768 data: 3 3 2 2 2 2 2 2
13076 180/300 Read block 655 len 12288
13077 181/300 Read block 4163 len 24576
13078 182/300 Read block 2228 len 8192
13079 183/300 Read block 4515 len 20480
13080 184/300 Read block 4629 len 12288
13081 185/300 Read block 4594 len 28672
13082 186/300 Write block 3585 len 20480 data: 2 2 2 2 2
13083 187/300 Read block 4815 len 4096
13084 188/300 Read block 2448 len 40960
13085 189/300 Write block 3124 len 12288 data: 4 4 5
13086 190/300 Read block 672 len 8192
13087 191/300 Read block 2443 len 28672
13088 192/300 Read block 1876 len 40960
13089 193/300 Read block 458 len 20480
13090 194/300 Read block 1438 len 36864
13091 195/300 Write block 3865 len 8192 data: 2 2
13092 196/300 Write block 3713 len 16384 data: 3 3 3 3
13093 197/300 Read block 2433 len 32768
13094 198/300 Read block 2180 len 36864
13095 199/300 Read block 1496 len 4096
13096 200/300 Read block 2466 len 40960
13097 201/300 Write block 3416 len 20480 data: 3 5 5 5 5
13098 202/300 Flush
13099 203/300 Read block 3620 len 4096
13100 204/300 Write block 4217 len 4096 data: 5
13101 205/300 Read block 3187 len 8192
13102 206/300 Write block 2445 len 36864 data: 5 5 6 5 5 6 4 4 6
13103 207/300 Read block 4481 len 8192
13104 208/300 Read block 4038 len 24576
13105 209/300 Read block 4888 len 36864
13106 210/300 Write block 3565 len 12288 data: 3 3 4
13107 211/300 Read block 120 len 12288
13108 212/300 Write block 706 len 12288 data: 6 5 2
13109 213/300 Read block 1418 len 28672
13110 214/300 Read block 4650 len 32768
13111 215/300 Read block 3420 len 28672
13112 216/300 Write block 4425 len 8192 data: 6 7
13113 217/300 Write block 2458 len 8192 data: 3 3
13114 218/300 Write block 646 len 8192 data: 2 2
13115 219/300 Write block 2938 len 20480 data: 2 2 2 2 2
13116 220/300 Read block 2702 len 20480
13117 221/300 Flush
13118 222/300 Write block 2317 len 40960 data: 4 4 4 4 3 4 3 3 4 3
13119 223/300 Read block 847 len 32768
13120 224/300 Read block 2144 len 4096
13121 225/300 Write block 529 len 32768 data: 3 3 3 3 3 3 2 3
13122 226/300 Write block 2726 len 12288 data: 3 3 2
13123 227/300 Write block 1924 len 12288 data: 3 3 3
13124 228/300 Flush
13125 229/300 Write block 481 len 24576 data: 2 2 3 3 3 3
13126 230/300 Read block 4616 len 40960
13127 231/300 Flush
13128 232/300 Read block 4280 len 24576
13129 233/300 Flush
13130 234/300 Read block 4809 len 24576
13131 235/300 Write block 132 len 16384 data: 5 4 2 2
13132 236/300 Write block 1436 len 12288 data: 3 3 3
13133 237/300 Write block 2746 len 24576 data: 4 4 4 4 4 4
13134 238/300 Read block 580 len 24576
13135 239/300 Write block 3370 len 8192 data: 5 5
13136 240/300 Read block 4123 len 36864
13137 241/300 Write block 279 len 16384 data: 4 5 6 6
13138 242/300 Read block 1557 len 36864
13139 243/300 Read block 1178 len 20480
13140 244/300 Write block 341 len 20480 data: 3 3 3 3 3
13141 245/300 Read block 4716 len 40960
13142 246/300 Read block 4376 len 28672
13143 247/300 Write block 1296 len 36864 data: 2 2 3 3 3 2 2 2 3
13144 248/300 Write block 3956 len 4096 data: 4
13145 249/300 Write block 1922 len 24576 data: 3 2 4 4 4 3
13146 250/300 Write block 1563 len 24576 data: 3 3 2 2 2 2
13147 251/300 Read block 150 len 32768
13148 252/300 Read block 4544 len 36864
13149 253/300 Read block 971 len 8192
13150 254/300 Read block 4558 len 8192
13151 255/300 Read block 977 len 28672
13152 256/300 Write block 1079 len 4096 data: 2
13153 257/300 Write block 2604 len 36864 data: 2 3 4 4 4 3 3 3 2
13154 258/300 Read block 2645 len 40960
13155 259/300 Write block 2443 len 36864 data: 3 3 6 6 7 6 6 7 5
13156 260/300 Write block 1364 len 40960 data: 4 3 2 3 3 3 3 3 3 3
13157 261/300 Read block 2644 len 24576
13158 262/300 Read block 4204 len 36864
13159 263/300 Write block 4461 len 32768 data: 5 5 4 3 2 2 2 3
13160 264/300 Read block 32 len 32768
13161 265/300 Write block 2464 len 8192 data: 2 2
13162 266/300 Read block 2293 len 16384
13163 267/300 Read block 4413 len 28672
13164 268/300 Write block 1886 len 40960 data: 2 2 2 3 3 3 3 3 3 3
13165 269/300 Flush
13166 270/300 Write block 697 len 24576 data: 2 2 2 2 2 3
13167 271/300 Read block 3297 len 16384
13168 272/300 Read block 1149 len 24576
13169 273/300 Write block 2144 len 36864 data: 2 2 2 2 2 2 2 2 3
13170 274/300 Read block 135 len 32768
13171 275/300 Write block 4174 len 40960 data: 5 4 3 2 2 2 3 3 3 3
13172 276/300 Write block 537 len 20480 data: 2 2 2 2 3
13173 277/300 Read block 295 len 12288
13174 278/300 Read block 3211 len 20480
13175 279/300 Read block 1910 len 36864
13176 280/300 Read block 2257 len 12288
13177 281/300 Read block 3630 len 4096
13178 282/300 Read block 3640 len 12288
13179 283/300 Write block 1052 len 20480 data: 3 3 3 3 4
13180 284/300 Write block 3890 len 40960 data: 3 3 3 2 2 2 2 2 2 3
13181 285/300 Write block 2596 len 20480 data: 3 2 2 2 2
13182 286/300 Write block 4860 len 12288 data: 2 2 2
13183 287/300 Write block 4889 len 20480 data: 4 4 4 4 5
13184 288/300 Write block 936 len 12288 data: 2 2 2
13185 289/300 Read block 3975 len 12288
13186 290/300 Write block 3158 len 4096 data: 5
13187 291/300 Write block 4368 len 32768 data: 4 3 3 2 2 2 3 3
13188 292/300 Write block 3069 len 28672 data: 4 4 3 3 3 3 3
13189 293/300 Write block 1715 len 16384 data: 3 3 3 3
13190 294/300 Read block 2429 len 16384
13191 295/300 Read block 858 len 16384
13192 296/300 Write block 3344 len 4096 data: 2
13193 297/300 Read block 4498 len 40960
13194 298/300 Read block 3733 len 8192
13195 299/300 Read block 242 len 4096
13196 300/300 Read block 4874 len 40960
13197 301/300 Write block 1092 len 24576 data: 3 3 3 4 4 3
13198 Main task has work to do, go find it
13199 got dsc Start(1)
13200 start 1
13201 Feb 25 20:49:35.426 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/1, method: POST, req_id: c87fe3c6-6f55-48d8-8a6d-21122989c919, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
13202 Replay: started 1, returned:Ok(())
13203 ----------------------------------------------------------------
13204 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13205 States: Active Offline Active
13206 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13207 4019 Acked Read 3 Done Sent Done false
13208 4020 Acked Write 10 Done Sent Done false
13209 4021 Acked Flush 0 Done Sent Done false
13210 4022 Acked Read 2 Done Sent Done false
13211 4023 Acked Read 2 Done Sent Done false
13212 4024 Acked Write 10 Done Sent Done false
13213 4025 Acked Read 5 Done Sent Done false
13214 4026 Acked Write 10 Done Sent Done false
13215 4027 Acked Write 5 Done Sent Done false
13216 4028 Acked Write 8 Done Sent Done false
13217 4029 Acked Write 3 Done Sent Done false
13218 4030 Acked Write 6 Done Sent Done false
13219 4031 Acked Read 6 Done Sent Done false
13220 4032 Acked Flush 0 Done Sent Done false
13221 4033 Acked Read 2 Done Sent Done false
13222 4034 Acked Write 1 Done Sent Done false
13223 4035 Acked Read 10 Done Sent Done false
13224 4036 Acked Write 7 Done Sent Done false
13225 4037 Acked Read 6 Done Sent Done false
13226 4038 Acked Write 2 Done Sent Done false
13227 4039 Acked Read 1 Done Sent Done false
13228 4040 Acked Read 2 Done Sent Done false
13229 4041 Acked Write 4 Done Sent Done false
13230 4042 Acked Read 1 Done Sent Done false
13231 4043 Acked Write 5 Done Sent Done false
13232 4044 Acked Read 2 Done Sent Done false
13233 4045 Acked Read 2 Done Sent Done false
13234 4046 Acked Read 10 Done Sent Done false
13235 4047 Acked Flush 0 Done Sent Done false
13236 4048 Acked Write 4 Done Sent Done false
13237 4049 Acked Read 1 Done Sent Done false
13238 4050 Acked Read 7 Done Sent Done false
13239 4051 Acked Write 1 Done Sent Done false
13240 4052 Acked Write 1 Done Sent Done false
13241 4053 Acked Read 10 Done Sent Done false
13242 4054 Acked Read 1 Done Sent Done false
13243 4055 Acked Flush 0 Done Sent Done false
13244 4056 Acked Read 9 Done Sent Done false
13245 4057 Acked Read 8 Done Sent Done false
13246 4058 Acked Read 10 Done Sent Done false
13247 4059 Acked Read 9 Done Sent Done false
13248 4060 Acked Write 10 Done Sent Done false
13249 4061 Acked Read 7 Done Sent Done false
13250 4062 Acked Read 8 Done Sent Done false
13251 4063 Acked Write 4 Done Sent Done false
13252 4064 Acked Read 10 Done Sent Done false
13253 4065 Acked Read 8 Done Sent Done false
13254 4066 Acked Read 10 Done Sent Done false
13255 4067 Acked Read 10 Done Sent Done false
13256 4068 Acked Read 2 Done Sent Done false
13257 4069 Acked Read 4 Done Sent Done false
13258 4070 Acked Read 8 Done Sent Done false
13259 4071 Acked Read 10 Done Sent Done false
13260 4072 Acked Write 4 Done Sent Done false
13261 4073 Acked Flush 0 Done Sent Done false
13262 4074 Acked Read 3 Done Sent Done false
13263 4075 Acked Read 4 Done Sent Done false
13264 4076 Acked Read 7 Done Sent Done false
13265 4077 Acked Read 3 Done Sent Done false
13266 4078 Acked Flush 0 Done Sent Done false
13267 4079 Acked Write 2 Done Sent Done false
13268 4080 Acked Read 4 Done Sent Done false
13269 4081 Acked Flush 0 Done Sent Done false
13270 4082 Acked Read 1 Done Sent Done false
13271 4083 Acked Read 9 Done Sent Done false
13272 4084 Acked Write 7 Done Sent Done false
13273 4085 Acked Write 1 Done Sent Done false
13274 4086 Acked Read 5 Done Sent Done false
13275 4087 Acked Read 1 Done Sent Done false
13276 4088 Acked Write 6 Done Sent Done false
13277 4089 Acked Write 2 Done Sent Done false
13278 4090 Acked Write 1 Done Sent Done false
13279 4091 Acked Read 6 Done Sent Done false
13280 4092 Acked Read 4 Done Sent Done false
13281 4093 Acked Write 9 Done Sent Done false
13282 4094 Acked Write 10 Done Sent Done false
13283 4095 Acked Read 5 Done Sent Done false
13284 4096 Acked Write 3 Done Sent Done false
13285 4097 Acked Write 3 Done Sent Done false
13286 4098 Acked Read 5 Done Sent Done false
13287 4099 Acked Read 10 Done Sent Done false
13288 4100 Acked Write 8 Done Sent Done false
13289 4101 Acked Read 8 Done Sent Done false
13290 4102 Acked Read 10 Done Sent Done false
13291 4103 Acked Read 5 Done Sent Done false
13292 4104 Acked Write 8 Done Sent Done false
13293 4105 Acked Read 2 Done Sent Done false
13294 4106 Acked Flush 0 Done Sent Done false
13295 4107 Acked Read 7 Done Sent Done false
13296 4108 Acked Write 4 Done Sent Done false
13297 4109 Acked Read 9 Done Sent Done false
13298 4110 Acked Write 4 Done Sent Done false
13299 4111 Acked Write 8 Done Sent Done false
13300 4112 Acked Read 9 Done Sent Done false
13301 4113 Acked Read 9 Done Sent Done false
13302 4114 Acked Read 7 Done Sent Done false
13303 4115 Acked Read 7 Done Sent Done false
13304 4116 Acked Read 10 Done Sent Done false
13305 4117 Acked Write 5 Done Sent Done false
13306 4118 Acked Write 9 Done Sent Done false
13307 4119 Acked Read 6 Done Sent Done false
13308 4120 Acked Read 3 Done Sent Done false
13309 4121 Acked Read 8 Done Sent Done false
13310 4122 Acked Flush 0 Done Sent Done false
13311 4123 Acked Read 2 Done Sent Done false
13312 4124 Acked Read 9 Done Sent Done false
13313 4125 Acked Write 8 Done Sent Done false
13314 4126 Acked Write 2 Done Sent Done false
13315 4127 Acked Read 1 Done Sent Done false
13316 4128 Acked Write 8 Done Sent Done false
13317 4129 Acked Read 1 Done Sent Done false
13318 4130 Acked Write 3 Done Sent Done false
13319 4131 Acked Read 6 Done Sent Done false
13320 4132 Acked Flush 0 Done Sent Done false
13321 4133 Acked Write 9 Done Sent Done false
13322 4134 Acked Read 6 Done Sent Done false
13323 4135 Acked Read 7 Done Sent Done false
13324 4136 Acked Read 9 Done Sent Done false
13325 4137 Acked Flush 0 Done Sent Done false
13326 4138 Acked Read 7 Done Sent Done false
13327 4139 Acked Write 3 Done Sent Done false
13328 4140 Acked Write 10 Done Sent Done false
13329 4141 Acked Read 10 Done Sent Done false
13330 4142 Acked Read 4 Done Sent Done false
13331 4143 Acked Write 6 Done Sent Done false
13332 4144 Acked Write 8 Done Sent Done false
13333 4145 Acked Flush 0 Done Sent Done false
13334 4146 Acked Write 2 Done Sent Done false
13335 4147 Acked Read 7 Done Sent Done false
13336 4148 Acked Write 9 Done Sent Done false
13337 4149 Acked Write 10 Done Sent Done false
13338 4150 Acked Write 3 Done Sent Done false
13339 4151 Acked Write 3 Done Sent Done false
13340 4152 Acked Write 6 Done Sent Done false
13341 4153 Acked Write 5 Done Sent Done false
13342 4154 Acked Read 1 Done Sent Done false
13343 4155 Acked Read 6 Done Sent Done false
13344 4156 Acked Read 4 Done Sent Done false
13345 4157 Acked Read 3 Done Sent Done false
13346 4158 Acked Read 8 Done Sent Done false
13347 4159 Acked Read 5 Done Sent Done false
13348 4160 Acked Read 7 Done Sent Done false
13349 4161 Acked Read 2 Done Sent Done false
13350 4162 Acked Write 9 Done Sent Done false
13351 4163 Acked Write 3 Done Sent Done false
13352 4164 Acked Read 4 Done Sent Done false
13353 4165 Acked Read 4 Done Sent Done false
13354 4166 Acked Write 6 Done Sent Done false
13355 4167 Acked Read 7 Done Sent Done false
13356 4168 Acked Write 3 Done Sent Done false
13357 4169 Acked Write 1 Done Sent Done false
13358 4170 Acked Write 2 Done Sent Done false
13359 4171 Acked Write 6 Done Sent Done false
13360 4172 Acked Write 8 Done Sent Done false
13361 4173 Acked Read 5 Done Sent Done false
13362 4174 Acked Write 5 Done Sent Done false
13363 4175 Acked Write 4 Done Sent Done false
13364 4176 Acked Write 1 Done Sent Done false
13365 4177 Acked Write 1 Done Sent Done false
13366 4178 Acked Read 3 Done Sent Done false
13367 4179 Acked Read 1 Done Sent Done false
13368 4180 Acked Write 5 Done Sent Done false
13369 4181 Acked Write 10 Done Sent Done false
13370 4182 Acked Flush 0 Done Sent Done false
13371 4183 Acked Read 3 Done Sent Done false
13372 4184 Acked Read 10 Done Sent Done false
13373 4185 Acked Read 8 Done Sent Done false
13374 4186 Acked Read 8 Done Sent Done false
13375 4187 Acked Read 5 Done Sent Done false
13376 4188 Acked Write 2 Done Sent Done false
13377 4189 Acked Write 5 Done Sent Done false
13378 4190 Acked Read 9 Done Sent Done false
13379 4191 Acked Read 6 Done Sent Done false
13380 4192 Acked Write 5 Done Sent Done false
13381 4193 Acked Read 7 Done Sent Done false
13382 4194 Acked Flush 0 Done Sent Done false
13383 4195 Acked Write 4 Done Sent Done false
13384 4196 Acked Write 10 Done Sent Done false
13385 4197 Acked Write 8 Done Sent Done false
13386 4198 Acked Read 3 Done Sent Done false
13387 4199 Acked Read 6 Done Sent Done false
13388 4200 Acked Read 2 Done Sent Done false
13389 4201 Acked Read 5 Done Sent Done false
13390 4202 Acked Read 3 Done Sent Done false
13391 4203 Acked Read 7 Done Sent Done false
13392 4204 Acked Write 5 Done Sent Done false
13393 4205 Acked Read 1 Done Sent Done false
13394 4206 Acked Read 10 Done Sent Done false
13395 4207 Acked Write 3 Done Sent Done false
13396 4208 Acked Read 2 Done Sent Done false
13397 4209 Acked Read 7 Done Sent Done false
13398 4210 Acked Read 10 Done Sent Done false
13399 4211 Acked Read 5 Done Sent Done false
13400 4212 Acked Read 9 Done Sent Done false
13401 4213 Acked Write 2 Done Sent Done false
13402 4214 Acked Write 4 Done Sent Done false
13403 4215 Acked Read 8 Done Sent Done false
13404 4216 Acked Read 9 Done Sent Done false
13405 4217 Acked Read 1 Done Sent Done false
13406 4218 Acked Read 10 Done Sent Done false
13407 4219 Acked Write 5 Done Sent Done false
13408 4220 Acked Flush 0 Done Sent Done false
13409 4221 Acked Read 1 Done Sent Done false
13410 4222 Acked Write 1 Done Sent Done false
13411 4223 Acked Read 2 Done Sent Done false
13412 4224 Acked Write 9 Done Sent Done false
13413 4225 Acked Read 2 Done Sent Done false
13414 4226 Acked Read 6 Done Sent Done false
13415 4227 Acked Read 9 Done Sent Done false
13416 4228 Acked Write 3 Done Sent Done false
13417 4229 Acked Read 3 Done Sent Done false
13418 4230 Acked Write 3 Done Sent Done false
13419 4231 Acked Read 7 Done Sent Done false
13420 4232 Acked Read 8 Done Sent Done false
13421 4233 Acked Read 7 Done Sent Done false
13422 4234 Acked Write 2 Done Sent Done false
13423 4235 Acked Write 2 Done Sent Done false
13424 4236 Acked Write 2 Done Sent Done false
13425 4237 Acked Write 5 Done Sent Done false
13426 4238 Acked Read 5 Done Sent Done false
13427 4239 Acked Flush 0 Done Sent Done false
13428 4240 Acked Write 10 Done Sent Done false
13429 4241 Acked Read 8 Done Sent Done false
13430 4242 Acked Read 1 Done Sent Done false
13431 4243 Acked Write 8 Done Sent Done false
13432 4244 Acked Write 3 Done Sent Done false
13433 4245 Acked Write 3 Done Sent Done false
13434 4246 Acked Flush 0 Done Sent Done false
13435 4247 Acked Write 6 Done Sent Done false
13436 4248 Acked Read 10 Done Sent Done false
13437 4249 Acked Flush 0 Done Sent Done false
13438 4250 Acked Read 6 Done Sent Done false
13439 4251 Acked Flush 0 Done Sent Done false
13440 4252 Acked Read 6 Done Sent Done false
13441 4253 Acked Write 4 Done Sent Done false
13442 4254 Acked Write 3 Done Sent Done false
13443 4255 Acked Write 6 Done Sent Done false
13444 4256 Acked Read 6 Done Sent Done false
13445 4257 Acked Write 2 Done Sent Done false
13446 4258 Acked Read 9 Done Sent Done false
13447 4259 Acked Write 4 Done Sent Done false
13448 4260 Acked Read 9 Done Sent Done false
13449 4261 Acked Read 5 Done Sent Done false
13450 4262 Acked Write 5 Done Sent Done false
13451 4263 Acked Read 10 Done Sent Done false
13452 4264 Acked Read 7 Done Sent Done false
13453 4265 Acked Write 9 Done Sent Done false
13454 4266 Acked Write 1 Done Sent Done false
13455 4267 Acked Write 6 Done Sent Done false
13456 4268 Acked Write 6 Done Sent Done false
13457 4269 Acked Read 8 Done Sent Done false
13458 4270 Acked Read 9 Done Sent Done false
13459 4271 Acked Read 2 Done Sent Done false
13460 4272 Acked Read 2 Done Sent Done false
13461 4273 Acked Read 7 Done Sent Done false
13462 4274 Acked Write 1 Done Sent Done false
13463 4275 Acked Write 9 Done Sent Done false
13464 4276 Acked Read 10 Done Sent Done false
13465 4277 Acked Write 9 Done Sent Done false
13466 4278 Acked Write 10 Done Sent Done false
13467 4279 Acked Read 6 Done Sent Done false
13468 4280 Acked Read 9 Done Sent Done false
13469 4281 Acked Write 8 Done Sent Done false
13470 4282 Acked Read 8 Done Sent Done false
13471 4283 Acked Write 2 Done Sent Done false
13472 4284 Acked Read 4 Done Sent Done false
13473 4285 Acked Read 7 Done Sent Done false
13474 4286 Acked Write 10 Done Sent Done false
13475 4287 Acked Flush 0 Done Sent Done false
13476 4288 Acked Write 6 Done Sent Done false
13477 4289 Acked Read 4 Done Sent Done false
13478 4290 Acked Read 6 Done Sent Done false
13479 4291 Acked Write 9 Done Sent Done false
13480 4292 Acked Read 8 Done Sent Done false
13481 4293 Acked Write 10 Done Sent Done false
13482 4294 Acked Write 5 Done Sent Done false
13483 4295 Acked Read 3 Done Sent Done false
13484 4296 Acked Read 5 Done Sent Done false
13485 4297 Acked Read 9 Done Sent Done false
13486 4298 Acked Read 3 Done Sent Done false
13487 4299 Acked Read 1 Done Sent Done false
13488 4300 Acked Read 3 Done Sent Done false
13489 4301 Acked Write 5 Done Sent Done false
13490 4302 Acked Write 10 Done Sent Done false
13491 4303 Acked Write 5 Done Sent Done false
13492 4304 Acked Write 3 Done Sent Done false
13493 4305 Acked Write 5 Done Sent Done false
13494 4306 Acked Write 3 Done Sent Done false
13495 4307 Acked Read 3 Done Sent Done false
13496 4308 Acked Write 1 Done Sent Done false
13497 4309 Acked Write 8 Done Sent Done false
13498 4310 Acked Write 7 Done Sent Done false
13499 4311 Acked Write 4 Done Sent Done false
13500 4312 Acked Read 4 Done Sent Done false
13501 4313 Acked Read 4 Done Sent Done false
13502 4314 Acked Write 1 Done Sent Done false
13503 4315 Acked Read 10 Done Sent Done false
13504 4316 Acked Read 2 Done Sent Done false
13505 4317 Acked Read 1 Done Sent Done false
13506 4318 Acked Read 10 Done Sent Done false
13507 4319 Acked Write 6 Done Sent Sent false
13508 STATES DS:0 DS:1 DS:2 TOTAL
13509 Sent 0 301 1 302
13510 Done 301 0 300 601
13511 Skipped 0 0 0 0
13512 Error 0 0 0 0
13513 Last Flush: Some(JobId(4287)) Some(JobId(4018)) Some(JobId(4287))
13514 Downstairs last five completed: 4018 4017 4016 4015 4014
13515 Upstairs last five completed: 4319 4318 4317 4316 4315
13516 CLIENT: Up:0 ds:301 act:2
135172025-02-25T20:49:36.343ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
13518 [8820] got start action
13519 [8820] I am going to restart
13520 Starting downstairs at port 8820
13521 Make output file at "/tmp/dsc/downstairs-8820.txt"
13522 [8820][1] reports Starting
13523 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
13524 [8820][1] reports Running
135252025-02-25T20:49:37.346ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
135262025-02-25T20:49:38.346ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
135272025-02-25T20:49:39.347ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
13528 ----------------------------------------------------------------
13529 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13530 States: Active Offline Active
13531 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13532 4019 Acked Read 3 Done Sent Done false
13533 4020 Acked Write 10 Done Sent Done false
13534 4021 Acked Flush 0 Done Sent Done false
13535 4022 Acked Read 2 Done Sent Done false
13536 4023 Acked Read 2 Done Sent Done false
13537 4024 Acked Write 10 Done Sent Done false
13538 4025 Acked Read 5 Done Sent Done false
13539 4026 Acked Write 10 Done Sent Done false
13540 4027 Acked Write 5 Done Sent Done false
13541 4028 Acked Write 8 Done Sent Done false
13542 4029 Acked Write 3 Done Sent Done false
13543 4030 Acked Write 6 Done Sent Done false
13544 4031 Acked Read 6 Done Sent Done false
13545 4032 Acked Flush 0 Done Sent Done false
13546 4033 Acked Read 2 Done Sent Done false
13547 4034 Acked Write 1 Done Sent Done false
13548 4035 Acked Read 10 Done Sent Done false
13549 4036 Acked Write 7 Done Sent Done false
13550 4037 Acked Read 6 Done Sent Done false
13551 4038 Acked Write 2 Done Sent Done false
13552 4039 Acked Read 1 Done Sent Done false
13553 4040 Acked Read 2 Done Sent Done false
13554 4041 Acked Write 4 Done Sent Done false
13555 4042 Acked Read 1 Done Sent Done false
13556 4043 Acked Write 5 Done Sent Done false
13557 4044 Acked Read 2 Done Sent Done false
13558 4045 Acked Read 2 Done Sent Done false
13559 4046 Acked Read 10 Done Sent Done false
13560 4047 Acked Flush 0 Done Sent Done false
13561 4048 Acked Write 4 Done Sent Done false
13562 4049 Acked Read 1 Done Sent Done false
13563 4050 Acked Read 7 Done Sent Done false
13564 4051 Acked Write 1 Done Sent Done false
13565 4052 Acked Write 1 Done Sent Done false
13566 4053 Acked Read 10 Done Sent Done false
13567 4054 Acked Read 1 Done Sent Done false
13568 4055 Acked Flush 0 Done Sent Done false
13569 4056 Acked Read 9 Done Sent Done false
13570 4057 Acked Read 8 Done Sent Done false
13571 4058 Acked Read 10 Done Sent Done false
13572 4059 Acked Read 9 Done Sent Done false
13573 4060 Acked Write 10 Done Sent Done false
13574 4061 Acked Read 7 Done Sent Done false
13575 4062 Acked Read 8 Done Sent Done false
13576 4063 Acked Write 4 Done Sent Done false
13577 4064 Acked Read 10 Done Sent Done false
13578 4065 Acked Read 8 Done Sent Done false
13579 4066 Acked Read 10 Done Sent Done false
13580 4067 Acked Read 10 Done Sent Done false
13581 4068 Acked Read 2 Done Sent Done false
13582 4069 Acked Read 4 Done Sent Done false
13583 4070 Acked Read 8 Done Sent Done false
13584 4071 Acked Read 10 Done Sent Done false
13585 4072 Acked Write 4 Done Sent Done false
13586 4073 Acked Flush 0 Done Sent Done false
13587 4074 Acked Read 3 Done Sent Done false
13588 4075 Acked Read 4 Done Sent Done false
13589 4076 Acked Read 7 Done Sent Done false
13590 4077 Acked Read 3 Done Sent Done false
13591 4078 Acked Flush 0 Done Sent Done false
13592 4079 Acked Write 2 Done Sent Done false
13593 4080 Acked Read 4 Done Sent Done false
13594 4081 Acked Flush 0 Done Sent Done false
13595 4082 Acked Read 1 Done Sent Done false
13596 4083 Acked Read 9 Done Sent Done false
13597 4084 Acked Write 7 Done Sent Done false
13598 4085 Acked Write 1 Done Sent Done false
13599 4086 Acked Read 5 Done Sent Done false
13600 4087 Acked Read 1 Done Sent Done false
13601 4088 Acked Write 6 Done Sent Done false
13602 4089 Acked Write 2 Done Sent Done false
13603 4090 Acked Write 1 Done Sent Done false
13604 4091 Acked Read 6 Done Sent Done false
13605 4092 Acked Read 4 Done Sent Done false
13606 4093 Acked Write 9 Done Sent Done false
13607 4094 Acked Write 10 Done Sent Done false
13608 4095 Acked Read 5 Done Sent Done false
13609 4096 Acked Write 3 Done Sent Done false
13610 4097 Acked Write 3 Done Sent Done false
13611 4098 Acked Read 5 Done Sent Done false
13612 4099 Acked Read 10 Done Sent Done false
13613 4100 Acked Write 8 Done Sent Done false
13614 4101 Acked Read 8 Done Sent Done false
13615 4102 Acked Read 10 Done Sent Done false
13616 4103 Acked Read 5 Done Sent Done false
13617 4104 Acked Write 8 Done Sent Done false
13618 4105 Acked Read 2 Done Sent Done false
13619 4106 Acked Flush 0 Done Sent Done false
13620 4107 Acked Read 7 Done Sent Done false
13621 4108 Acked Write 4 Done Sent Done false
13622 4109 Acked Read 9 Done Sent Done false
13623 4110 Acked Write 4 Done Sent Done false
13624 4111 Acked Write 8 Done Sent Done false
13625 4112 Acked Read 9 Done Sent Done false
13626 4113 Acked Read 9 Done Sent Done false
13627 4114 Acked Read 7 Done Sent Done false
13628 4115 Acked Read 7 Done Sent Done false
13629 4116 Acked Read 10 Done Sent Done false
13630 4117 Acked Write 5 Done Sent Done false
13631 4118 Acked Write 9 Done Sent Done false
13632 4119 Acked Read 6 Done Sent Done false
13633 4120 Acked Read 3 Done Sent Done false
13634 4121 Acked Read 8 Done Sent Done false
13635 4122 Acked Flush 0 Done Sent Done false
13636 4123 Acked Read 2 Done Sent Done false
13637 4124 Acked Read 9 Done Sent Done false
13638 4125 Acked Write 8 Done Sent Done false
13639 4126 Acked Write 2 Done Sent Done false
13640 4127 Acked Read 1 Done Sent Done false
13641 4128 Acked Write 8 Done Sent Done false
13642 4129 Acked Read 1 Done Sent Done false
13643 4130 Acked Write 3 Done Sent Done false
13644 4131 Acked Read 6 Done Sent Done false
13645 4132 Acked Flush 0 Done Sent Done false
13646 4133 Acked Write 9 Done Sent Done false
13647 4134 Acked Read 6 Done Sent Done false
13648 4135 Acked Read 7 Done Sent Done false
13649 4136 Acked Read 9 Done Sent Done false
13650 4137 Acked Flush 0 Done Sent Done false
13651 4138 Acked Read 7 Done Sent Done false
13652 4139 Acked Write 3 Done Sent Done false
13653 4140 Acked Write 10 Done Sent Done false
13654 4141 Acked Read 10 Done Sent Done false
13655 4142 Acked Read 4 Done Sent Done false
13656 4143 Acked Write 6 Done Sent Done false
13657 4144 Acked Write 8 Done Sent Done false
13658 4145 Acked Flush 0 Done Sent Done false
13659 4146 Acked Write 2 Done Sent Done false
13660 4147 Acked Read 7 Done Sent Done false
13661 4148 Acked Write 9 Done Sent Done false
13662 4149 Acked Write 10 Done Sent Done false
13663 4150 Acked Write 3 Done Sent Done false
13664 4151 Acked Write 3 Done Sent Done false
13665 4152 Acked Write 6 Done Sent Done false
13666 4153 Acked Write 5 Done Sent Done false
13667 4154 Acked Read 1 Done Sent Done false
13668 4155 Acked Read 6 Done Sent Done false
13669 4156 Acked Read 4 Done Sent Done false
13670 4157 Acked Read 3 Done Sent Done false
13671 4158 Acked Read 8 Done Sent Done false
13672 4159 Acked Read 5 Done Sent Done false
13673 4160 Acked Read 7 Done Sent Done false
13674 4161 Acked Read 2 Done Sent Done false
13675 4162 Acked Write 9 Done Sent Done false
13676 4163 Acked Write 3 Done Sent Done false
13677 4164 Acked Read 4 Done Sent Done false
13678 4165 Acked Read 4 Done Sent Done false
13679 4166 Acked Write 6 Done Sent Done false
13680 4167 Acked Read 7 Done Sent Done false
13681 4168 Acked Write 3 Done Sent Done false
13682 4169 Acked Write 1 Done Sent Done false
13683 4170 Acked Write 2 Done Sent Done false
13684 4171 Acked Write 6 Done Sent Done false
13685 4172 Acked Write 8 Done Sent Done false
13686 4173 Acked Read 5 Done Sent Done false
13687 4174 Acked Write 5 Done Sent Done false
13688 4175 Acked Write 4 Done Sent Done false
13689 4176 Acked Write 1 Done Sent Done false
13690 4177 Acked Write 1 Done Sent Done false
13691 4178 Acked Read 3 Done Sent Done false
13692 4179 Acked Read 1 Done Sent Done false
13693 4180 Acked Write 5 Done Sent Done false
13694 4181 Acked Write 10 Done Sent Done false
13695 4182 Acked Flush 0 Done Sent Done false
13696 4183 Acked Read 3 Done Sent Done false
13697 4184 Acked Read 10 Done Sent Done false
13698 4185 Acked Read 8 Done Sent Done false
13699 4186 Acked Read 8 Done Sent Done false
13700 4187 Acked Read 5 Done Sent Done false
13701 4188 Acked Write 2 Done Sent Done false
13702 4189 Acked Write 5 Done Sent Done false
13703 4190 Acked Read 9 Done Sent Done false
13704 4191 Acked Read 6 Done Sent Done false
13705 4192 Acked Write 5 Done Sent Done false
13706 4193 Acked Read 7 Done Sent Done false
13707 4194 Acked Flush 0 Done Sent Done false
13708 4195 Acked Write 4 Done Sent Done false
13709 4196 Acked Write 10 Done Sent Done false
13710 4197 Acked Write 8 Done Sent Done false
13711 4198 Acked Read 3 Done Sent Done false
13712 4199 Acked Read 6 Done Sent Done false
13713 4200 Acked Read 2 Done Sent Done false
13714 4201 Acked Read 5 Done Sent Done false
13715 4202 Acked Read 3 Done Sent Done false
13716 4203 Acked Read 7 Done Sent Done false
13717 4204 Acked Write 5 Done Sent Done false
13718 4205 Acked Read 1 Done Sent Done false
13719 4206 Acked Read 10 Done Sent Done false
13720 4207 Acked Write 3 Done Sent Done false
13721 4208 Acked Read 2 Done Sent Done false
13722 4209 Acked Read 7 Done Sent Done false
13723 4210 Acked Read 10 Done Sent Done false
13724 4211 Acked Read 5 Done Sent Done false
13725 4212 Acked Read 9 Done Sent Done false
13726 4213 Acked Write 2 Done Sent Done false
13727 4214 Acked Write 4 Done Sent Done false
13728 4215 Acked Read 8 Done Sent Done false
13729 4216 Acked Read 9 Done Sent Done false
13730 4217 Acked Read 1 Done Sent Done false
13731 4218 Acked Read 10 Done Sent Done false
13732 4219 Acked Write 5 Done Sent Done false
13733 4220 Acked Flush 0 Done Sent Done false
13734 4221 Acked Read 1 Done Sent Done false
13735 4222 Acked Write 1 Done Sent Done false
13736 4223 Acked Read 2 Done Sent Done false
13737 4224 Acked Write 9 Done Sent Done false
13738 4225 Acked Read 2 Done Sent Done false
13739 4226 Acked Read 6 Done Sent Done false
13740 4227 Acked Read 9 Done Sent Done false
13741 4228 Acked Write 3 Done Sent Done false
13742 4229 Acked Read 3 Done Sent Done false
13743 4230 Acked Write 3 Done Sent Done false
13744 4231 Acked Read 7 Done Sent Done false
13745 4232 Acked Read 8 Done Sent Done false
13746 4233 Acked Read 7 Done Sent Done false
13747 4234 Acked Write 2 Done Sent Done false
13748 4235 Acked Write 2 Done Sent Done false
13749 4236 Acked Write 2 Done Sent Done false
13750 4237 Acked Write 5 Done Sent Done false
13751 4238 Acked Read 5 Done Sent Done false
13752 4239 Acked Flush 0 Done Sent Done false
13753 4240 Acked Write 10 Done Sent Done false
13754 4241 Acked Read 8 Done Sent Done false
13755 4242 Acked Read 1 Done Sent Done false
13756 4243 Acked Write 8 Done Sent Done false
13757 4244 Acked Write 3 Done Sent Done false
13758 4245 Acked Write 3 Done Sent Done false
13759 4246 Acked Flush 0 Done Sent Done false
13760 4247 Acked Write 6 Done Sent Done false
13761 4248 Acked Read 10 Done Sent Done false
13762 4249 Acked Flush 0 Done Sent Done false
13763 4250 Acked Read 6 Done Sent Done false
13764 4251 Acked Flush 0 Done Sent Done false
13765 4252 Acked Read 6 Done Sent Done false
13766 4253 Acked Write 4 Done Sent Done false
13767 4254 Acked Write 3 Done Sent Done false
13768 4255 Acked Write 6 Done Sent Done false
13769 4256 Acked Read 6 Done Sent Done false
13770 4257 Acked Write 2 Done Sent Done false
13771 4258 Acked Read 9 Done Sent Done false
13772 4259 Acked Write 4 Done Sent Done false
13773 4260 Acked Read 9 Done Sent Done false
13774 4261 Acked Read 5 Done Sent Done false
13775 4262 Acked Write 5 Done Sent Done false
13776 4263 Acked Read 10 Done Sent Done false
13777 4264 Acked Read 7 Done Sent Done false
13778 4265 Acked Write 9 Done Sent Done false
13779 4266 Acked Write 1 Done Sent Done false
13780 4267 Acked Write 6 Done Sent Done false
13781 4268 Acked Write 6 Done Sent Done false
13782 4269 Acked Read 8 Done Sent Done false
13783 4270 Acked Read 9 Done Sent Done false
13784 4271 Acked Read 2 Done Sent Done false
13785 4272 Acked Read 2 Done Sent Done false
13786 4273 Acked Read 7 Done Sent Done false
13787 4274 Acked Write 1 Done Sent Done false
13788 4275 Acked Write 9 Done Sent Done false
13789 4276 Acked Read 10 Done Sent Done false
13790 4277 Acked Write 9 Done Sent Done false
13791 4278 Acked Write 10 Done Sent Done false
13792 4279 Acked Read 6 Done Sent Done false
13793 4280 Acked Read 9 Done Sent Done false
13794 4281 Acked Write 8 Done Sent Done false
13795 4282 Acked Read 8 Done Sent Done false
13796 4283 Acked Write 2 Done Sent Done false
13797 4284 Acked Read 4 Done Sent Done false
13798 4285 Acked Read 7 Done Sent Done false
13799 4286 Acked Write 10 Done Sent Done false
13800 4287 Acked Flush 0 Done Sent Done false
13801 4288 Acked Write 6 Done Sent Done false
13802 4289 Acked Read 4 Done Sent Done false
13803 4290 Acked Read 6 Done Sent Done false
13804 4291 Acked Write 9 Done Sent Done false
13805 4292 Acked Read 8 Done Sent Done false
13806 4293 Acked Write 10 Done Sent Done false
13807 4294 Acked Write 5 Done Sent Done false
13808 4295 Acked Read 3 Done Sent Done false
13809 4296 Acked Read 5 Done Sent Done false
13810 4297 Acked Read 9 Done Sent Done false
13811 4298 Acked Read 3 Done Sent Done false
13812 4299 Acked Read 1 Done Sent Done false
13813 4300 Acked Read 3 Done Sent Done false
13814 4301 Acked Write 5 Done Sent Done false
13815 4302 Acked Write 10 Done Sent Done false
13816 4303 Acked Write 5 Done Sent Done false
13817 4304 Acked Write 3 Done Sent Done false
13818 4305 Acked Write 5 Done Sent Done false
13819 4306 Acked Write 3 Done Sent Done false
13820 4307 Acked Read 3 Done Sent Done false
13821 4308 Acked Write 1 Done Sent Done false
13822 4309 Acked Write 8 Done Sent Done false
13823 4310 Acked Write 7 Done Sent Done false
13824 4311 Acked Write 4 Done Sent Done false
13825 4312 Acked Read 4 Done Sent Done false
13826 4313 Acked Read 4 Done Sent Done false
13827 4314 Acked Write 1 Done Sent Done false
13828 4315 Acked Read 10 Done Sent Done false
13829 4316 Acked Read 2 Done Sent Done false
13830 4317 Acked Read 1 Done Sent Done false
13831 4318 Acked Read 10 Done Sent Done false
13832 4319 Acked Write 6 Done Sent Done false
13833 STATES DS:0 DS:1 DS:2 TOTAL
13834 Sent 0 301 0 301
13835 Done 301 0 301 602
13836 Skipped 0 0 0 0
13837 Error 0 0 0 0
13838 Last Flush: Some(JobId(4287)) Some(JobId(4018)) Some(JobId(4287))
13839 Downstairs last five completed: 4018 4017 4016 4015 4014
13840 Upstairs last five completed: 4319 4318 4317 4316 4315
13841 CLIENT: Up:0 ds:301 act:2
138422025-02-25T20:49:40.349ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138432025-02-25T20:49:41.173ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138442025-02-25T20:49:41.174ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48227) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138452025-02-25T20:49:41.175ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138462025-02-25T20:49:41.175ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138472025-02-25T20:49:41.175ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138482025-02-25T20:49:41.175ZINFOcrucible: send last flush ID to this DS: Some(JobId(4018)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138492025-02-25T20:49:41.175ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138502025-02-25T20:49:41.175ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4018)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
138512025-02-25T20:49:41.176ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4018)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
13852 ----------------------------------------------------------------
13853 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13854 Downstairs last five completed: 4320 4319 4318 4317 4316
13855 Upstairs last five completed: 4320 4319 4318 4317 4316
13856 CLIENT: Up:0 ds:0 act:3
13857 Replay: All jobs finished
13858 Main task has work to do, go find it
13859 got dsc Stop(0)
13860 stop 0
13861 Feb 25 20:49:43.432 INFO[8810] Got stop action so:false kr:false
13862 request completed, latency_us: 392, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 816e436e-6a53-4aca-bcd8-c57d234c719c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
138632025-02-25T20:49:43.432ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
13864 Feb 25 20:49:43.432 INFO request completed, latency_us: 139, response_code: 200{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:49:43.432939333Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
13865 , uri: /state/cid/0, method: GET, req_id: af2779b7-b458-4c11-9455-d66f9959f372, remote_addr:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:49:43.433000649Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
13866 127.0.0.1:43930, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:49:43.433038206Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
13867 local_addr: 127.0.0.1:9998
13868 [8810] Exited with: ExitStatus(unix_wait_status(9))
13869 [8810][0] reports Exit
13870 Feb 25 20:49:47.435 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 84e43acd-0690-4bf0-b418-f09ad944828e, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
13871 001/300 Write block 1584 len 16384 data: 8 7 6 5
13872 002/300 Read block 4604 len 28672
13873 003/300 Read block 964 len 4096
13874 004/300 Read block 2357 len 24576
13875 005/300 Flush
13876 006/300 Flush
13877 007/300 Read block 57 len 4096
13878 008/300 Write block 708 len 8192 data: 3 3
13879 009/300 Write block 3268 len 12288 data: 4 3 4
13880 010/300 Read block 4193 len 40960
13881 011/300 Write block 2766 len 36864 data: 2 3 3 5 6 5 6 6 6
13882 012/300 Flush
13883 013/300 Read block 616 len 24576
13884 014/300 Read block 4532 len 20480
13885 015/300 Flush
13886 016/300 Read block 670 len 12288
13887 017/300 Write block 1826 len 16384 data: 5 5 5 4
13888 018/300 Read block 2990 len 28672
13889 019/300 Read block 2684 len 12288
13890 020/300 Write block 2797 len 16384 data: 3 2 2 4
13891 021/300 Write block 3169 len 16384 data: 3 3 3 3
13892 022/300 Read block 691 len 4096
13893 023/300 Write block 4215 len 28672 data: 4 4 6 5 4 4 4
13894 024/300 Read block 4872 len 32768
13895 025/300 Write block 1257 len 32768 data: 3 3 3 4 4 4 4 4
13896 026/300 Read block 3122 len 12288
13897 027/300 Write block 183 len 24576 data: 3 3 3 4 4 4
13898 028/300 Write block 3276 len 24576 data: 4 4 5 4 4 5
13899 029/300 Write block 3389 len 8192 data: 3 3
13900 030/300 Read block 362 len 16384
13901 031/300 Write block 1013 len 8192 data: 5 5
13902 032/300 Write block 2348 len 4096 data: 4
13903 033/300 Write block 4942 len 12288 data: 3 3 3
13904 034/300 Write block 3075 len 28672 data: 4 2 2 2 2 2 2
13905 035/300 Read block 2767 len 40960
13906 036/300 Write block 4936 len 32768 data: 3 3 3 4 4 4 4 4
13907 037/300 Write block 2082 len 4096 data: 4
13908 038/300 Write block 434 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13909 039/300 Read block 4691 len 28672
13910 040/300 Read block 4149 len 36864
13911 041/300 Write block 4629 len 32768 data: 3 4 4 3 2 3 4 4
13912 042/300 Write block 2273 len 12288 data: 4 3 3
13913 043/300 Write block 2751 len 32768 data: 5 3 3 4 4 5 5 4
13914 044/300 Read block 1898 len 20480
13915 045/300 Write block 4089 len 16384 data: 3 3 2 3
13916 046/300 Read block 1835 len 12288
13917 047/300 Read block 4898 len 24576
13918 048/300 Read block 1455 len 40960
13919 049/300 Read block 4684 len 4096
13920 050/300 Write block 2800 len 24576 data: 5 4 3 4 4 4
13921 051/300 Read block 2448 len 20480
13922 052/300 Read block 1790 len 16384
13923 053/300 Read block 494 len 40960
13924 054/300 Read block 133 len 12288
13925 055/300 Read block 838 len 24576
13926 056/300 Write block 673 len 12288 data: 3 4 5
13927 057/300 Write block 4161 len 28672 data: 8 8 8 8 9 9 7
13928 058/300 Read block 4306 len 20480
13929 059/300 Write block 1305 len 20480 data: 4 4 3 3 3
13930 060/300 Write block 4326 len 16384 data: 3 3 3 3
13931 061/300 Read block 3558 len 28672
13932 062/300 Read block 2668 len 36864
13933 063/300 Write block 2659 len 28672 data: 2 2 2 2 2 2 3
13934 064/300 Read block 4987 len 32768
13935 065/300 Write block 1839 len 8192 data: 2 2
13936 066/300 Flush
13937 067/300 Read block 2401 len 8192
13938 068/300 Write block 60 len 8192 data: 4 4
13939 069/300 Write block 1360 len 4096 data: 3
13940 070/300 Read block 1403 len 8192
13941 071/300 Read block 1410 len 16384
13942 072/300 Write block 2566 len 12288 data: 3 3 3
13943 073/300 Write block 4133 len 40960 data: 2 2 2 2 2 2 2 2 4 5
13944 074/300 Read block 847 len 24576
13945 075/300 Read block 4340 len 28672
13946 076/300 Write block 1287 len 28672 data: 2 2 2 2 2 2 2
13947 077/300 Write block 153 len 12288 data: 2 2 2
13948 078/300 Write block 2365 len 20480 data: 3 3 3 3 3
13949 079/300 Write block 2019 len 4096 data: 2
13950 080/300 Read block 4557 len 8192
13951 081/300 Write block 3781 len 36864 data: 4 2 2 2 2 3 3 2 3
13952 082/300 Write block 674 len 8192 data: 5 6
13953 083/300 Read block 895 len 24576
13954 084/300 Read block 4647 len 36864
13955 085/300 Flush
13956 086/300 Write block 1359 len 36864 data: 3 4 5 4 5 5 4 3 4
13957 087/300 Write block 420 len 36864 data: 3 3 5 5 5 6 6 5 3
13958 088/300 Write block 4456 len 8192 data: 4 4
13959 089/300 Write block 2424 len 20480 data: 4 4 6 6 6
13960 090/300 Write block 2945 len 16384 data: 2 2 2 2
13961 091/300 Write block 917 len 32768 data: 3 3 4 4 4 4 4 4
13962 092/300 Write block 4376 len 36864 data: 3 3 3 2 2 2 2 2 2
13963 093/300 Write block 2646 len 28672 data: 4 6 5 5 5 5 4
13964 094/300 Read block 4039 len 4096
13965 095/300 Write block 4013 len 24576 data: 2 2 3 2 2 2
13966 096/300 Write block 1199 len 12288 data: 5 5 5
13967 097/300 Read block 615 len 28672
13968 098/300 Write block 3132 len 24576 data: 3 3 4 4 5 5
13969 099/300 Read block 2728 len 8192
13970 100/300 Read block 122 len 4096
13971 101/300 Write block 654 len 12288 data: 2 2 3
13972 102/300 Write block 2359 len 4096 data: 4
13973 103/300 Write block 2015 len 40960 data: 3 3 2 2 3 2 2 2 2 2
13974 104/300 Read block 63 len 8192
13975 105/300 Flush
13976 106/300 Flush
13977 107/300 Read block 1860 len 40960
13978 108/300 Read block 2892 len 12288
13979 109/300 Write block 1529 len 16384 data: 3 3 5 5
13980 110/300 Read block 2661 len 16384
13981 111/300 Write block 556 len 4096 data: 3
13982 112/300 Read block 524 len 4096
13983 113/300 Read block 4179 len 28672
13984 114/300 Read block 3850 len 28672
13985 115/300 Write block 4987 len 12288 data: 3 3 4
13986 116/300 Flush
13987 117/300 Write block 617 len 8192 data: 2 2
13988 118/300 Read block 1266 len 8192
13989 119/300 Write block 2750 len 36864 data: 5 6 4 4 5 5 6 6 5
13990 120/300 Flush
13991 121/300 Write block 278 len 8192 data: 3 5
13992 122/300 Write block 1408 len 28672 data: 2 4 4 4 3 3 3
13993 123/300 Read block 91 len 8192
13994 124/300 Read block 4083 len 4096
13995 125/300 Read block 2083 len 32768
13996 126/300 Write block 4055 len 16384 data: 2 3 3 4
13997 127/300 Write block 3309 len 32768 data: 2 3 3 3 3 3 4 3
13998 128/300 Read block 47 len 24576
13999 129/300 Read block 2542 len 32768
14000 130/300 Read block 2866 len 32768
14001 131/300 Read block 235 len 32768
14002 132/300 Write block 2351 len 12288 data: 5 4 5
14003 133/300 Write block 2900 len 28672 data: 4 4 3 3 3 3 4
14004 134/300 Read block 3682 len 12288
14005 135/300 Read block 3564 len 12288
14006 136/300 Write block 849 len 40960 data: 3 3 3 3 3 5 5 6 4 4
14007 137/300 Read block 1324 len 20480
14008 138/300 Read block 515 len 8192
14009 139/300 Write block 4259 len 36864 data: 2 2 2 2 3 3 3 3 3
14010 140/300 Write block 4375 len 12288 data: 4 4 4
14011 141/300 Read block 3078 len 36864
14012 142/300 Read block 2991 len 32768
14013 143/300 Read block 3975 len 36864
14014 144/300 Read block 2788 len 24576
14015 145/300 Write block 430 len 12288 data: 3 3 2
14016 146/300 Write block 4794 len 28672 data: 5 6 7 7 7 7 4
14017 147/300 Write block 50 len 12288 data: 2 2 2
14018 148/300 Read block 861 len 36864
14019 149/300 Read block 838 len 8192
14020 150/300 Flush
14021 151/300 Read block 3602 len 32768
14022 152/300 Write block 978 len 40960 data: 4 3 4 4 4 5 6 6 6 5
14023 153/300 Write block 3871 len 28672 data: 4 4 4 4 4 4 5
14024 154/300 Read block 1798 len 24576
14025 155/300 Write block 4685 len 40960 data: 4 4 4 4 4 5 5 4 3 4
14026 156/300 Read block 2657 len 24576
14027 157/300 Flush
14028 158/300 Read block 1330 len 16384
14029 159/300 Write block 3968 len 40960 data: 4 4 4 4 4 4 4 4 3 4
14030 160/300 Flush
14031 161/300 Write block 531 len 8192 data: 4 4
14032 162/300 Read block 1245 len 28672
14033 163/300 Read block 1318 len 4096
14034 164/300 Read block 3830 len 4096
14035 165/300 Read block 3010 len 36864
14036 166/300 Read block 4562 len 28672
14037 167/300 Write block 1018 len 40960 data: 4 5 4 4 3 3 3 4 3 3
14038 168/300 Write block 2170 len 28672 data: 3 3 2 2 2 2 2
14039 169/300 Write block 2495 len 16384 data: 3 2 3 3
14040 170/300 Write block 791 len 32768 data: 2 2 2 2 3 3 3 3
14041 171/300 Flush
14042 172/300 Read block 3915 len 24576
14043 173/300 Read block 4583 len 20480
14044 174/300 Flush
14045 175/300 Read block 150 len 24576
14046 176/300 Write block 257 len 12288 data: 4 3 3
14047 177/300 Read block 2430 len 20480
14048 178/300 Write block 1924 len 24576 data: 5 5 5 4 2 2
14049 179/300 Write block 3824 len 24576 data: 4 4 4 4 4 4
14050 180/300 Flush
14051 181/300 Read block 4564 len 36864
14052 182/300 Read block 4925 len 36864
14053 183/300 Read block 2897 len 32768
14054 184/300 Write block 1657 len 32768 data: 2 2 2 2 2 2 3 2
14055 185/300 Read block 4786 len 28672
14056 186/300 Read block 136 len 20480
14057 187/300 Write block 3293 len 12288 data: 3 4 4
14058 188/300 Read block 3852 len 24576
14059 189/300 Read block 1876 len 4096
14060 190/300 Read block 816 len 4096
14061 191/300 Read block 3559 len 12288
14062 192/300 Read block 4458 len 8192
14063 193/300 Read block 1022 len 20480
14064 194/300 Write block 999 len 32768 data: 3 3 3 3 4 4 4 3
14065 195/300 Write block 346 len 36864 data: 3 3 3 3 3 3 4 5 5
14066 196/300 Read block 689 len 8192
14067 197/300 Read block 4909 len 20480
14068 198/300 Read block 793 len 32768
14069 199/300 Flush
14070 200/300 Write block 1008 len 20480 data: 4 4 3 3 5
14071 201/300 Read block 3674 len 12288
14072 202/300 Read block 2859 len 20480
14073 203/300 Flush
14074 204/300 Write block 3720 len 16384 data: 5 5 6 6
14075 205/300 Write block 437 len 32768 data: 3 3 3 3 3 3 3 2
14076 206/300 Read block 1867 len 20480
14077 207/300 Write block 130 len 40960 data: 6 5 6 5 3 3 2 2 2 2
14078 208/300 Read block 51 len 16384
14079 209/300 Flush
14080 210/300 Write block 4173 len 32768 data: 6 6 5 4 3 3 3 4
14081 211/300 Flush
14082 212/300 Read block 699 len 8192
14083 213/300 Write block 1777 len 24576 data: 6 5 6 4 3 4
14084 214/300 Flush
14085 215/300 Write block 1018 len 28672 data: 5 6 5 5 4 4 4
14086 216/300 Write block 2021 len 8192 data: 3 3
14087 217/300 Read block 1031 len 8192
14088 218/300 Write block 1449 len 8192 data: 2 2
14089 219/300 Read block 518 len 16384
14090 220/300 Write block 1220 len 32768 data: 4 4 3 3 2 2 2 3
14091 221/300 Write block 1557 len 24576 data: 4 4 4 5 4 5
14092 222/300 Write block 1303 len 24576 data: 3 4 5 5 4 4
14093 223/300 Read block 4002 len 12288
14094 224/300 Read block 4695 len 28672
14095 225/300 Read block 4081 len 16384
14096 226/300 Read block 1769 len 28672
14097 227/300 Write block 3501 len 20480 data: 3 3 3 3 2
14098 228/300 Write block 437 len 28672 data: 4 4 4 4 4 4 4
14099 229/300 Write block 666 len 40960 data: 4 4 4 2 2 2 3 4 6 7
14100 230/300 Read block 882 len 8192
14101 231/300 Read block 391 len 40960
14102 232/300 Read block 3359 len 32768
14103 233/300 Read block 3916 len 32768
14104 234/300 Read block 3018 len 40960
14105 235/300 Write block 1156 len 28672 data: 2 2 2 2 2 2 2
14106 236/300 Flush
14107 237/300 Read block 4586 len 4096
14108 238/300 Flush
14109 239/300 Read block 505 len 4096
14110 240/300 Write block 3701 len 8192 data: 3 3
14111 241/300 Read block 3842 len 40960
14112 242/300 Write block 1444 len 20480 data: 2 2 2 2 2
14113 243/300 Write block 397 len 4096 data: 4
14114 244/300 Read block 4195 len 8192
14115 245/300 Read block 918 len 16384
14116 246/300 Write block 1606 len 36864 data: 4 4 3 2 3 3 4 4 4
14117 247/300 Write block 4245 len 8192 data: 3 4
14118 248/300 Write block 4321 len 16384 data: 2 2 2 2
14119 249/300 Write block 4949 len 24576 data: 4 4 4 4 5 3
14120 250/300 Read block 3984 len 8192
14121 251/300 Read block 3906 len 36864
14122 252/300 Read block 1686 len 40960
14123 253/300 Write block 4473 len 8192 data: 3 3
14124 254/300 Write block 691 len 4096 data: 4
14125 255/300 Write block 2244 len 4096 data: 7
14126 256/300 Write block 2902 len 16384 data: 4 4 4 4
14127 257/300 Write block 4279 len 32768 data: 3 3 2 2 2 2 2 2
14128 258/300 Flush
14129 259/300 Read block 1710 len 24576
14130 260/300 Write block 1540 len 8192 data: 3 3
14131 261/300 Write block 1870 len 36864 data: 3 3 2 2 5 6 6 6 6
14132 262/300 Read block 100 len 20480
14133 263/300 Read block 4429 len 12288
14134 264/300 Write block 1193 len 8192 data: 4 3
14135 265/300 Read block 4993 len 20480
14136 266/300 Write block 3635 len 40960 data: 5 5 5 6 6 7 7 8 7 7
14137 267/300 Read block 4111 len 28672
14138 268/300 Read block 2385 len 4096
14139 269/300 Read block 1404 len 16384
14140 270/300 Read block 1950 len 40960
14141 271/300 Write block 1728 len 36864 data: 4 4 5 4 3 2 3 3 2
14142 272/300 Read block 2151 len 28672
14143 273/300 Read block 2318 len 36864
14144 274/300 Read block 2513 len 28672
14145 275/300 Write block 2742 len 40960 data: 4 5 3 4 5 5 5 5 6 7
14146 276/300 Write block 834 len 4096 data: 2
14147 277/300 Read block 1086 len 4096
14148 278/300 Write block 2188 len 32768 data: 5 5 4 4 4 4 4 3
14149 279/300 Read block 2552 len 12288
14150 280/300 Write block 1559 len 16384 data: 5 6 5 6
14151 281/300 Read block 2309 len 24576
14152 282/300 Write block 1148 len 12288 data: 2 2 2
14153 283/300 Read block 4097 len 8192
14154 284/300 Read block 136 len 4096
14155 285/300 Write block 926 len 28672 data: 4 3 3 3 2 3 3
14156 286/300 Write block 3062 len 8192 data: 3 3
14157 287/300 Write block 1616 len 36864 data: 4 5 4 3 3 4 3 4 4
14158 288/300 Read block 4562 len 24576
14159 289/300 Read block 3162 len 20480
14160 290/300 Read block 85 len 32768
14161 291/300 Write block 657 len 40960 data: 3 3 3 4 5 5 5 5 5 5
14162 292/300 Read block 3997 len 8192
14163 293/300 Write block 1090 len 32768 data: 3 3 4 4 4 5 5 4
14164 294/300 Flush
14165 295/300 Write block 3850 len 28672 data: 2 2 3 3 4 4 4
14166 296/300 Read block 1907 len 20480
14167 297/300 Read block 807 len 28672
14168 298/300 Read block 1596 len 28672
14169 299/300 Read block 3968 len 12288
14170 300/300 Read block 52 len 8192
14171 301/300 Write block 1254 len 20480 data: 3 3 3 4 4
14172 Main task has work to do, go find it
14173 got dsc Start(0)
14174 start 0
14175 Feb 25 20:49:47.674 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: 95893905-0432-47aa-8f76-81178031919f, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
14176 Replay: started 0, returned:Ok(())
14177 ----------------------------------------------------------------
14178 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14179 States: Offline Active Active
14180 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14181 4321 Acked Write 4 Sent Done Done false
14182 4322 Acked Read 7 Sent Done Done false
14183 4323 Acked Read 1 Sent Done Done false
14184 4324 Acked Read 6 Sent Done Done false
14185 4325 Acked Flush 0 Sent Done Done false
14186 4326 Acked Flush 0 Sent Done Done false
14187 4327 Acked Read 1 Sent Done Done false
14188 4328 Acked Write 2 Sent Done Done false
14189 4329 Acked Write 3 Sent Done Done false
14190 4330 Acked Read 10 Sent Done Done false
14191 4331 Acked Write 9 Sent Done Done false
14192 4332 Acked Flush 0 Sent Done Done false
14193 4333 Acked Read 6 Sent Done Done false
14194 4334 Acked Read 5 Sent Done Done false
14195 4335 Acked Flush 0 Sent Done Done false
14196 4336 Acked Read 3 Sent Done Done false
14197 4337 Acked Write 4 Sent Done Done false
14198 4338 Acked Read 7 Sent Done Done false
14199 4339 Acked Read 3 Sent Done Done false
14200 4340 Acked Write 4 Sent Done Done false
14201 4341 Acked Write 4 Sent Done Done false
14202 4342 Acked Read 1 Sent Done Done false
14203 4343 Acked Write 7 Sent Done Done false
14204 4344 Acked Read 8 Sent Done Done false
14205 4345 Acked Write 8 Sent Done Done false
14206 4346 Acked Read 3 Sent Done Done false
14207 4347 Acked Write 6 Sent Done Done false
14208 4348 Acked Write 6 Sent Done Done false
14209 4349 Acked Write 2 Sent Done Done false
14210 4350 Acked Read 4 Sent Done Done false
14211 4351 Acked Write 2 Sent Done Done false
14212 4352 Acked Write 1 Sent Done Done false
14213 4353 Acked Write 3 Sent Done Done false
14214 4354 Acked Write 7 Sent Done Done false
14215 4355 Acked Read 10 Sent Done Done false
14216 4356 Acked Write 8 Sent Done Done false
14217 4357 Acked Write 1 Sent Done Done false
14218 4358 Acked Write 10 Sent Done Done false
14219 4359 Acked Read 7 Sent Done Done false
14220 4360 Acked Read 9 Sent Done Done false
14221 4361 Acked Write 8 Sent Done Done false
14222 4362 Acked Write 3 Sent Done Done false
14223 4363 Acked Write 8 Sent Done Done false
14224 4364 Acked Read 5 Sent Done Done false
14225 4365 Acked Write 4 Sent Done Done false
14226 4366 Acked Read 3 Sent Done Done false
14227 4367 Acked Read 6 Sent Done Done false
14228 4368 Acked Read 10 Sent Done Done false
14229 4369 Acked Read 1 Sent Done Done false
14230 4370 Acked Write 6 Sent Done Done false
14231 4371 Acked Read 5 Sent Done Done false
14232 4372 Acked Read 4 Sent Done Done false
14233 4373 Acked Read 10 Sent Done Done false
14234 4374 Acked Read 3 Sent Done Done false
14235 4375 Acked Read 6 Sent Done Done false
14236 4376 Acked Write 3 Sent Done Done false
14237 4377 Acked Write 7 Sent Done Done false
14238 4378 Acked Read 5 Sent Done Done false
14239 4379 Acked Write 5 Sent Done Done false
14240 4380 Acked Write 4 Sent Done Done false
14241 4381 Acked Read 7 Sent Done Done false
14242 4382 Acked Read 9 Sent Done Done false
14243 4383 Acked Write 7 Sent Done Done false
14244 4384 Acked Read 8 Sent Done Done false
14245 4385 Acked Write 2 Sent Done Done false
14246 4386 Acked Flush 0 Sent Done Done false
14247 4387 Acked Read 2 Sent Done Done false
14248 4388 Acked Write 2 Sent Done Done false
14249 4389 Acked Write 1 Sent Done Done false
14250 4390 Acked Read 2 Sent Done Done false
14251 4391 Acked Read 4 Sent Done Done false
14252 4392 Acked Write 3 Sent Done Done false
14253 4393 Acked Write 10 Sent Done Done false
14254 4394 Acked Read 6 Sent Done Done false
14255 4395 Acked Read 7 Sent Done Done false
14256 4396 Acked Write 7 Sent Done Done false
14257 4397 Acked Write 3 Sent Done Done false
14258 4398 Acked Write 5 Sent Done Done false
14259 4399 Acked Write 1 Sent Done Done false
14260 4400 Acked Read 2 Sent Done Done false
14261 4401 Acked Write 9 Sent Done Done false
14262 4402 Acked Write 2 Sent Done Done false
14263 4403 Acked Read 6 Sent Done Done false
14264 4404 Acked Read 9 Sent Done Done false
14265 4405 Acked Flush 0 Sent Done Done false
14266 4406 Acked Write 9 Sent Done Done false
14267 4407 Acked Write 9 Sent Done Done false
14268 4408 Acked Write 2 Sent Done Done false
14269 4409 Acked Write 5 Sent Done Done false
14270 4410 Acked Write 4 Sent Done Done false
14271 4411 Acked Write 8 Sent Done Done false
14272 4412 Acked Write 9 Sent Done Done false
14273 4413 Acked Write 7 Sent Done Done false
14274 4414 Acked Read 1 Sent Done Done false
14275 4415 Acked Write 6 Sent Done Done false
14276 4416 Acked Write 3 Sent Done Done false
14277 4417 Acked Read 7 Sent Done Done false
14278 4418 Acked Write 6 Sent Done Done false
14279 4419 Acked Read 2 Sent Done Done false
14280 4420 Acked Read 1 Sent Done Done false
14281 4421 Acked Write 3 Sent Done Done false
14282 4422 Acked Write 1 Sent Done Done false
14283 4423 Acked Write 10 Sent Done Done false
14284 4424 Acked Read 2 Sent Done Done false
14285 4425 Acked Flush 0 Sent Done Done false
14286 4426 Acked Flush 0 Sent Done Done false
14287 4427 Acked Read 10 Sent Done Done false
14288 4428 Acked Read 3 Sent Done Done false
14289 4429 Acked Write 4 Sent Done Done false
14290 4430 Acked Read 4 Sent Done Done false
14291 4431 Acked Write 1 Sent Done Done false
14292 4432 Acked Read 1 Sent Done Done false
14293 4433 Acked Read 7 Sent Done Done false
14294 4434 Acked Read 7 Sent Done Done false
14295 4435 Acked Write 3 Sent Done Done false
14296 4436 Acked Flush 0 Sent Done Done false
14297 4437 Acked Write 2 Sent Done Done false
14298 4438 Acked Read 2 Sent Done Done false
14299 4439 Acked Write 9 Sent Done Done false
14300 4440 Acked Flush 0 Sent Done Done false
14301 4441 Acked Write 2 Sent Done Done false
14302 4442 Acked Write 7 Sent Done Done false
14303 4443 Acked Read 2 Sent Done Done false
14304 4444 Acked Read 1 Sent Done Done false
14305 4445 Acked Read 8 Sent Done Done false
14306 4446 Acked Write 4 Sent Done Done false
14307 4447 Acked Write 8 Sent Done Done false
14308 4448 Acked Read 6 Sent Done Done false
14309 4449 Acked Read 8 Sent Done Done false
14310 4450 Acked Read 8 Sent Done Done false
14311 4451 Acked Read 8 Sent Done Done false
14312 4452 Acked Write 3 Sent Done Done false
14313 4453 Acked Write 7 Sent Done Done false
14314 4454 Acked Read 3 Sent Done Done false
14315 4455 Acked Read 3 Sent Done Done false
14316 4456 Acked Write 10 Sent Done Done false
14317 4457 Acked Read 5 Sent Done Done false
14318 4458 Acked Read 2 Sent Done Done false
14319 4459 Acked Write 9 Sent Done Done false
14320 4460 Acked Write 3 Sent Done Done false
14321 4461 Acked Read 9 Sent Done Done false
14322 4462 Acked Read 8 Sent Done Done false
14323 4463 Acked Read 9 Sent Done Done false
14324 4464 Acked Read 6 Sent Done Done false
14325 4465 Acked Write 3 Sent Done Done false
14326 4466 Acked Write 7 Sent Done Done false
14327 4467 Acked Write 3 Sent Done Done false
14328 4468 Acked Read 9 Sent Done Done false
14329 4469 Acked Read 2 Sent Done Done false
14330 4470 Acked Flush 0 Sent Done Done false
14331 4471 Acked Read 8 Sent Done Done false
14332 4472 Acked Write 10 Sent Done Done false
14333 4473 Acked Write 7 Sent Done Done false
14334 4474 Acked Read 6 Sent Done Done false
14335 4475 Acked Write 10 Sent Done Done false
14336 4476 Acked Read 6 Sent Done Done false
14337 4477 Acked Flush 0 Sent Done Done false
14338 4478 Acked Read 4 Sent Done Done false
14339 4479 Acked Write 10 Sent Done Done false
14340 4480 Acked Flush 0 Sent Done Done false
14341 4481 Acked Write 2 Sent Done Done false
14342 4482 Acked Read 7 Sent Done Done false
14343 4483 Acked Read 1 Sent Done Done false
14344 4484 Acked Read 1 Sent Done Done false
14345 4485 Acked Read 9 Sent Done Done false
14346 4486 Acked Read 7 Sent Done Done false
14347 4487 Acked Write 10 Sent Done Done false
14348 4488 Acked Write 7 Sent Done Done false
14349 4489 Acked Write 4 Sent Done Done false
14350 4490 Acked Write 8 Sent Done Done false
14351 4491 Acked Flush 0 Sent Done Done false
14352 4492 Acked Read 6 Sent Done Done false
14353 4493 Acked Read 5 Sent Done Done false
14354 4494 Acked Flush 0 Sent Done Done false
14355 4495 Acked Read 6 Sent Done Done false
14356 4496 Acked Write 3 Sent Done Done false
14357 4497 Acked Read 5 Sent Done Done false
14358 4498 Acked Write 6 Sent Done Done false
14359 4499 Acked Write 6 Sent Done Done false
14360 4500 Acked Flush 0 Sent Done Done false
14361 4501 Acked Read 9 Sent Done Done false
14362 4502 Acked Read 9 Sent Done Done false
14363 4503 Acked Read 8 Sent Done Done false
14364 4504 Acked Write 8 Sent Done Done false
14365 4505 Acked Read 7 Sent Done Done false
14366 4506 Acked Read 5 Sent Done Done false
14367 4507 Acked Write 3 Sent Done Done false
14368 4508 Acked Read 6 Sent Done Done false
14369 4509 Acked Read 1 Sent Done Done false
14370 4510 Acked Read 1 Sent Done Done false
14371 4511 Acked Read 3 Sent Done Done false
14372 4512 Acked Read 2 Sent Done Done false
14373 4513 Acked Read 5 Sent Done Done false
14374 4514 Acked Write 8 Sent Done Done false
14375 4515 Acked Write 9 Sent Done Done false
14376 4516 Acked Read 2 Sent Done Done false
14377 4517 Acked Read 5 Sent Done Done false
14378 4518 Acked Read 8 Sent Done Done false
14379 4519 Acked Flush 0 Sent Done Done false
14380 4520 Acked Write 5 Sent Done Done false
14381 4521 Acked Read 3 Sent Done Done false
14382 4522 Acked Read 5 Sent Done Done false
14383 4523 Acked Flush 0 Sent Done Done false
14384 4524 Acked Write 4 Sent Done Done false
14385 4525 Acked Write 8 Sent Done Done false
14386 4526 Acked Read 5 Sent Done Done false
14387 4527 Acked Write 10 Sent Done Done false
14388 4528 Acked Read 4 Sent Done Done false
14389 4529 Acked Flush 0 Sent Done Done false
14390 4530 Acked Write 8 Sent Done Done false
14391 4531 Acked Flush 0 Sent Done Done false
14392 4532 Acked Read 2 Sent Done Done false
14393 4533 Acked Write 6 Sent Done Done false
14394 4534 Acked Flush 0 Sent Done Done false
14395 4535 Acked Write 7 Sent Done Done false
14396 4536 Acked Write 2 Sent Done Done false
14397 4537 Acked Read 2 Sent Done Done false
14398 4538 Acked Write 2 Sent Done Done false
14399 4539 Acked Read 4 Sent Done Done false
14400 4540 Acked Write 8 Sent Done Done false
14401 4541 Acked Write 6 Sent Done Done false
14402 4542 Acked Write 6 Sent Done Done false
14403 4543 Acked Read 3 Sent Done Done false
14404 4544 Acked Read 7 Sent Done Done false
14405 4545 Acked Read 4 Sent Done Done false
14406 4546 Acked Read 7 Sent Done Done false
14407 4547 Acked Write 5 Sent Done Done false
14408 4548 Acked Write 7 Sent Done Done false
14409 4549 Acked Write 10 Sent Done Done false
14410 4550 Acked Read 2 Sent Done Done false
14411 4551 Acked Read 10 Sent Done Done false
14412 4552 Acked Read 8 Sent Done Done false
14413 4553 Acked Read 8 Sent Done Done false
14414 4554 Acked Read 10 Sent Done Done false
14415 4555 Acked Write 7 Sent Done Done false
14416 4556 Acked Flush 0 Sent Done Done false
14417 4557 Acked Read 1 Sent Done Done false
14418 4558 Acked Flush 0 Sent Done Done false
14419 4559 Acked Read 1 Sent Done Done false
14420 4560 Acked Write 2 Sent Done Done false
14421 4561 Acked Read 10 Sent Done Done false
14422 4562 Acked Write 5 Sent Done Done false
14423 4563 Acked Write 1 Sent Done Done false
14424 4564 Acked Read 2 Sent Done Done false
14425 4565 Acked Read 4 Sent Done Done false
14426 4566 Acked Write 9 Sent Done Done false
14427 4567 Acked Write 2 Sent Done Done false
14428 4568 Acked Write 4 Sent Done Done false
14429 4569 Acked Write 6 Sent Done Done false
14430 4570 Acked Read 2 Sent Done Done false
14431 4571 Acked Read 9 Sent Done Done false
14432 4572 Acked Read 10 Sent Done Done false
14433 4573 Acked Write 2 Sent Done Done false
14434 4574 Acked Write 1 Sent Done Done false
14435 4575 Acked Write 1 Sent Done Done false
14436 4576 Acked Write 4 Sent Done Done false
14437 4577 Acked Write 8 Sent Done Done false
14438 4578 Acked Flush 0 Sent Done Done false
14439 4579 Acked Read 6 Sent Done Done false
14440 4580 Acked Write 2 Sent Done Done false
14441 4581 Acked Write 9 Sent Done Done false
14442 4582 Acked Read 5 Sent Done Done false
14443 4583 Acked Read 3 Sent Done Done false
14444 4584 Acked Write 2 Sent Done Done false
14445 4585 Acked Read 5 Sent Done Done false
14446 4586 Acked Write 10 Sent Done Done false
14447 4587 Acked Read 7 Sent Done Done false
14448 4588 Acked Read 1 Sent Done Done false
14449 4589 Acked Read 4 Sent Done Done false
14450 4590 Acked Read 10 Sent Done Done false
14451 4591 Acked Write 9 Sent Done Done false
14452 4592 Acked Read 7 Sent Done Done false
14453 4593 Acked Read 9 Sent Done Done false
14454 4594 Acked Read 7 Sent Done Done false
14455 4595 Acked Write 10 Sent Done Done false
14456 4596 Acked Write 1 Sent Done Done false
14457 4597 Acked Read 1 Sent Done Done false
14458 4598 Acked Write 8 Sent Done Done false
14459 4599 Acked Read 3 Sent Done Done false
14460 4600 Acked Write 4 Sent Done Done false
14461 4601 Acked Read 6 Sent Done Done false
14462 4602 Acked Write 3 Sent Done Done false
14463 4603 Acked Read 2 Sent Done Done false
14464 4604 Acked Read 1 Sent Done Done false
14465 4605 Acked Write 7 Sent Done Done false
14466 4606 Acked Write 2 Sent Done Done false
14467 4607 Acked Write 9 Sent Done Done false
14468 4608 Acked Read 6 Sent Done Done false
14469 4609 Acked Read 5 Sent Done Done false
14470 4610 Acked Read 8 Sent Done Done false
14471 4611 Acked Write 10 Sent Done Done false
14472 4612 Acked Read 2 Sent Done Done false
14473 4613 Acked Write 8 Sent Done Done false
14474 4614 Acked Flush 0 Sent Done Done false
14475 4615 Acked Write 7 Sent Done Done false
14476 4616 Acked Read 5 Sent Done Done false
14477 4617 Acked Read 7 Sent Done Done false
14478 4618 Acked Read 7 Sent Done Done false
14479 4619 Acked Read 3 Sent Done Done false
14480 4620 Acked Read 2 Sent Done Done false
14481 4621 Acked Write 5 Sent Done Done false
14482 STATES DS:0 DS:1 DS:2 TOTAL
14483 Sent 301 0 0 301
14484 Done 0 301 301 602
14485 Skipped 0 0 0 0
14486 Error 0 0 0 0
14487 Last Flush: Some(JobId(4320)) Some(JobId(4614)) Some(JobId(4614))
14488 Downstairs last five completed: 4320 4319 4318 4317 4316
14489 Upstairs last five completed: 4621 4620 4619 4618 4617
14490 CLIENT: Up:0 ds:301 act:2
144912025-02-25T20:49:48.361ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
144922025-02-25T20:49:49.363ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
14493 [8810] got start action
14494 [8810] I am going to restart
14495 Starting downstairs at port 8810
14496 Make output file at "/tmp/dsc/downstairs-8810.txt"
14497 [8810][0] reports Starting
14498 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218)
14499 [8810][0] reports Running
145002025-02-25T20:49:50.364ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
14501 ----------------------------------------------------------------
14502 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14503 States: Offline Active Active
14504 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14505 4321 Acked Write 4 Sent Done Done false
14506 4322 Acked Read 7 Sent Done Done false
14507 4323 Acked Read 1 Sent Done Done false
14508 4324 Acked Read 6 Sent Done Done false
14509 4325 Acked Flush 0 Sent Done Done false
14510 4326 Acked Flush 0 Sent Done Done false
14511 4327 Acked Read 1 Sent Done Done false
14512 4328 Acked Write 2 Sent Done Done false
14513 4329 Acked Write 3 Sent Done Done false
14514 4330 Acked Read 10 Sent Done Done false
14515 4331 Acked Write 9 Sent Done Done false
14516 4332 Acked Flush 0 Sent Done Done false
14517 4333 Acked Read 6 Sent Done Done false
14518 4334 Acked Read 5 Sent Done Done false
14519 4335 Acked Flush 0 Sent Done Done false
14520 4336 Acked Read 3 Sent Done Done false
14521 4337 Acked Write 4 Sent Done Done false
14522 4338 Acked Read 7 Sent Done Done false
14523 4339 Acked Read 3 Sent Done Done false
14524 4340 Acked Write 4 Sent Done Done false
14525 4341 Acked Write 4 Sent Done Done false
14526 4342 Acked Read 1 Sent Done Done false
14527 4343 Acked Write 7 Sent Done Done false
14528 4344 Acked Read 8 Sent Done Done false
14529 4345 Acked Write 8 Sent Done Done false
14530 4346 Acked Read 3 Sent Done Done false
14531 4347 Acked Write 6 Sent Done Done false
14532 4348 Acked Write 6 Sent Done Done false
14533 4349 Acked Write 2 Sent Done Done false
14534 4350 Acked Read 4 Sent Done Done false
14535 4351 Acked Write 2 Sent Done Done false
14536 4352 Acked Write 1 Sent Done Done false
14537 4353 Acked Write 3 Sent Done Done false
14538 4354 Acked Write 7 Sent Done Done false
14539 4355 Acked Read 10 Sent Done Done false
14540 4356 Acked Write 8 Sent Done Done false
14541 4357 Acked Write 1 Sent Done Done false
14542 4358 Acked Write 10 Sent Done Done false
14543 4359 Acked Read 7 Sent Done Done false
14544 4360 Acked Read 9 Sent Done Done false
14545 4361 Acked Write 8 Sent Done Done false
14546 4362 Acked Write 3 Sent Done Done false
14547 4363 Acked Write 8 Sent Done Done false
14548 4364 Acked Read 5 Sent Done Done false
14549 4365 Acked Write 4 Sent Done Done false
14550 4366 Acked Read 3 Sent Done Done false
14551 4367 Acked Read 6 Sent Done Done false
14552 4368 Acked Read 10 Sent Done Done false
14553 4369 Acked Read 1 Sent Done Done false
14554 4370 Acked Write 6 Sent Done Done false
14555 4371 Acked Read 5 Sent Done Done false
14556 4372 Acked Read 4 Sent Done Done false
14557 4373 Acked Read 10 Sent Done Done false
14558 4374 Acked Read 3 Sent Done Done false
14559 4375 Acked Read 6 Sent Done Done false
14560 4376 Acked Write 3 Sent Done Done false
14561 4377 Acked Write 7 Sent Done Done false
14562 4378 Acked Read 5 Sent Done Done false
14563 4379 Acked Write 5 Sent Done Done false
14564 4380 Acked Write 4 Sent Done Done false
14565 4381 Acked Read 7 Sent Done Done false
14566 4382 Acked Read 9 Sent Done Done false
14567 4383 Acked Write 7 Sent Done Done false
14568 4384 Acked Read 8 Sent Done Done false
14569 4385 Acked Write 2 Sent Done Done false
14570 4386 Acked Flush 0 Sent Done Done false
14571 4387 Acked Read 2 Sent Done Done false
14572 4388 Acked Write 2 Sent Done Done false
14573 4389 Acked Write 1 Sent Done Done false
14574 4390 Acked Read 2 Sent Done Done false
14575 4391 Acked Read 4 Sent Done Done false
14576 4392 Acked Write 3 Sent Done Done false
14577 4393 Acked Write 10 Sent Done Done false
14578 4394 Acked Read 6 Sent Done Done false
14579 4395 Acked Read 7 Sent Done Done false
14580 4396 Acked Write 7 Sent Done Done false
14581 4397 Acked Write 3 Sent Done Done false
14582 4398 Acked Write 5 Sent Done Done false
14583 4399 Acked Write 1 Sent Done Done false
14584 4400 Acked Read 2 Sent Done Done false
14585 4401 Acked Write 9 Sent Done Done false
14586 4402 Acked Write 2 Sent Done Done false
14587 4403 Acked Read 6 Sent Done Done false
14588 4404 Acked Read 9 Sent Done Done false
14589 4405 Acked Flush 0 Sent Done Done false
14590 4406 Acked Write 9 Sent Done Done false
14591 4407 Acked Write 9 Sent Done Done false
14592 4408 Acked Write 2 Sent Done Done false
14593 4409 Acked Write 5 Sent Done Done false
14594 4410 Acked Write 4 Sent Done Done false
14595 4411 Acked Write 8 Sent Done Done false
14596 4412 Acked Write 9 Sent Done Done false
14597 4413 Acked Write 7 Sent Done Done false
14598 4414 Acked Read 1 Sent Done Done false
14599 4415 Acked Write 6 Sent Done Done false
14600 4416 Acked Write 3 Sent Done Done false
14601 4417 Acked Read 7 Sent Done Done false
14602 4418 Acked Write 6 Sent Done Done false
14603 4419 Acked Read 2 Sent Done Done false
14604 4420 Acked Read 1 Sent Done Done false
14605 4421 Acked Write 3 Sent Done Done false
14606 4422 Acked Write 1 Sent Done Done false
14607 4423 Acked Write 10 Sent Done Done false
14608 4424 Acked Read 2 Sent Done Done false
14609 4425 Acked Flush 0 Sent Done Done false
14610 4426 Acked Flush 0 Sent Done Done false
14611 4427 Acked Read 10 Sent Done Done false
14612 4428 Acked Read 3 Sent Done Done false
14613 4429 Acked Write 4 Sent Done Done false
14614 4430 Acked Read 4 Sent Done Done false
14615 4431 Acked Write 1 Sent Done Done false
14616 4432 Acked Read 1 Sent Done Done false
14617 4433 Acked Read 7 Sent Done Done false
14618 4434 Acked Read 7 Sent Done Done false
14619 4435 Acked Write 3 Sent Done Done false
14620 4436 Acked Flush 0 Sent Done Done false
14621 4437 Acked Write 2 Sent Done Done false
14622 4438 Acked Read 2 Sent Done Done false
14623 4439 Acked Write 9 Sent Done Done false
14624 4440 Acked Flush 0 Sent Done Done false
14625 4441 Acked Write 2 Sent Done Done false
14626 4442 Acked Write 7 Sent Done Done false
14627 4443 Acked Read 2 Sent Done Done false
14628 4444 Acked Read 1 Sent Done Done false
14629 4445 Acked Read 8 Sent Done Done false
14630 4446 Acked Write 4 Sent Done Done false
14631 4447 Acked Write 8 Sent Done Done false
14632 4448 Acked Read 6 Sent Done Done false
14633 4449 Acked Read 8 Sent Done Done false
14634 4450 Acked Read 8 Sent Done Done false
14635 4451 Acked Read 8 Sent Done Done false
14636 4452 Acked Write 3 Sent Done Done false
14637 4453 Acked Write 7 Sent Done Done false
14638 4454 Acked Read 3 Sent Done Done false
14639 4455 Acked Read 3 Sent Done Done false
14640 4456 Acked Write 10 Sent Done Done false
14641 4457 Acked Read 5 Sent Done Done false
14642 4458 Acked Read 2 Sent Done Done false
14643 4459 Acked Write 9 Sent Done Done false
14644 4460 Acked Write 3 Sent Done Done false
14645 4461 Acked Read 9 Sent Done Done false
14646 4462 Acked Read 8 Sent Done Done false
14647 4463 Acked Read 9 Sent Done Done false
14648 4464 Acked Read 6 Sent Done Done false
14649 4465 Acked Write 3 Sent Done Done false
14650 4466 Acked Write 7 Sent Done Done false
14651 4467 Acked Write 3 Sent Done Done false
14652 4468 Acked Read 9 Sent Done Done false
14653 4469 Acked Read 2 Sent Done Done false
14654 4470 Acked Flush 0 Sent Done Done false
14655 4471 Acked Read 8 Sent Done Done false
14656 4472 Acked Write 10 Sent Done Done false
14657 4473 Acked Write 7 Sent Done Done false
14658 4474 Acked Read 6 Sent Done Done false
14659 4475 Acked Write 10 Sent Done Done false
14660 4476 Acked Read 6 Sent Done Done false
14661 4477 Acked Flush 0 Sent Done Done false
14662 4478 Acked Read 4 Sent Done Done false
14663 4479 Acked Write 10 Sent Done Done false
14664 4480 Acked Flush 0 Sent Done Done false
14665 4481 Acked Write 2 Sent Done Done false
14666 4482 Acked Read 7 Sent Done Done false
14667 4483 Acked Read 1 Sent Done Done false
14668 4484 Acked Read 1 Sent Done Done false
14669 4485 Acked Read 9 Sent Done Done false
14670 4486 Acked Read 7 Sent Done Done false
14671 4487 Acked Write 10 Sent Done Done false
14672 4488 Acked Write 7 Sent Done Done false
14673 4489 Acked Write 4 Sent Done Done false
14674 4490 Acked Write 8 Sent Done Done false
14675 4491 Acked Flush 0 Sent Done Done false
14676 4492 Acked Read 6 Sent Done Done false
14677 4493 Acked Read 5 Sent Done Done false
14678 4494 Acked Flush 0 Sent Done Done false
14679 4495 Acked Read 6 Sent Done Done false
14680 4496 Acked Write 3 Sent Done Done false
14681 4497 Acked Read 5 Sent Done Done false
14682 4498 Acked Write 6 Sent Done Done false
14683 4499 Acked Write 6 Sent Done Done false
14684 4500 Acked Flush 0 Sent Done Done false
14685 4501 Acked Read 9 Sent Done Done false
14686 4502 Acked Read 9 Sent Done Done false
14687 4503 Acked Read 8 Sent Done Done false
14688 4504 Acked Write 8 Sent Done Done false
14689 4505 Acked Read 7 Sent Done Done false
14690 4506 Acked Read 5 Sent Done Done false
14691 4507 Acked Write 3 Sent Done Done false
14692 4508 Acked Read 6 Sent Done Done false
14693 4509 Acked Read 1 Sent Done Done false
14694 4510 Acked Read 1 Sent Done Done false
14695 4511 Acked Read 3 Sent Done Done false
14696 4512 Acked Read 2 Sent Done Done false
14697 4513 Acked Read 5 Sent Done Done false
14698 4514 Acked Write 8 Sent Done Done false
14699 4515 Acked Write 9 Sent Done Done false
14700 4516 Acked Read 2 Sent Done Done false
14701 4517 Acked Read 5 Sent Done Done false
14702 4518 Acked Read 8 Sent Done Done false
14703 4519 Acked Flush 0 Sent Done Done false
14704 4520 Acked Write 5 Sent Done Done false
14705 4521 Acked Read 3 Sent Done Done false
14706 4522 Acked Read 5 Sent Done Done false
14707 4523 Acked Flush 0 Sent Done Done false
14708 4524 Acked Write 4 Sent Done Done false
14709 4525 Acked Write 8 Sent Done Done false
14710 4526 Acked Read 5 Sent Done Done false
14711 4527 Acked Write 10 Sent Done Done false
14712 4528 Acked Read 4 Sent Done Done false
14713 4529 Acked Flush 0 Sent Done Done false
14714 4530 Acked Write 8 Sent Done Done false
14715 4531 Acked Flush 0 Sent Done Done false
14716 4532 Acked Read 2 Sent Done Done false
14717 4533 Acked Write 6 Sent Done Done false
14718 4534 Acked Flush 0 Sent Done Done false
14719 4535 Acked Write 7 Sent Done Done false
14720 4536 Acked Write 2 Sent Done Done false
14721 4537 Acked Read 2 Sent Done Done false
14722 4538 Acked Write 2 Sent Done Done false
14723 4539 Acked Read 4 Sent Done Done false
14724 4540 Acked Write 8 Sent Done Done false
14725 4541 Acked Write 6 Sent Done Done false
14726 4542 Acked Write 6 Sent Done Done false
14727 4543 Acked Read 3 Sent Done Done false
14728 4544 Acked Read 7 Sent Done Done false
14729 4545 Acked Read 4 Sent Done Done false
14730 4546 Acked Read 7 Sent Done Done false
14731 4547 Acked Write 5 Sent Done Done false
14732 4548 Acked Write 7 Sent Done Done false
14733 4549 Acked Write 10 Sent Done Done false
14734 4550 Acked Read 2 Sent Done Done false
14735 4551 Acked Read 10 Sent Done Done false
14736 4552 Acked Read 8 Sent Done Done false
14737 4553 Acked Read 8 Sent Done Done false
14738 4554 Acked Read 10 Sent Done Done false
14739 4555 Acked Write 7 Sent Done Done false
14740 4556 Acked Flush 0 Sent Done Done false
14741 4557 Acked Read 1 Sent Done Done false
14742 4558 Acked Flush 0 Sent Done Done false
14743 4559 Acked Read 1 Sent Done Done false
14744 4560 Acked Write 2 Sent Done Done false
14745 4561 Acked Read 10 Sent Done Done false
14746 4562 Acked Write 5 Sent Done Done false
14747 4563 Acked Write 1 Sent Done Done false
14748 4564 Acked Read 2 Sent Done Done false
14749 4565 Acked Read 4 Sent Done Done false
14750 4566 Acked Write 9 Sent Done Done false
14751 4567 Acked Write 2 Sent Done Done false
14752 4568 Acked Write 4 Sent Done Done false
14753 4569 Acked Write 6 Sent Done Done false
14754 4570 Acked Read 2 Sent Done Done false
14755 4571 Acked Read 9 Sent Done Done false
14756 4572 Acked Read 10 Sent Done Done false
14757 4573 Acked Write 2 Sent Done Done false
14758 4574 Acked Write 1 Sent Done Done false
14759 4575 Acked Write 1 Sent Done Done false
14760 4576 Acked Write 4 Sent Done Done false
14761 4577 Acked Write 8 Sent Done Done false
14762 4578 Acked Flush 0 Sent Done Done false
14763 4579 Acked Read 6 Sent Done Done false
14764 4580 Acked Write 2 Sent Done Done false
14765 4581 Acked Write 9 Sent Done Done false
14766 4582 Acked Read 5 Sent Done Done false
14767 4583 Acked Read 3 Sent Done Done false
14768 4584 Acked Write 2 Sent Done Done false
14769 4585 Acked Read 5 Sent Done Done false
14770 4586 Acked Write 10 Sent Done Done false
14771 4587 Acked Read 7 Sent Done Done false
14772 4588 Acked Read 1 Sent Done Done false
14773 4589 Acked Read 4 Sent Done Done false
14774 4590 Acked Read 10 Sent Done Done false
14775 4591 Acked Write 9 Sent Done Done false
14776 4592 Acked Read 7 Sent Done Done false
14777 4593 Acked Read 9 Sent Done Done false
14778 4594 Acked Read 7 Sent Done Done false
14779 4595 Acked Write 10 Sent Done Done false
14780 4596 Acked Write 1 Sent Done Done false
14781 4597 Acked Read 1 Sent Done Done false
14782 4598 Acked Write 8 Sent Done Done false
14783 4599 Acked Read 3 Sent Done Done false
14784 4600 Acked Write 4 Sent Done Done false
14785 4601 Acked Read 6 Sent Done Done false
14786 4602 Acked Write 3 Sent Done Done false
14787 4603 Acked Read 2 Sent Done Done false
14788 4604 Acked Read 1 Sent Done Done false
14789 4605 Acked Write 7 Sent Done Done false
14790 4606 Acked Write 2 Sent Done Done false
14791 4607 Acked Write 9 Sent Done Done false
14792 4608 Acked Read 6 Sent Done Done false
14793 4609 Acked Read 5 Sent Done Done false
14794 4610 Acked Read 8 Sent Done Done false
14795 4611 Acked Write 10 Sent Done Done false
14796 4612 Acked Read 2 Sent Done Done false
14797 4613 Acked Write 8 Sent Done Done false
14798 4614 Acked Flush 0 Sent Done Done false
14799 4615 Acked Write 7 Sent Done Done false
14800 4616 Acked Read 5 Sent Done Done false
14801 4617 Acked Read 7 Sent Done Done false
14802 4618 Acked Read 7 Sent Done Done false
14803 4619 Acked Read 3 Sent Done Done false
14804 4620 Acked Read 2 Sent Done Done false
14805 4621 Acked Write 5 Sent Done Done false
14806 STATES DS:0 DS:1 DS:2 TOTAL
14807 Sent 301 0 0 301
14808 Done 0 301 301 602
14809 Skipped 0 0 0 0
14810 Error 0 0 0 0
14811 Last Flush: Some(JobId(4320)) Some(JobId(4614)) Some(JobId(4614))
14812 Downstairs last five completed: 4320 4319 4318 4317 4316
14813 Upstairs last five completed: 4621 4620 4619 4618 4617
14814 CLIENT: Up:0 ds:301 act:2
148152025-02-25T20:49:52.366ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148162025-02-25T20:49:53.368ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148172025-02-25T20:49:53.433ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148182025-02-25T20:49:53.433ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49004) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148192025-02-25T20:49:53.434ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148202025-02-25T20:49:53.435ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148212025-02-25T20:49:53.435ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148222025-02-25T20:49:53.435ZINFOcrucible: send last flush ID to this DS: Some(JobId(4320)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148232025-02-25T20:49:53.435ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148242025-02-25T20:49:53.435ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4320)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148252025-02-25T20:49:53.435ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4320)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
14826 ----------------------------------------------------------------
14827 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14828 Downstairs last five completed: 4622 4621 4620 4619 4618
14829 Upstairs last five completed: 4622 4621 4620 4619 4618
14830 CLIENT: Up:0 ds:0 act:3
14831 Replay: All jobs finished
14832 Main task has work to do, go find it
14833 got dsc Stop(1)
14834 stop 1
14835 Feb 25 20:49:55.680 INFO request completed, latency_us: 404, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c4c304e3-15fa-4123-aebb-de54bbb3a791, [8820] Got stop action so:false kr:false
14836 remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
14837 Feb 25 20:49:55.680 INFO request completed, latency_us: 111, response_code: 200, uri: /state/cid/1, method: GET, req_id: 45c288b9-921d-4c0f-b1af-9d4c62dbc584, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
148382025-02-25T20:49:55.680ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148392025-02-25T20:49:55.680ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148402025-02-25T20:49:55.680ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
148412025-02-25T20:49:55.680ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
14842 [8820] Exited with: ExitStatus(unix_wait_status(9))
14843 [8820][1] reports Exit
14844 Feb 25 20:49:59.683 INFO request completed, latency_us: 242, response_code: 200, uri: /state/cid/1, method: GET, req_id: 89fc30f4-53c3-4084-81b9-f741d4ad9e6c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
14845 001/300 Write block 3690 len 16384 data: 2 2 2 2
14846 002/300 Read block 2839 len 16384
14847 003/300 Read block 2691 len 20480
14848 004/300 Write block 721 len 24576 data: 5 5 4 4 5 5
14849 005/300 Read block 2871 len 4096
14850 006/300 Flush
14851 007/300 Write block 2302 len 8192 data: 2 2
14852 008/300 Write block 1627 len 40960 data: 4 4 5 5 5 5 3 4 4 3
14853 009/300 Write block 4490 len 4096 data: 3
14854 010/300 Read block 936 len 4096
14855 011/300 Write block 1902 len 32768 data: 3 3 2 2 2 2 2 3
14856 012/300 Read block 4737 len 20480
14857 013/300 Write block 985 len 24576 data: 7 7 6 4 4 3
14858 014/300 Read block 2508 len 12288
14859 015/300 Read block 4287 len 12288
14860 016/300 Read block 3903 len 8192
14861 017/300 Read block 3783 len 24576
14862 018/300 Read block 3878 len 20480
14863 019/300 Read block 1838 len 16384
14864 020/300 Flush
14865 021/300 Read block 4562 len 40960
14866 022/300 Read block 544 len 40960
14867 023/300 Read block 4787 len 16384
14868 024/300 Read block 3544 len 36864
14869 025/300 Read block 4455 len 8192
14870 026/300 Read block 773 len 32768
14871 027/300 Read block 4325 len 20480
14872 028/300 Write block 1284 len 40960 data: 2 3 3 3 3 3 3 3 3 3
14873 029/300 Write block 2053 len 16384 data: 2 2 3 4
14874 030/300 Read block 1616 len 4096
14875 031/300 Write block 2656 len 16384 data: 2 3 2 3
14876 032/300 Read block 460 len 36864
14877 033/300 Read block 3617 len 12288
14878 034/300 Read block 1587 len 28672
14879 035/300 Write block 897 len 36864 data: 2 2 3 3 3 3 3 3 3
14880 036/300 Write block 1794 len 20480 data: 5 5 6 7 7
14881 037/300 Write block 3998 len 40960 data: 4 4 4 4 4 4 2 2 2 2
14882 038/300 Write block 1341 len 28672 data: 2 3 3 3 4 4 5
14883 039/300 Write block 3112 len 40960 data: 4 4 3 3 4 3 3 3 3 4
14884 040/300 Flush
14885 041/300 Read block 2319 len 28672
14886 042/300 Read block 1550 len 28672
14887 043/300 Write block 2072 len 20480 data: 2 2 2 2 3
14888 044/300 Write block 1898 len 20480 data: 3 3 3 3 4
14889 045/300 Read block 2168 len 8192
14890 046/300 Write block 4451 len 4096 data: 2
14891 047/300 Write block 613 len 36864 data: 2 2 2 2 3 3 3 3 3
14892 048/300 Read block 66 len 28672
14893 049/300 Read block 2321 len 28672
14894 050/300 Write block 389 len 36864 data: 4 4 4 3 3 3 3 3 5
14895 051/300 Write block 3127 len 20480 data: 4 3 3 3 3
14896 052/300 Read block 3224 len 32768
14897 053/300 Read block 608 len 40960
14898 054/300 Write block 3237 len 40960 data: 4 4 3 3 3 6 5 6 5 5
14899 055/300 Flush
14900 056/300 Write block 637 len 8192 data: 2 2
14901 057/300 Read block 3755 len 24576
14902 058/300 Write block 1548 len 16384 data: 3 3 3 3
14903 059/300 Read block 3968 len 8192
14904 060/300 Read block 2660 len 28672
14905 061/300 Write block 3542 len 40960 data: 3 3 2 2 2 2 2 2 2 2
14906 062/300 Read block 3718 len 12288
14907 063/300 Write block 1681 len 16384 data: 3 3 2 2
14908 064/300 Write block 2363 len 20480 data: 4 4 4 4 4
14909 065/300 Read block 1124 len 40960
14910 066/300 Read block 2441 len 32768
14911 067/300 Write block 2799 len 24576 data: 3 6 5 4 5 5
14912 068/300 Read block 2551 len 16384
14913 069/300 Write block 2927 len 20480 data: 3 3 3 3 2
14914 070/300 Read block 706 len 20480
14915 071/300 Read block 2793 len 12288
14916 072/300 Read block 1517 len 28672
14917 073/300 Write block 1247 len 24576 data: 3 3 3 2 2 3
14918 074/300 Write block 1956 len 28672 data: 3 3 4 4 3 4 4
14919 075/300 Write block 1942 len 40960 data: 2 2 2 2 3 4 5 4 4 4
14920 076/300 Write block 2880 len 16384 data: 2 3 4 4
14921 077/300 Flush
14922 078/300 Read block 3233 len 16384
14923 079/300 Write block 2921 len 8192 data: 6 6
14924 080/300 Write block 1504 len 32768 data: 2 4 5 4 3 3 2 2
14925 081/300 Write block 4804 len 36864 data: 2 2 2 2 2 2 2 2 2
14926 082/300 Write block 1998 len 36864 data: 3 3 4 3 3 3 3 3 2
14927 083/300 Read block 4966 len 16384
14928 084/300 Read block 1249 len 16384
14929 085/300 Write block 1428 len 8192 data: 2 2
14930 086/300 Read block 2283 len 36864
14931 087/300 Read block 2194 len 8192
14932 088/300 Flush
14933 089/300 Flush
14934 090/300 Read block 3163 len 40960
14935 091/300 Read block 4653 len 28672
14936 092/300 Read block 2282 len 20480
14937 093/300 Read block 1503 len 20480
14938 094/300 Write block 2736 len 32768 data: 2 3 4 4 4 4 5 6
14939 095/300 Write block 169 len 36864 data: 2 2 3 3 3 3 3 4 3
14940 096/300 Read block 790 len 20480
14941 097/300 Flush
14942 098/300 Read block 1595 len 28672
14943 099/300 Read block 945 len 40960
14944 100/300 Read block 4065 len 8192
14945 101/300 Read block 4271 len 32768
14946 102/300 Write block 1008 len 16384 data: 5 5 4 4
14947 103/300 Read block 4492 len 20480
14948 104/300 Flush
14949 105/300 Write block 3409 len 20480 data: 2 2 2 2 2
14950 106/300 Write block 2215 len 40960 data: 2 2 2 2 2 2 2 3 3 3
14951 107/300 Read block 4170 len 40960
14952 108/300 Read block 3470 len 28672
14953 109/300 Read block 4367 len 16384
14954 110/300 Read block 3324 len 24576
14955 111/300 Write block 9 len 32768 data: 3 3 3 2 2 2 2 2
14956 112/300 Read block 3493 len 4096
14957 113/300 Read block 1377 len 40960
14958 114/300 Write block 3633 len 16384 data: 5 5 6 6
14959 115/300 Write block 3850 len 24576 data: 3 3 4 4 5 5
14960 116/300 Flush
14961 117/300 Read block 4263 len 8192
14962 118/300 Write block 2828 len 12288 data: 4 4 4
14963 119/300 Read block 149 len 4096
14964 120/300 Write block 76 len 32768 data: 4 4 4 3 3 4 4 4
14965 121/300 Write block 3513 len 32768 data: 3 3 3 3 3 3 2 3
14966 122/300 Write block 934 len 16384 data: 4 2 3 3
14967 123/300 Write block 1510 len 40960 data: 3 3 2 2 2 2 2 2 2 2
14968 124/300 Read block 3749 len 16384
14969 125/300 Read block 30 len 20480
14970 126/300 Write block 2396 len 24576 data: 3 3 2 2 2 4
14971 127/300 Write block 3175 len 24576 data: 4 3 4 3 3 4
14972 128/300 Write block 2662 len 8192 data: 3 3
14973 129/300 Read block 3768 len 16384
14974 130/300 Flush
14975 131/300 Write block 1114 len 40960 data: 3 3 4 4 4 3 4 5 4 4
14976 132/300 Write block 50 len 16384 data: 3 3 3 2
14977 133/300 Write block 822 len 20480 data: 3 2 2 2 2
14978 134/300 Write block 1387 len 28672 data: 3 4 4 4 4 4 5
14979 135/300 Read block 642 len 28672
14980 136/300 Flush
14981 137/300 Write block 1455 len 16384 data: 5 5 6 6
14982 138/300 Read block 4643 len 24576
14983 139/300 Write block 477 len 40960 data: 2 2 2 2 3 3 4 4 4 4
14984 140/300 Flush
14985 141/300 Write block 993 len 4096 data: 4
14986 142/300 Read block 4639 len 36864
14987 143/300 Flush
14988 144/300 Write block 2215 len 12288 data: 3 3 3
14989 145/300 Read block 3565 len 20480
14990 146/300 Flush
14991 147/300 Write block 413 len 20480 data: 5 6 5 5 4
14992 148/300 Write block 2883 len 8192 data: 5 2
14993 149/300 Write block 3620 len 4096 data: 4
14994 150/300 Read block 4660 len 28672
14995 151/300 Read block 1413 len 28672
14996 152/300 Write block 1613 len 20480 data: 5 5 4 5 6
14997 153/300 Write block 501 len 28672 data: 4 4 4 3 2 2 3
14998 154/300 Read block 2625 len 32768
14999 155/300 Read block 1790 len 28672
15000 156/300 Flush
15001 157/300 Write block 4222 len 32768 data: 3 3 3 2 4 4 4 4
15002 158/300 Flush
15003 159/300 Read block 434 len 20480
15004 160/300 Read block 1220 len 32768
15005 161/300 Flush
15006 162/300 Write block 2545 len 36864 data: 4 4 3 3 4 4 4 5 5
15007 163/300 Write block 4546 len 20480 data: 2 2 2 2 2
15008 164/300 Read block 3041 len 36864
15009 165/300 Read block 724 len 8192
15010 166/300 Write block 2843 len 20480 data: 4 4 5 3 3
15011 167/300 Write block 4139 len 8192 data: 3 3
15012 168/300 Read block 1992 len 4096
15013 169/300 Read block 1642 len 20480
15014 170/300 Read block 778 len 32768
15015 171/300 Write block 960 len 4096 data: 5
15016 172/300 Read block 3724 len 20480
15017 173/300 Write block 588 len 12288 data: 3 2 3
15018 174/300 Read block 3 len 24576
15019 175/300 Flush
15020 176/300 Read block 2123 len 28672
15021 177/300 Flush
15022 178/300 Write block 3887 len 28672 data: 4 4 4 4 4 4 3
15023 179/300 Write block 124 len 12288 data: 5 6 6
15024 180/300 Flush
15025 181/300 Read block 1978 len 12288
15026 182/300 Write block 4162 len 12288 data: 9 9 9
15027 183/300 Read block 3563 len 20480
15028 184/300 Read block 1618 len 20480
15029 185/300 Write block 1335 len 28672 data: 2 2 2 2 2 2 3
15030 186/300 Write block 3263 len 16384 data: 2 3 3 4
15031 187/300 Write block 2219 len 8192 data: 3 3
15032 188/300 Write block 1781 len 24576 data: 4 5 4 4 4 4
15033 189/300 Read block 3315 len 24576
15034 190/300 Read block 4343 len 32768
15035 191/300 Read block 3647 len 8192
15036 192/300 Read block 4721 len 16384
15037 193/300 Flush
15038 194/300 Read block 1753 len 8192
15039 195/300 Write block 4760 len 16384 data: 3 3 3 2
15040 196/300 Write block 2990 len 16384 data: 6 5 5 4
15041 197/300 Read block 4036 len 4096
15042 198/300 Read block 4797 len 36864
15043 199/300 Read block 4193 len 36864
15044 200/300 Flush
15045 201/300 Write block 4534 len 28672 data: 4 3 3 2 2 2 2
15046 202/300 Write block 2012 len 40960 data: 4 4 4 4 4 3 3 4 3 4
15047 203/300 Read block 56 len 12288
15048 204/300 Write block 2048 len 24576 data: 2 2 2 2 2 3
15049 205/300 Read block 4611 len 32768
15050 206/300 Read block 132 len 12288
15051 207/300 Read block 1333 len 20480
15052 208/300 Read block 3975 len 8192
15053 209/300 Read block 67 len 12288
15054 210/300 Write block 2783 len 4096 data: 2
15055 211/300 Flush
15056 212/300 Write block 3522 len 8192 data: 3 3
15057 213/300 Read block 2688 len 8192
15058 214/300 Read block 1586 len 36864
15059 215/300 Read block 3871 len 24576
15060 216/300 Read block 3096 len 12288
15061 217/300 Write block 1527 len 16384 data: 2 3 4 4
15062 218/300 Read block 843 len 28672
15063 219/300 Flush
15064 220/300 Write block 3255 len 4096 data: 4
15065 221/300 Write block 3649 len 12288 data: 4 4 2
15066 222/300 Write block 4742 len 16384 data: 2 2 2 3
15067 223/300 Write block 3986 len 28672 data: 3 2 3 4 4 4 4
15068 224/300 Write block 1846 len 8192 data: 2 2
15069 225/300 Read block 3327 len 8192
15070 226/300 Read block 2663 len 4096
15071 227/300 Write block 1648 len 8192 data: 2 3
15072 228/300 Read block 3880 len 16384
15073 229/300 Write block 2958 len 16384 data: 4 4 4 4
15074 230/300 Write block 2957 len 36864 data: 4 5 5 5 5 4 4 3 3
15075 231/300 Read block 222 len 20480
15076 232/300 Read block 4076 len 16384
15077 233/300 Write block 3252 len 12288 data: 5 5 4
15078 234/300 Read block 184 len 32768
15079 235/300 Read block 3536 len 40960
15080 236/300 Read block 3144 len 12288
15081 237/300 Write block 2568 len 28672 data: 4 3 3 3 2 2 2
15082 238/300 Flush
15083 239/300 Write block 1420 len 20480 data: 5 5 5 3 3
15084 240/300 Read block 2474 len 20480
15085 241/300 Write block 4400 len 36864 data: 5 3 3 3 3 3 4 3 3
15086 242/300 Write block 2009 len 8192 data: 7 7
15087 243/300 Write block 4291 len 4096 data: 3
15088 244/300 Read block 4086 len 40960
15089 245/300 Flush
15090 246/300 Write block 4095 len 8192 data: 3 3
15091 247/300 Read block 4227 len 32768
15092 248/300 Read block 2960 len 8192
15093 249/300 Write block 4125 len 40960 data: 4 4 3 3 3 3 3 2 3 3
15094 250/300 Flush
15095 251/300 Write block 4991 len 32768 data: 4 4 4 3 3 3 3 3
15096 252/300 Read block 1053 len 20480
15097 253/300 Read block 1160 len 24576
15098 254/300 Read block 4510 len 16384
15099 255/300 Read block 1705 len 12288
15100 256/300 Read block 1902 len 40960
15101 257/300 Write block 1085 len 20480 data: 2 2 2 3 3
15102 258/300 Write block 1225 len 20480 data: 3 3 4 2 2
15103 259/300 Write block 4169 len 20480 data: 5 5 5 5 7
15104 260/300 Read block 3566 len 28672
15105 261/300 Read block 1820 len 32768
15106 262/300 Read block 3492 len 32768
15107 263/300 Read block 2749 len 12288
15108 264/300 Read block 910 len 24576
15109 265/300 Read block 1583 len 24576
15110 266/300 Read block 1407 len 24576
15111 267/300 Read block 2808 len 20480
15112 268/300 Read block 3118 len 20480
15113 269/300 Write block 3122 len 28672 data: 4 4 5 5 6 5 4
15114 270/300 Read block 1707 len 28672
15115 271/300 Flush
15116 272/300 Write block 1320 len 12288 data: 4 5 5
15117 273/300 Read block 2042 len 28672
15118 274/300 Write block 2647 len 8192 data: 7 6
15119 275/300 Read block 2798 len 8192
15120 276/300 Read block 4272 len 40960
15121 277/300 Read block 4873 len 12288
15122 278/300 Read block 2115 len 20480
15123 279/300 Write block 1428 len 24576 data: 3 3 3 4 4 4
15124 280/300 Write block 1753 len 8192 data: 4 3
15125 281/300 Read block 242 len 8192
15126 282/300 Write block 3928 len 20480 data: 3 3 2 2 2
15127 283/300 Write block 4768 len 16384 data: 2 2 2 3
15128 284/300 Read block 2661 len 24576
15129 285/300 Read block 3283 len 32768
15130 286/300 Read block 26 len 32768
15131 287/300 Read block 4640 len 8192
15132 288/300 Write block 4807 len 8192 data: 3 3
15133 289/300 Write block 1043 len 40960 data: 4 4 4 4 3 3 3 3 2 4
15134 290/300 Write block 1228 len 12288 data: 3 3 3
15135 291/300 Write block 3014 len 20480 data: 4 4 4 3 2
15136 292/300 Read block 2046 len 24576
15137 293/300 Read block 2641 len 12288
15138 294/300 Write block 1320 len 8192 data: 5 6
15139 295/300 Read block 316 len 12288
15140 296/300 Write block 1192 len 16384 data: 4 5 4 3
15141 297/300 Read block 3353 len 28672
15142 298/300 Write block 3843 len 8192 data: 3 3
15143 299/300 Flush
15144 300/300 Read block 1371 len 8192
15145 301/300 Read block 4837 len 40960
15146 Main task has work to do, go find it
15147 got dsc Start(1)
15148 start 1
15149 Feb 25 20:49:59.925 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: fca7e2e9-49e0-4116-afa9-d34cb1c67cb3, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
15150 Replay: started 1, returned:Ok(())
15151 ----------------------------------------------------------------
15152 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15153 States: Active Offline Active
15154 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15155 4623 Acked Write 4 Done Sent Done false
15156 4624 Acked Read 4 Done Sent Done false
15157 4625 Acked Read 5 Done Sent Done false
15158 4626 Acked Write 6 Done Sent Done false
15159 4627 Acked Read 1 Done Sent Done false
15160 4628 Acked Flush 0 Done Sent Done false
15161 4629 Acked Write 2 Done Sent Done false
15162 4630 Acked Write 10 Done Sent Done false
15163 4631 Acked Write 1 Done Sent Done false
15164 4632 Acked Read 1 Done Sent Done false
15165 4633 Acked Write 8 Done Sent Done false
15166 4634 Acked Read 5 Done Sent Done false
15167 4635 Acked Write 6 Done Sent Done false
15168 4636 Acked Read 3 Done Sent Done false
15169 4637 Acked Read 3 Done Sent Done false
15170 4638 Acked Read 2 Done Sent Done false
15171 4639 Acked Read 6 Done Sent Done false
15172 4640 Acked Read 5 Done Sent Done false
15173 4641 Acked Read 4 Done Sent Done false
15174 4642 Acked Flush 0 Done Sent Done false
15175 4643 Acked Read 10 Done Sent Done false
15176 4644 Acked Read 10 Done Sent Done false
15177 4645 Acked Read 4 Done Sent Done false
15178 4646 Acked Read 9 Done Sent Done false
15179 4647 Acked Read 2 Done Sent Done false
15180 4648 Acked Read 8 Done Sent Done false
15181 4649 Acked Read 5 Done Sent Done false
15182 4650 Acked Write 10 Done Sent Done false
15183 4651 Acked Write 4 Done Sent Done false
15184 4652 Acked Read 1 Done Sent Done false
15185 4653 Acked Write 4 Done Sent Done false
15186 4654 Acked Read 9 Done Sent Done false
15187 4655 Acked Read 3 Done Sent Done false
15188 4656 Acked Read 7 Done Sent Done false
15189 4657 Acked Write 9 Done Sent Done false
15190 4658 Acked Write 5 Done Sent Done false
15191 4659 Acked Write 10 Done Sent Done false
15192 4660 Acked Write 7 Done Sent Done false
15193 4661 Acked Write 10 Done Sent Done false
15194 4662 Acked Flush 0 Done Sent Done false
15195 4663 Acked Read 7 Done Sent Done false
15196 4664 Acked Read 7 Done Sent Done false
15197 4665 Acked Write 5 Done Sent Done false
15198 4666 Acked Write 5 Done Sent Done false
15199 4667 Acked Read 2 Done Sent Done false
15200 4668 Acked Write 1 Done Sent Done false
15201 4669 Acked Write 9 Done Sent Done false
15202 4670 Acked Read 7 Done Sent Done false
15203 4671 Acked Read 7 Done Sent Done false
15204 4672 Acked Write 9 Done Sent Done false
15205 4673 Acked Write 5 Done Sent Done false
15206 4674 Acked Read 8 Done Sent Done false
15207 4675 Acked Read 10 Done Sent Done false
15208 4676 Acked Write 10 Done Sent Done false
15209 4677 Acked Flush 0 Done Sent Done false
15210 4678 Acked Write 2 Done Sent Done false
15211 4679 Acked Read 6 Done Sent Done false
15212 4680 Acked Write 4 Done Sent Done false
15213 4681 Acked Read 2 Done Sent Done false
15214 4682 Acked Read 7 Done Sent Done false
15215 4683 Acked Write 10 Done Sent Done false
15216 4684 Acked Read 3 Done Sent Done false
15217 4685 Acked Write 4 Done Sent Done false
15218 4686 Acked Write 5 Done Sent Done false
15219 4687 Acked Read 10 Done Sent Done false
15220 4688 Acked Read 8 Done Sent Done false
15221 4689 Acked Write 6 Done Sent Done false
15222 4690 Acked Read 4 Done Sent Done false
15223 4691 Acked Write 5 Done Sent Done false
15224 4692 Acked Read 5 Done Sent Done false
15225 4693 Acked Read 3 Done Sent Done false
15226 4694 Acked Read 7 Done Sent Done false
15227 4695 Acked Write 6 Done Sent Done false
15228 4696 Acked Write 7 Done Sent Done false
15229 4697 Acked Write 10 Done Sent Done false
15230 4698 Acked Write 4 Done Sent Done false
15231 4699 Acked Flush 0 Done Sent Done false
15232 4700 Acked Read 4 Done Sent Done false
15233 4701 Acked Write 2 Done Sent Done false
15234 4702 Acked Write 8 Done Sent Done false
15235 4703 Acked Write 9 Done Sent Done false
15236 4704 Acked Write 9 Done Sent Done false
15237 4705 Acked Read 4 Done Sent Done false
15238 4706 Acked Read 4 Done Sent Done false
15239 4707 Acked Write 2 Done Sent Done false
15240 4708 Acked Read 9 Done Sent Done false
15241 4709 Acked Read 2 Done Sent Done false
15242 4710 Acked Flush 0 Done Sent Done false
15243 4711 Acked Flush 0 Done Sent Done false
15244 4712 Acked Read 10 Done Sent Done false
15245 4713 Acked Read 7 Done Sent Done false
15246 4714 Acked Read 5 Done Sent Done false
15247 4715 Acked Read 5 Done Sent Done false
15248 4716 Acked Write 8 Done Sent Done false
15249 4717 Acked Write 9 Done Sent Done false
15250 4718 Acked Read 5 Done Sent Done false
15251 4719 Acked Flush 0 Done Sent Done false
15252 4720 Acked Read 7 Done Sent Done false
15253 4721 Acked Read 10 Done Sent Done false
15254 4722 Acked Read 2 Done Sent Done false
15255 4723 Acked Read 8 Done Sent Done false
15256 4724 Acked Write 4 Done Sent Done false
15257 4725 Acked Read 5 Done Sent Done false
15258 4726 Acked Flush 0 Done Sent Done false
15259 4727 Acked Write 5 Done Sent Done false
15260 4728 Acked Write 10 Done Sent Done false
15261 4729 Acked Read 10 Done Sent Done false
15262 4730 Acked Read 7 Done Sent Done false
15263 4731 Acked Read 4 Done Sent Done false
15264 4732 Acked Read 6 Done Sent Done false
15265 4733 Acked Write 8 Done Sent Done false
15266 4734 Acked Read 1 Done Sent Done false
15267 4735 Acked Read 10 Done Sent Done false
15268 4736 Acked Write 4 Done Sent Done false
15269 4737 Acked Write 6 Done Sent Done false
15270 4738 Acked Flush 0 Done Sent Done false
15271 4739 Acked Read 2 Done Sent Done false
15272 4740 Acked Write 3 Done Sent Done false
15273 4741 Acked Read 1 Done Sent Done false
15274 4742 Acked Write 8 Done Sent Done false
15275 4743 Acked Write 8 Done Sent Done false
15276 4744 Acked Write 4 Done Sent Done false
15277 4745 Acked Write 10 Done Sent Done false
15278 4746 Acked Read 4 Done Sent Done false
15279 4747 Acked Read 5 Done Sent Done false
15280 4748 Acked Write 6 Done Sent Done false
15281 4749 Acked Write 6 Done Sent Done false
15282 4750 Acked Write 2 Done Sent Done false
15283 4751 Acked Read 4 Done Sent Done false
15284 4752 Acked Flush 0 Done Sent Done false
15285 4753 Acked Write 10 Done Sent Done false
15286 4754 Acked Write 4 Done Sent Done false
15287 4755 Acked Write 5 Done Sent Done false
15288 4756 Acked Write 7 Done Sent Done false
15289 4757 Acked Read 7 Done Sent Done false
15290 4758 Acked Flush 0 Done Sent Done false
15291 4759 Acked Write 4 Done Sent Done false
15292 4760 Acked Read 6 Done Sent Done false
15293 4761 Acked Write 10 Done Sent Done false
15294 4762 Acked Flush 0 Done Sent Done false
15295 4763 Acked Write 1 Done Sent Done false
15296 4764 Acked Read 9 Done Sent Done false
15297 4765 Acked Flush 0 Done Sent Done false
15298 4766 Acked Write 3 Done Sent Done false
15299 4767 Acked Read 5 Done Sent Done false
15300 4768 Acked Flush 0 Done Sent Done false
15301 4769 Acked Write 5 Done Sent Done false
15302 4770 Acked Write 2 Done Sent Done false
15303 4771 Acked Write 1 Done Sent Done false
15304 4772 Acked Read 7 Done Sent Done false
15305 4773 Acked Read 7 Done Sent Done false
15306 4774 Acked Write 5 Done Sent Done false
15307 4775 Acked Write 7 Done Sent Done false
15308 4776 Acked Read 8 Done Sent Done false
15309 4777 Acked Read 7 Done Sent Done false
15310 4778 Acked Flush 0 Done Sent Done false
15311 4779 Acked Write 8 Done Sent Done false
15312 4780 Acked Flush 0 Done Sent Done false
15313 4781 Acked Read 5 Done Sent Done false
15314 4782 Acked Read 8 Done Sent Done false
15315 4783 Acked Flush 0 Done Sent Done false
15316 4784 Acked Write 9 Done Sent Done false
15317 4785 Acked Write 5 Done Sent Done false
15318 4786 Acked Read 9 Done Sent Done false
15319 4787 Acked Read 2 Done Sent Done false
15320 4788 Acked Write 5 Done Sent Done false
15321 4789 Acked Write 2 Done Sent Done false
15322 4790 Acked Read 1 Done Sent Done false
15323 4791 Acked Read 5 Done Sent Done false
15324 4792 Acked Read 8 Done Sent Done false
15325 4793 Acked Write 1 Done Sent Done false
15326 4794 Acked Read 5 Done Sent Done false
15327 4795 Acked Write 3 Done Sent Done false
15328 4796 Acked Read 6 Done Sent Done false
15329 4797 Acked Flush 0 Done Sent Done false
15330 4798 Acked Read 7 Done Sent Done false
15331 4799 Acked Flush 0 Done Sent Done false
15332 4800 Acked Write 7 Done Sent Done false
15333 4801 Acked Write 3 Done Sent Done false
15334 4802 Acked Flush 0 Done Sent Done false
15335 4803 Acked Read 3 Done Sent Done false
15336 4804 Acked Write 3 Done Sent Done false
15337 4805 Acked Read 5 Done Sent Done false
15338 4806 Acked Read 5 Done Sent Done false
15339 4807 Acked Write 7 Done Sent Done false
15340 4808 Acked Write 4 Done Sent Done false
15341 4809 Acked Write 2 Done Sent Done false
15342 4810 Acked Write 6 Done Sent Done false
15343 4811 Acked Read 6 Done Sent Done false
15344 4812 Acked Read 8 Done Sent Done false
15345 4813 Acked Read 2 Done Sent Done false
15346 4814 Acked Read 4 Done Sent Done false
15347 4815 Acked Flush 0 Done Sent Done false
15348 4816 Acked Read 2 Done Sent Done false
15349 4817 Acked Write 4 Done Sent Done false
15350 4818 Acked Write 4 Done Sent Done false
15351 4819 Acked Read 1 Done Sent Done false
15352 4820 Acked Read 9 Done Sent Done false
15353 4821 Acked Read 9 Done Sent Done false
15354 4822 Acked Flush 0 Done Sent Done false
15355 4823 Acked Write 7 Done Sent Done false
15356 4824 Acked Write 10 Done Sent Done false
15357 4825 Acked Read 3 Done Sent Done false
15358 4826 Acked Write 6 Done Sent Done false
15359 4827 Acked Read 8 Done Sent Done false
15360 4828 Acked Read 3 Done Sent Done false
15361 4829 Acked Read 5 Done Sent Done false
15362 4830 Acked Read 2 Done Sent Done false
15363 4831 Acked Read 3 Done Sent Done false
15364 4832 Acked Write 1 Done Sent Done false
15365 4833 Acked Flush 0 Done Sent Done false
15366 4834 Acked Write 2 Done Sent Done false
15367 4835 Acked Read 2 Done Sent Done false
15368 4836 Acked Read 9 Done Sent Done false
15369 4837 Acked Read 6 Done Sent Done false
15370 4838 Acked Read 3 Done Sent Done false
15371 4839 Acked Write 4 Done Sent Done false
15372 4840 Acked Read 7 Done Sent Done false
15373 4841 Acked Flush 0 Done Sent Done false
15374 4842 Acked Write 1 Done Sent Done false
15375 4843 Acked Write 3 Done Sent Done false
15376 4844 Acked Write 4 Done Sent Done false
15377 4845 Acked Write 7 Done Sent Done false
15378 4846 Acked Write 2 Done Sent Done false
15379 4847 Acked Read 2 Done Sent Done false
15380 4848 Acked Read 1 Done Sent Done false
15381 4849 Acked Write 2 Done Sent Done false
15382 4850 Acked Read 4 Done Sent Done false
15383 4851 Acked Write 4 Done Sent Done false
15384 4852 Acked Write 9 Done Sent Done false
15385 4853 Acked Read 5 Done Sent Done false
15386 4854 Acked Read 4 Done Sent Done false
15387 4855 Acked Write 3 Done Sent Done false
15388 4856 Acked Read 8 Done Sent Done false
15389 4857 Acked Read 10 Done Sent Done false
15390 4858 Acked Read 3 Done Sent Done false
15391 4859 Acked Write 7 Done Sent Done false
15392 4860 Acked Flush 0 Done Sent Done false
15393 4861 Acked Write 5 Done Sent Done false
15394 4862 Acked Read 5 Done Sent Done false
15395 4863 Acked Write 9 Done Sent Done false
15396 4864 Acked Write 2 Done Sent Done false
15397 4865 Acked Write 1 Done Sent Done false
15398 4866 Acked Read 10 Done Sent Done false
15399 4867 Acked Flush 0 Done Sent Done false
15400 4868 Acked Write 2 Done Sent Done false
15401 4869 Acked Read 8 Done Sent Done false
15402 4870 Acked Read 2 Done Sent Done false
15403 4871 Acked Write 10 Done Sent Done false
15404 4872 Acked Flush 0 Done Sent Done false
15405 4873 Acked Write 8 Done Sent Done false
15406 4874 Acked Read 5 Done Sent Done false
15407 4875 Acked Read 6 Done Sent Done false
15408 4876 Acked Read 4 Done Sent Done false
15409 4877 Acked Read 3 Done Sent Done false
15410 4878 Acked Read 10 Done Sent Done false
15411 4879 Acked Write 5 Done Sent Done false
15412 4880 Acked Write 5 Done Sent Done false
15413 4881 Acked Write 5 Done Sent Done false
15414 4882 Acked Read 7 Done Sent Done false
15415 4883 Acked Read 8 Done Sent Done false
15416 4884 Acked Read 8 Done Sent Done false
15417 4885 Acked Read 3 Done Sent Done false
15418 4886 Acked Read 6 Done Sent Done false
15419 4887 Acked Read 6 Done Sent Done false
15420 4888 Acked Read 6 Done Sent Done false
15421 4889 Acked Read 5 Done Sent Done false
15422 4890 Acked Read 5 Done Sent Done false
15423 4891 Acked Write 7 Done Sent Done false
15424 4892 Acked Read 7 Done Sent Done false
15425 4893 Acked Flush 0 Done Sent Done false
15426 4894 Acked Write 3 Done Sent Done false
15427 4895 Acked Read 7 Done Sent Done false
15428 4896 Acked Write 2 Done Sent Done false
15429 4897 Acked Read 2 Done Sent Done false
15430 4898 Acked Read 10 Done Sent Done false
15431 4899 Acked Read 3 Done Sent Done false
15432 4900 Acked Read 5 Done Sent Done false
15433 4901 Acked Write 6 Done Sent Done false
15434 4902 Acked Write 2 Done Sent Done false
15435 4903 Acked Read 2 Done Sent Done false
15436 4904 Acked Write 5 Done Sent Done false
15437 4905 Acked Write 4 Done Sent Done false
15438 4906 Acked Read 6 Done Sent Done false
15439 4907 Acked Read 8 Done Sent Done false
15440 4908 Acked Read 8 Done Sent Done false
15441 4909 Acked Read 2 Done Sent Done false
15442 4910 Acked Write 2 Done Sent Done false
15443 4911 Acked Write 10 Done Sent Done false
15444 4912 Acked Write 3 Done Sent Done false
15445 4913 Acked Write 5 Done Sent Done false
15446 4914 Acked Read 6 Done Sent Done false
15447 4915 Acked Read 3 Done Sent Done false
15448 4916 Acked Write 2 Done Sent Done false
15449 4917 Acked Read 3 Done Sent Done false
15450 4918 Acked Write 4 Done Sent Done false
15451 4919 Acked Read 7 Done Sent Done false
15452 4920 Acked Write 2 Done Sent Done false
15453 4921 Acked Flush 0 Done Sent Done false
15454 4922 Acked Read 2 Done Sent Done false
15455 4923 Acked Read 10 Done Sent Done false
15456 STATES DS:0 DS:1 DS:2 TOTAL
15457 Sent 0 301 0 301
15458 Done 301 0 301 602
15459 Skipped 0 0 0 0
15460 Error 0 0 0 0
15461 Last Flush: Some(JobId(4921)) Some(JobId(4622)) Some(JobId(4921))
15462 Downstairs last five completed: 4622 4621 4620 4619 4618
15463 Upstairs last five completed: 4923 4922 4921 4920 4919
15464 CLIENT: Up:0 ds:301 act:2
154652025-02-25T20:50:00.427ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
154662025-02-25T20:50:01.379ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
15467 [8820] got start action
15468 [8820] I am going to restart
15469 Starting downstairs at port 8820
15470 Make output file at "/tmp/dsc/downstairs-8820.txt"
15471 [8820][1] reports Starting
15472 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
15473 [8820][1] reports Running
154742025-02-25T20:50:01.880ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
154752025-02-25T20:50:02.381ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
154762025-02-25T20:50:03.383ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
15477 ----------------------------------------------------------------
15478 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15479 States: Active Offline Active
15480 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15481 4623 Acked Write 4 Done Sent Done false
15482 4624 Acked Read 4 Done Sent Done false
15483 4625 Acked Read 5 Done Sent Done false
15484 4626 Acked Write 6 Done Sent Done false
15485 4627 Acked Read 1 Done Sent Done false
15486 4628 Acked Flush 0 Done Sent Done false
15487 4629 Acked Write 2 Done Sent Done false
15488 4630 Acked Write 10 Done Sent Done false
15489 4631 Acked Write 1 Done Sent Done false
15490 4632 Acked Read 1 Done Sent Done false
15491 4633 Acked Write 8 Done Sent Done false
15492 4634 Acked Read 5 Done Sent Done false
15493 4635 Acked Write 6 Done Sent Done false
15494 4636 Acked Read 3 Done Sent Done false
15495 4637 Acked Read 3 Done Sent Done false
15496 4638 Acked Read 2 Done Sent Done false
15497 4639 Acked Read 6 Done Sent Done false
15498 4640 Acked Read 5 Done Sent Done false
15499 4641 Acked Read 4 Done Sent Done false
15500 4642 Acked Flush 0 Done Sent Done false
15501 4643 Acked Read 10 Done Sent Done false
15502 4644 Acked Read 10 Done Sent Done false
15503 4645 Acked Read 4 Done Sent Done false
15504 4646 Acked Read 9 Done Sent Done false
15505 4647 Acked Read 2 Done Sent Done false
15506 4648 Acked Read 8 Done Sent Done false
15507 4649 Acked Read 5 Done Sent Done false
15508 4650 Acked Write 10 Done Sent Done false
15509 4651 Acked Write 4 Done Sent Done false
15510 4652 Acked Read 1 Done Sent Done false
15511 4653 Acked Write 4 Done Sent Done false
15512 4654 Acked Read 9 Done Sent Done false
15513 4655 Acked Read 3 Done Sent Done false
15514 4656 Acked Read 7 Done Sent Done false
15515 4657 Acked Write 9 Done Sent Done false
15516 4658 Acked Write 5 Done Sent Done false
15517 4659 Acked Write 10 Done Sent Done false
15518 4660 Acked Write 7 Done Sent Done false
15519 4661 Acked Write 10 Done Sent Done false
15520 4662 Acked Flush 0 Done Sent Done false
15521 4663 Acked Read 7 Done Sent Done false
15522 4664 Acked Read 7 Done Sent Done false
15523 4665 Acked Write 5 Done Sent Done false
15524 4666 Acked Write 5 Done Sent Done false
15525 4667 Acked Read 2 Done Sent Done false
15526 4668 Acked Write 1 Done Sent Done false
15527 4669 Acked Write 9 Done Sent Done false
15528 4670 Acked Read 7 Done Sent Done false
15529 4671 Acked Read 7 Done Sent Done false
15530 4672 Acked Write 9 Done Sent Done false
15531 4673 Acked Write 5 Done Sent Done false
15532 4674 Acked Read 8 Done Sent Done false
15533 4675 Acked Read 10 Done Sent Done false
15534 4676 Acked Write 10 Done Sent Done false
15535 4677 Acked Flush 0 Done Sent Done false
15536 4678 Acked Write 2 Done Sent Done false
15537 4679 Acked Read 6 Done Sent Done false
15538 4680 Acked Write 4 Done Sent Done false
15539 4681 Acked Read 2 Done Sent Done false
15540 4682 Acked Read 7 Done Sent Done false
15541 4683 Acked Write 10 Done Sent Done false
15542 4684 Acked Read 3 Done Sent Done false
15543 4685 Acked Write 4 Done Sent Done false
15544 4686 Acked Write 5 Done Sent Done false
15545 4687 Acked Read 10 Done Sent Done false
15546 4688 Acked Read 8 Done Sent Done false
15547 4689 Acked Write 6 Done Sent Done false
15548 4690 Acked Read 4 Done Sent Done false
15549 4691 Acked Write 5 Done Sent Done false
15550 4692 Acked Read 5 Done Sent Done false
15551 4693 Acked Read 3 Done Sent Done false
15552 4694 Acked Read 7 Done Sent Done false
15553 4695 Acked Write 6 Done Sent Done false
15554 4696 Acked Write 7 Done Sent Done false
15555 4697 Acked Write 10 Done Sent Done false
15556 4698 Acked Write 4 Done Sent Done false
15557 4699 Acked Flush 0 Done Sent Done false
15558 4700 Acked Read 4 Done Sent Done false
15559 4701 Acked Write 2 Done Sent Done false
15560 4702 Acked Write 8 Done Sent Done false
15561 4703 Acked Write 9 Done Sent Done false
15562 4704 Acked Write 9 Done Sent Done false
15563 4705 Acked Read 4 Done Sent Done false
15564 4706 Acked Read 4 Done Sent Done false
15565 4707 Acked Write 2 Done Sent Done false
15566 4708 Acked Read 9 Done Sent Done false
15567 4709 Acked Read 2 Done Sent Done false
15568 4710 Acked Flush 0 Done Sent Done false
15569 4711 Acked Flush 0 Done Sent Done false
15570 4712 Acked Read 10 Done Sent Done false
15571 4713 Acked Read 7 Done Sent Done false
15572 4714 Acked Read 5 Done Sent Done false
15573 4715 Acked Read 5 Done Sent Done false
15574 4716 Acked Write 8 Done Sent Done false
15575 4717 Acked Write 9 Done Sent Done false
15576 4718 Acked Read 5 Done Sent Done false
15577 4719 Acked Flush 0 Done Sent Done false
15578 4720 Acked Read 7 Done Sent Done false
15579 4721 Acked Read 10 Done Sent Done false
15580 4722 Acked Read 2 Done Sent Done false
15581 4723 Acked Read 8 Done Sent Done false
15582 4724 Acked Write 4 Done Sent Done false
15583 4725 Acked Read 5 Done Sent Done false
15584 4726 Acked Flush 0 Done Sent Done false
15585 4727 Acked Write 5 Done Sent Done false
15586 4728 Acked Write 10 Done Sent Done false
15587 4729 Acked Read 10 Done Sent Done false
15588 4730 Acked Read 7 Done Sent Done false
15589 4731 Acked Read 4 Done Sent Done false
15590 4732 Acked Read 6 Done Sent Done false
15591 4733 Acked Write 8 Done Sent Done false
15592 4734 Acked Read 1 Done Sent Done false
15593 4735 Acked Read 10 Done Sent Done false
15594 4736 Acked Write 4 Done Sent Done false
15595 4737 Acked Write 6 Done Sent Done false
15596 4738 Acked Flush 0 Done Sent Done false
15597 4739 Acked Read 2 Done Sent Done false
15598 4740 Acked Write 3 Done Sent Done false
15599 4741 Acked Read 1 Done Sent Done false
15600 4742 Acked Write 8 Done Sent Done false
15601 4743 Acked Write 8 Done Sent Done false
15602 4744 Acked Write 4 Done Sent Done false
15603 4745 Acked Write 10 Done Sent Done false
15604 4746 Acked Read 4 Done Sent Done false
15605 4747 Acked Read 5 Done Sent Done false
15606 4748 Acked Write 6 Done Sent Done false
15607 4749 Acked Write 6 Done Sent Done false
15608 4750 Acked Write 2 Done Sent Done false
15609 4751 Acked Read 4 Done Sent Done false
15610 4752 Acked Flush 0 Done Sent Done false
15611 4753 Acked Write 10 Done Sent Done false
15612 4754 Acked Write 4 Done Sent Done false
15613 4755 Acked Write 5 Done Sent Done false
15614 4756 Acked Write 7 Done Sent Done false
15615 4757 Acked Read 7 Done Sent Done false
15616 4758 Acked Flush 0 Done Sent Done false
15617 4759 Acked Write 4 Done Sent Done false
15618 4760 Acked Read 6 Done Sent Done false
15619 4761 Acked Write 10 Done Sent Done false
15620 4762 Acked Flush 0 Done Sent Done false
15621 4763 Acked Write 1 Done Sent Done false
15622 4764 Acked Read 9 Done Sent Done false
15623 4765 Acked Flush 0 Done Sent Done false
15624 4766 Acked Write 3 Done Sent Done false
15625 4767 Acked Read 5 Done Sent Done false
15626 4768 Acked Flush 0 Done Sent Done false
15627 4769 Acked Write 5 Done Sent Done false
15628 4770 Acked Write 2 Done Sent Done false
15629 4771 Acked Write 1 Done Sent Done false
15630 4772 Acked Read 7 Done Sent Done false
15631 4773 Acked Read 7 Done Sent Done false
15632 4774 Acked Write 5 Done Sent Done false
15633 4775 Acked Write 7 Done Sent Done false
15634 4776 Acked Read 8 Done Sent Done false
15635 4777 Acked Read 7 Done Sent Done false
15636 4778 Acked Flush 0 Done Sent Done false
15637 4779 Acked Write 8 Done Sent Done false
15638 4780 Acked Flush 0 Done Sent Done false
15639 4781 Acked Read 5 Done Sent Done false
15640 4782 Acked Read 8 Done Sent Done false
15641 4783 Acked Flush 0 Done Sent Done false
15642 4784 Acked Write 9 Done Sent Done false
15643 4785 Acked Write 5 Done Sent Done false
15644 4786 Acked Read 9 Done Sent Done false
15645 4787 Acked Read 2 Done Sent Done false
15646 4788 Acked Write 5 Done Sent Done false
15647 4789 Acked Write 2 Done Sent Done false
15648 4790 Acked Read 1 Done Sent Done false
15649 4791 Acked Read 5 Done Sent Done false
15650 4792 Acked Read 8 Done Sent Done false
15651 4793 Acked Write 1 Done Sent Done false
15652 4794 Acked Read 5 Done Sent Done false
15653 4795 Acked Write 3 Done Sent Done false
15654 4796 Acked Read 6 Done Sent Done false
15655 4797 Acked Flush 0 Done Sent Done false
15656 4798 Acked Read 7 Done Sent Done false
15657 4799 Acked Flush 0 Done Sent Done false
15658 4800 Acked Write 7 Done Sent Done false
15659 4801 Acked Write 3 Done Sent Done false
15660 4802 Acked Flush 0 Done Sent Done false
15661 4803 Acked Read 3 Done Sent Done false
15662 4804 Acked Write 3 Done Sent Done false
15663 4805 Acked Read 5 Done Sent Done false
15664 4806 Acked Read 5 Done Sent Done false
15665 4807 Acked Write 7 Done Sent Done false
15666 4808 Acked Write 4 Done Sent Done false
15667 4809 Acked Write 2 Done Sent Done false
15668 4810 Acked Write 6 Done Sent Done false
15669 4811 Acked Read 6 Done Sent Done false
15670 4812 Acked Read 8 Done Sent Done false
15671 4813 Acked Read 2 Done Sent Done false
15672 4814 Acked Read 4 Done Sent Done false
15673 4815 Acked Flush 0 Done Sent Done false
15674 4816 Acked Read 2 Done Sent Done false
15675 4817 Acked Write 4 Done Sent Done false
15676 4818 Acked Write 4 Done Sent Done false
15677 4819 Acked Read 1 Done Sent Done false
15678 4820 Acked Read 9 Done Sent Done false
15679 4821 Acked Read 9 Done Sent Done false
15680 4822 Acked Flush 0 Done Sent Done false
15681 4823 Acked Write 7 Done Sent Done false
15682 4824 Acked Write 10 Done Sent Done false
15683 4825 Acked Read 3 Done Sent Done false
15684 4826 Acked Write 6 Done Sent Done false
15685 4827 Acked Read 8 Done Sent Done false
15686 4828 Acked Read 3 Done Sent Done false
15687 4829 Acked Read 5 Done Sent Done false
15688 4830 Acked Read 2 Done Sent Done false
15689 4831 Acked Read 3 Done Sent Done false
15690 4832 Acked Write 1 Done Sent Done false
15691 4833 Acked Flush 0 Done Sent Done false
15692 4834 Acked Write 2 Done Sent Done false
15693 4835 Acked Read 2 Done Sent Done false
15694 4836 Acked Read 9 Done Sent Done false
15695 4837 Acked Read 6 Done Sent Done false
15696 4838 Acked Read 3 Done Sent Done false
15697 4839 Acked Write 4 Done Sent Done false
15698 4840 Acked Read 7 Done Sent Done false
15699 4841 Acked Flush 0 Done Sent Done false
15700 4842 Acked Write 1 Done Sent Done false
15701 4843 Acked Write 3 Done Sent Done false
15702 4844 Acked Write 4 Done Sent Done false
15703 4845 Acked Write 7 Done Sent Done false
15704 4846 Acked Write 2 Done Sent Done false
15705 4847 Acked Read 2 Done Sent Done false
15706 4848 Acked Read 1 Done Sent Done false
15707 4849 Acked Write 2 Done Sent Done false
15708 4850 Acked Read 4 Done Sent Done false
15709 4851 Acked Write 4 Done Sent Done false
15710 4852 Acked Write 9 Done Sent Done false
15711 4853 Acked Read 5 Done Sent Done false
15712 4854 Acked Read 4 Done Sent Done false
15713 4855 Acked Write 3 Done Sent Done false
15714 4856 Acked Read 8 Done Sent Done false
15715 4857 Acked Read 10 Done Sent Done false
15716 4858 Acked Read 3 Done Sent Done false
15717 4859 Acked Write 7 Done Sent Done false
15718 4860 Acked Flush 0 Done Sent Done false
15719 4861 Acked Write 5 Done Sent Done false
15720 4862 Acked Read 5 Done Sent Done false
15721 4863 Acked Write 9 Done Sent Done false
15722 4864 Acked Write 2 Done Sent Done false
15723 4865 Acked Write 1 Done Sent Done false
15724 4866 Acked Read 10 Done Sent Done false
15725 4867 Acked Flush 0 Done Sent Done false
15726 4868 Acked Write 2 Done Sent Done false
15727 4869 Acked Read 8 Done Sent Done false
15728 4870 Acked Read 2 Done Sent Done false
15729 4871 Acked Write 10 Done Sent Done false
15730 4872 Acked Flush 0 Done Sent Done false
15731 4873 Acked Write 8 Done Sent Done false
15732 4874 Acked Read 5 Done Sent Done false
15733 4875 Acked Read 6 Done Sent Done false
15734 4876 Acked Read 4 Done Sent Done false
15735 4877 Acked Read 3 Done Sent Done false
15736 4878 Acked Read 10 Done Sent Done false
15737 4879 Acked Write 5 Done Sent Done false
15738 4880 Acked Write 5 Done Sent Done false
15739 4881 Acked Write 5 Done Sent Done false
15740 4882 Acked Read 7 Done Sent Done false
15741 4883 Acked Read 8 Done Sent Done false
15742 4884 Acked Read 8 Done Sent Done false
15743 4885 Acked Read 3 Done Sent Done false
15744 4886 Acked Read 6 Done Sent Done false
15745 4887 Acked Read 6 Done Sent Done false
15746 4888 Acked Read 6 Done Sent Done false
15747 4889 Acked Read 5 Done Sent Done false
15748 4890 Acked Read 5 Done Sent Done false
15749 4891 Acked Write 7 Done Sent Done false
15750 4892 Acked Read 7 Done Sent Done false
15751 4893 Acked Flush 0 Done Sent Done false
15752 4894 Acked Write 3 Done Sent Done false
15753 4895 Acked Read 7 Done Sent Done false
15754 4896 Acked Write 2 Done Sent Done false
15755 4897 Acked Read 2 Done Sent Done false
15756 4898 Acked Read 10 Done Sent Done false
15757 4899 Acked Read 3 Done Sent Done false
15758 4900 Acked Read 5 Done Sent Done false
15759 4901 Acked Write 6 Done Sent Done false
15760 4902 Acked Write 2 Done Sent Done false
15761 4903 Acked Read 2 Done Sent Done false
15762 4904 Acked Write 5 Done Sent Done false
15763 4905 Acked Write 4 Done Sent Done false
15764 4906 Acked Read 6 Done Sent Done false
15765 4907 Acked Read 8 Done Sent Done false
15766 4908 Acked Read 8 Done Sent Done false
15767 4909 Acked Read 2 Done Sent Done false
15768 4910 Acked Write 2 Done Sent Done false
15769 4911 Acked Write 10 Done Sent Done false
15770 4912 Acked Write 3 Done Sent Done false
15771 4913 Acked Write 5 Done Sent Done false
15772 4914 Acked Read 6 Done Sent Done false
15773 4915 Acked Read 3 Done Sent Done false
15774 4916 Acked Write 2 Done Sent Done false
15775 4917 Acked Read 3 Done Sent Done false
15776 4918 Acked Write 4 Done Sent Done false
15777 4919 Acked Read 7 Done Sent Done false
15778 4920 Acked Write 2 Done Sent Done false
15779 4921 Acked Flush 0 Done Sent Done false
15780 4922 Acked Read 2 Done Sent Done false
15781 4923 Acked Read 10 Done Sent Done false
15782 STATES DS:0 DS:1 DS:2 TOTAL
15783 Sent 0 301 0 301
15784 Done 301 0 301 602
15785 Skipped 0 0 0 0
15786 Error 0 0 0 0
15787 Last Flush: Some(JobId(4921)) Some(JobId(4622)) Some(JobId(4921))
15788 Downstairs last five completed: 4622 4621 4620 4619 4618
15789 Upstairs last five completed: 4923 4922 4921 4920 4919
15790 CLIENT: Up:0 ds:301 act:2
157912025-02-25T20:50:04.429ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157922025-02-25T20:50:05.387ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157932025-02-25T20:50:05.682ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157942025-02-25T20:50:05.682ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58552) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157952025-02-25T20:50:05.683ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157962025-02-25T20:50:05.684ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157972025-02-25T20:50:05.684ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157982025-02-25T20:50:05.684ZINFOcrucible: send last flush ID to this DS: Some(JobId(4622)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
157992025-02-25T20:50:05.684ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
158002025-02-25T20:50:05.684ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4622)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
158012025-02-25T20:50:05.684ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4622)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
15802 ----------------------------------------------------------------
15803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15804 Downstairs last five completed: 4924 4923 4922 4921 4920
15805 Upstairs last five completed: 4924 4923 4922 4921 4920
15806 CLIENT: Up:0 ds:0 act:3
15807 Replay: All jobs finished
15808 Main task has work to do, go find it
15809 got dsc Stop(0)
15810 stop 0
15811 Feb 25 20:50:07.930 INFO[8810] Got stop action so:false kr:false
15812 request completed, latency_us: 343, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d9d3c111-e24f-4ab7-b53e-ca23caed55f6, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
158132025-02-25T20:50:07.931ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
158142025-02-25T20:50:07.931ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
158152025-02-25T20:50:07.931ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
15816 Feb 25 20:50:07.931 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8b71cb5a-7e4b-4c37-9ba8-ca1dcc409cb9, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
158172025-02-25T20:50:07.931ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
15818 [8810] Exited with: ExitStatus(unix_wait_status(9))
15819 [8810][0] reports Exit
15820 Feb 25 20:50:11.933 INFO request completed, latency_us: 332, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5e89c078-92c2-4013-895f-d6def3a2bb7b, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
15821 001/300 Read block 2419 len 12288
15822 002/300 Read block 2903 len 28672
15823 003/300 Read block 351 len 8192
15824 004/300 Read block 454 len 4096
15825 005/300 Read block 1963 len 32768
15826 006/300 Write block 1369 len 4096 data: 4
15827 007/300 Write block 3236 len 36864 data: 3 5 5 4 4 4 7 6 7
15828 008/300 Read block 4610 len 12288
15829 009/300 Read block 4057 len 32768
15830 010/300 Read block 2238 len 8192
15831 011/300 Write block 815 len 24576 data: 5 5 4 4 4 4
15832 012/300 Read block 3725 len 12288
15833 013/300 Flush
15834 014/300 Flush
15835 015/300 Flush
15836 016/300 Read block 4085 len 12288
15837 017/300 Write block 1473 len 40960 data: 3 3 3 5 5 6 5 5 5 3
15838 018/300 Write block 512 len 4096 data: 3
15839 019/300 Write block 173 len 20480 data: 4 4 4 5 4
15840 020/300 Read block 2247 len 36864
15841 021/300 Write block 4884 len 8192 data: 4 4
15842 022/300 Read block 1776 len 40960
15843 023/300 Read block 789 len 32768
15844 024/300 Flush
15845 025/300 Flush
15846 026/300 Write block 3209 len 20480 data: 2 2 2 2 2
15847 027/300 Write block 2935 len 8192 data: 2 2
15848 028/300 Read block 4047 len 20480
15849 029/300 Read block 4444 len 20480
15850 030/300 Write block 612 len 32768 data: 2 3 3 3 3 4 4 4
15851 031/300 Read block 4401 len 28672
15852 032/300 Write block 108 len 16384 data: 3 3 3 3
15853 033/300 Write block 1318 len 24576 data: 3 4 6 7 6 7
15854 034/300 Write block 2953 len 24576 data: 3 3 3 4 5 6
15855 035/300 Read block 4944 len 24576
15856 036/300 Write block 4791 len 20480 data: 2 3 3 6 7
15857 037/300 Read block 4983 len 24576
15858 038/300 Read block 3987 len 36864
15859 039/300 Write block 1414 len 12288 data: 4 4 5
15860 040/300 Read block 231 len 24576
15861 041/300 Flush
15862 042/300 Write block 4368 len 24576 data: 5 4 4 3 3 3
15863 043/300 Read block 4121 len 40960
15864 044/300 Read block 2617 len 8192
15865 045/300 Write block 3091 len 40960 data: 2 2 2 2 2 2 2 2 3 3
15866 046/300 Write block 3612 len 8192 data: 4 4
15867 047/300 Read block 2348 len 28672
15868 048/300 Read block 1222 len 16384
15869 049/300 Read block 3321 len 20480
15870 050/300 Write block 1653 len 40960 data: 2 2 2 2 3 3 3 3 3 3
15871 051/300 Write block 2451 len 24576 data: 6 5 7 6 6 4
15872 052/300 Write block 3848 len 28672 data: 2 2 4 4 5 5 6
15873 053/300 Read block 1082 len 28672
15874 054/300 Read block 4358 len 12288
15875 055/300 Write block 1024 len 24576 data: 5 5 4 4 3 2
15876 056/300 Read block 1805 len 8192
15877 057/300 Write block 2969 len 16384 data: 3 3 3 3
15878 058/300 Read block 402 len 20480
15879 059/300 Write block 1736 len 12288 data: 3 2 2
15880 060/300 Read block 1676 len 40960
15881 061/300 Read block 3779 len 4096
15882 062/300 Read block 2632 len 36864
15883 063/300 Read block 2999 len 20480
15884 064/300 Read block 2894 len 36864
15885 065/300 Flush
15886 066/300 Write block 2518 len 16384 data: 5 5 5 4
15887 067/300 Flush
15888 068/300 Read block 4822 len 4096
15889 069/300 Write block 651 len 4096 data: 2
15890 070/300 Write block 804 len 16384 data: 2 2 2 2
15891 071/300 Write block 1284 len 4096 data: 3
15892 072/300 Write block 3103 len 16384 data: 4 4 5 4
15893 073/300 Write block 173 len 36864 data: 5 5 5 6 5 3 3 3 3
15894 074/300 Flush
15895 075/300 Write block 3555 len 20480 data: 3 4 4 3 3
15896 076/300 Read block 3098 len 16384
15897 077/300 Read block 4998 len 4096
15898 078/300 Read block 463 len 16384
15899 079/300 Flush
15900 080/300 Write block 478 len 20480 data: 3 3 3 4 4
15901 081/300 Read block 825 len 28672
15902 082/300 Write block 3879 len 40960 data: 4 4 4 6 6 6 6 6 5 5
15903 083/300 Write block 2373 len 32768 data: 3 2 3 3 5 5 5 3
15904 084/300 Read block 3438 len 4096
15905 085/300 Read block 4337 len 8192
15906 086/300 Write block 3329 len 16384 data: 3 3 3 3
15907 087/300 Write block 4993 len 12288 data: 5 4 4
15908 088/300 Read block 3884 len 4096
15909 089/300 Read block 2907 len 8192
15910 090/300 Write block 4220 len 12288 data: 5 5 4
15911 091/300 Write block 4839 len 40960 data: 4 3 2 2 2 2 2 3 4 4
15912 092/300 Read block 2989 len 28672
15913 093/300 Read block 3937 len 16384
15914 094/300 Read block 2689 len 12288
15915 095/300 Write block 4498 len 20480 data: 2 3 2 2 2
15916 096/300 Write block 2570 len 20480 data: 4 4 3 3 3
15917 097/300 Write block 3876 len 24576 data: 5 6 5 5 5 5
15918 098/300 Flush
15919 099/300 Write block 3913 len 20480 data: 3 2 3 3 3
15920 100/300 Read block 432 len 12288
15921 101/300 Read block 4829 len 4096
15922 102/300 Read block 4841 len 36864
15923 103/300 Read block 3394 len 12288
15924 104/300 Read block 4124 len 20480
15925 105/300 Read block 1540 len 16384
15926 106/300 Read block 2239 len 36864
15927 107/300 Read block 1234 len 20480
15928 108/300 Read block 4580 len 4096
15929 109/300 Read block 932 len 36864
15930 110/300 Read block 913 len 20480
15931 111/300 Read block 1939 len 40960
15932 112/300 Write block 3629 len 12288 data: 6 6 4
15933 113/300 Read block 2838 len 36864
15934 114/300 Write block 4120 len 24576 data: 4 4 4 5 4 5
15935 115/300 Write block 870 len 12288 data: 2 2 2
15936 116/300 Read block 3199 len 36864
15937 117/300 Read block 351 len 32768
15938 118/300 Write block 4589 len 32768 data: 3 3 3 3 3 3 4 4
15939 119/300 Write block 915 len 28672 data: 2 3 4 4 5 5 5
15940 120/300 Read block 4493 len 4096
15941 121/300 Read block 3975 len 36864
15942 122/300 Write block 4920 len 28672 data: 5 5 4 3 4 4 3
15943 123/300 Read block 2617 len 20480
15944 124/300 Read block 60 len 24576
15945 125/300 Read block 1980 len 36864
15946 126/300 Write block 3310 len 32768 data: 4 4 4 4 4 5 4 3
15947 127/300 Write block 3413 len 24576 data: 3 2 2 4 6 6
15948 128/300 Write block 2488 len 20480 data: 4 4 3 3 3
15949 129/300 Flush
15950 130/300 Write block 1886 len 20480 data: 3 3 3 4 4
15951 131/300 Read block 2513 len 16384
15952 132/300 Read block 1576 len 36864
15953 133/300 Read block 1305 len 12288
15954 134/300 Write block 3593 len 20480 data: 3 3 3 3 2
15955 135/300 Read block 888 len 36864
15956 136/300 Flush
15957 137/300 Read block 1248 len 28672
15958 138/300 Write block 3579 len 4096 data: 3
15959 139/300 Write block 1116 len 36864 data: 5 5 5 4 5 6 5 5 4
15960 140/300 Read block 2813 len 32768
15961 141/300 Read block 4261 len 40960
15962 142/300 Read block 3301 len 12288
15963 143/300 Write block 87 len 36864 data: 4 4 4 4 5 6 6 5 4
15964 144/300 Write block 2493 len 40960 data: 3 3 4 3 4 4 4 4 4 4
15965 145/300 Flush
15966 146/300 Write block 3168 len 24576 data: 3 4 4 4 4 3
15967 147/300 Write block 972 len 12288 data: 3 4 4
15968 148/300 Flush
15969 149/300 Read block 126 len 20480
15970 150/300 Read block 3958 len 36864
15971 151/300 Read block 4970 len 32768
15972 152/300 Read block 2150 len 40960
15973 153/300 Read block 4242 len 8192
15974 154/300 Write block 1708 len 40960 data: 3 3 2 3 3 3 3 4 4 4
15975 155/300 Read block 2274 len 4096
15976 156/300 Write block 1024 len 8192 data: 6 6
15977 157/300 Flush
15978 158/300 Read block 4493 len 24576
15979 159/300 Write block 3979 len 40960 data: 4 4 4 4 3 3 3 4 3 4
15980 160/300 Write block 725 len 40960 data: 6 6 5 5 6 4 4 2 2 2
15981 161/300 Write block 2091 len 20480 data: 2 2 2 2 2
15982 162/300 Read block 3840 len 32768
15983 163/300 Write block 667 len 40960 data: 5 5 3 3 3 4 5 7 8 5
15984 164/300 Write block 2382 len 16384 data: 3 4 3 3
15985 165/300 Write block 1051 len 12288 data: 3 5 4
15986 166/300 Write block 3755 len 28672 data: 3 3 3 3 4 4 4
15987 167/300 Read block 1030 len 20480
15988 168/300 Read block 2333 len 12288
15989 169/300 Flush
15990 170/300 Read block 2794 len 8192
15991 171/300 Write block 4363 len 12288 data: 3 3 3
15992 172/300 Write block 3085 len 8192 data: 3 3
15993 173/300 Read block 1261 len 24576
15994 174/300 Write block 3983 len 28672 data: 4 4 4 5 4 5 5
15995 175/300 Write block 4805 len 20480 data: 3 3 4 4 3
15996 176/300 Read block 3730 len 36864
15997 177/300 Read block 405 len 12288
15998 178/300 Write block 4354 len 40960 data: 2 3 3 3 3 3 3 3 3 4
15999 179/300 Read block 4633 len 12288
16000 180/300 Write block 3414 len 12288 data: 3 3 5
16001 181/300 Write block 2373 len 28672 data: 4 3 4 4 6 6 6
16002 182/300 Read block 542 len 20480
16003 183/300 Write block 2085 len 36864 data: 3 3 3 3 3 3 3 3 3
16004 184/300 Read block 2116 len 20480
16005 185/300 Write block 4426 len 24576 data: 8 6 6 6 5 5
16006 186/300 Flush
16007 187/300 Read block 1491 len 36864
16008 188/300 Read block 4489 len 40960
16009 189/300 Read block 848 len 32768
16010 190/300 Write block 2063 len 20480 data: 2 2 2 2 2
16011 191/300 Write block 858 len 32768 data: 5 4 4 5 5 6 4 4
16012 192/300 Read block 1251 len 12288
16013 193/300 Write block 3825 len 40960 data: 5 5 5 5 5 4 4 3 3 3
16014 194/300 Flush
16015 195/300 Write block 4707 len 28672 data: 2 2 3 3 4 4 4
16016 196/300 Read block 2401 len 32768
16017 197/300 Read block 1798 len 32768
16018 198/300 Read block 3169 len 36864
16019 199/300 Read block 4930 len 24576
16020 200/300 Read block 2421 len 20480
16021 201/300 Read block 3465 len 40960
16022 202/300 Read block 4879 len 16384
16023 203/300 Read block 1223 len 12288
16024 204/300 Write block 219 len 40960 data: 5 6 4 4 3 3 2 4 4 4
16025 205/300 Flush
16026 206/300 Write block 2112 len 20480 data: 5 4 4 4 4
16027 207/300 Read block 122 len 12288
16028 208/300 Write block 1965 len 16384 data: 3 3 3 3
16029 209/300 Write block 836 len 36864 data: 3 3 3 3 4 4 6 6 6
16030 210/300 Read block 4986 len 40960
16031 211/300 Write block 3477 len 24576 data: 3 3 4 3 3 3
16032 212/300 Write block 3764 len 12288 data: 5 5 5
16033 213/300 Read block 4335 len 28672
16034 214/300 Write block 2876 len 40960 data: 2 2 2 2 3 4 5 6 3 2
16035 215/300 Write block 3873 len 12288 data: 5 5 5
16036 216/300 Read block 1508 len 36864
16037 217/300 Read block 3853 len 4096
16038 218/300 Write block 149 len 24576 data: 2 2 2 2 3 3
16039 219/300 Read block 2824 len 16384
16040 220/300 Read block 591 len 4096
16041 221/300 Write block 79 len 12288 data: 4 4 5
16042 222/300 Write block 4233 len 12288 data: 5 6 6
16043 223/300 Read block 4306 len 24576
16044 224/300 Write block 2166 len 20480 data: 5 5 4 4 4
16045 225/300 Read block 2887 len 32768
16046 226/300 Read block 1391 len 24576
16047 227/300 Write block 1139 len 36864 data: 3 3 3 2 2 2 2 2 2
16048 228/300 Read block 3892 len 40960
16049 229/300 Read block 1966 len 20480
16050 230/300 Read block 1475 len 36864
16051 231/300 Read block 1483 len 28672
16052 232/300 Read block 1094 len 12288
16053 233/300 Write block 2102 len 36864 data: 4 3 3 3 4 5 5 3 2
16054 234/300 Read block 4681 len 40960
16055 235/300 Write block 2680 len 24576 data: 6 7 6 6 6 4
16056 236/300 Flush
16057 237/300 Write block 3758 len 28672 data: 4 5 5 5 5 5 6
16058 238/300 Write block 1585 len 12288 data: 8 7 6
16059 239/300 Read block 3433 len 4096
16060 240/300 Write block 1073 len 8192 data: 4 3
16061 241/300 Read block 212 len 32768
16062 242/300 Write block 2986 len 32768 data: 6 7 9 6 7 6 6 5
16063 243/300 Write block 4613 len 4096 data: 4
16064 244/300 Write block 4900 len 32768 data: 5 4 3 3 3 2 2 2
16065 245/300 Write block 728 len 12288 data: 6 7 5
16066 246/300 Write block 3195 len 28672 data: 2 2 2 4 4 4 4
16067 247/300 Read block 2995 len 36864
16068 248/300 Read block 1659 len 16384
16069 249/300 Read block 1467 len 8192
16070 250/300 Write block 4574 len 20480 data: 4 4 3 4 4
16071 251/300 Read block 3957 len 20480
16072 252/300 Flush
16073 253/300 Write block 4294 len 36864 data: 3 2 3 3 3 3 5 3 3
16074 254/300 Read block 4096 len 40960
16075 255/300 Write block 3832 len 32768 data: 4 4 4 3 4 4 3 3
16076 256/300 Read block 384 len 40960
16077 257/300 Read block 3345 len 24576
16078 258/300 Write block 4366 len 16384 data: 3 3 6 5
16079 259/300 Write block 3004 len 40960 data: 4 4 5 4 6 7 6 6 6 5
16080 260/300 Read block 2143 len 36864
16081 261/300 Write block 1524 len 40960 data: 3 3 3 3 4 5 5 6 6 5
16082 262/300 Write block 3745 len 20480 data: 3 3 4 3 3
16083 263/300 Write block 4515 len 24576 data: 4 3 2 3 3 3
16084 264/300 Read block 1340 len 36864
16085 265/300 Write block 2221 len 12288 data: 3 4 4
16086 266/300 Write block 316 len 28672 data: 2 2 2 2 3 4 5
16087 267/300 Write block 3596 len 16384 data: 4 3 2 2
16088 268/300 Read block 4819 len 8192
16089 269/300 Read block 4367 len 12288
16090 270/300 Read block 3870 len 20480
16091 271/300 Read block 532 len 40960
16092 272/300 Read block 3435 len 24576
16093 273/300 Read block 1860 len 20480
16094 274/300 Read block 270 len 16384
16095 275/300 Read block 1852 len 20480
16096 276/300 Read block 1540 len 40960
16097 277/300 Write block 4691 len 40960 data: 6 5 4 5 4 4 4 4 3 4
16098 278/300 Read block 3075 len 28672
16099 279/300 Read block 800 len 12288
16100 280/300 Write block 91 len 36864 data: 6 7 7 6 5 3 3 3 3
16101 281/300 Read block 2231 len 12288
16102 282/300 Write block 3696 len 36864 data: 2 2 2 2 2 4 4 3 4
16103 283/300 Read block 3020 len 20480
16104 284/300 Read block 1150 len 28672
16105 285/300 Write block 2090 len 4096 data: 4
16106 286/300 Read block 4996 len 4096
16107 287/300 Flush
16108 288/300 Read block 4742 len 36864
16109 289/300 Write block 4162 len 36864 data: 10 10 10 10 10 8 6 6 6
16110 290/300 Write block 1336 len 20480 data: 3 3 3 3 3
16111 291/300 Write block 4752 len 20480 data: 3 3 3 4 4
16112 292/300 Read block 1835 len 16384
16113 293/300 Write block 2700 len 36864 data: 3 3 3 3 3 4 5 5 6
16114 294/300 Read block 1643 len 36864
16115 295/300 Read block 2756 len 24576
16116 296/300 Read block 4493 len 12288
16117 297/300 Write block 2642 len 24576 data: 4 4 4 4 5 8
16118 298/300 Read block 4062 len 20480
16119 299/300 Read block 4374 len 4096
16120 300/300 Read block 1952 len 8192
16121 301/300 Write block 440 len 4096 data: 5
16122 Main task has work to do, go find it
16123 got dsc Start(0)
16124 start 0
16125 Feb 25 20:50:12.189 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7a44752c-ace7-4aca-9cfb-8cc7ce5fbe6d, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
16126 Replay: started 0, returned:Ok(())
16127 ----------------------------------------------------------------
16128 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16129 States: Offline Active Active
16130 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16131 4925 Acked Read 3 Sent Done Done false
16132 4926 Acked Read 7 Sent Done Done false
16133 4927 Acked Read 2 Sent Done Done false
16134 4928 Acked Read 1 Sent Done Done false
16135 4929 Acked Read 8 Sent Done Done false
16136 4930 Acked Write 1 Sent Done Done false
16137 4931 Acked Write 9 Sent Done Done false
16138 4932 Acked Read 3 Sent Done Done false
16139 4933 Acked Read 8 Sent Done Done false
16140 4934 Acked Read 2 Sent Done Done false
16141 4935 Acked Write 6 Sent Done Done false
16142 4936 Acked Read 3 Sent Done Done false
16143 4937 Acked Flush 0 Sent Done Done false
16144 4938 Acked Flush 0 Sent Done Done false
16145 4939 Acked Flush 0 Sent Done Done false
16146 4940 Acked Read 3 Sent Done Done false
16147 4941 Acked Write 10 Sent Done Done false
16148 4942 Acked Write 1 Sent Done Done false
16149 4943 Acked Write 5 Sent Done Done false
16150 4944 Acked Read 9 Sent Done Done false
16151 4945 Acked Write 2 Sent Done Done false
16152 4946 Acked Read 10 Sent Done Done false
16153 4947 Acked Read 8 Sent Done Done false
16154 4948 Acked Flush 0 Sent Done Done false
16155 4949 Acked Flush 0 Sent Done Done false
16156 4950 Acked Write 5 Sent Done Done false
16157 4951 Acked Write 2 Sent Done Done false
16158 4952 Acked Read 5 Sent Done Done false
16159 4953 Acked Read 5 Sent Done Done false
16160 4954 Acked Write 8 Sent Done Done false
16161 4955 Acked Read 7 Sent Done Done false
16162 4956 Acked Write 4 Sent Done Done false
16163 4957 Acked Write 6 Sent Done Done false
16164 4958 Acked Write 6 Sent Done Done false
16165 4959 Acked Read 6 Sent Done Done false
16166 4960 Acked Write 5 Sent Done Done false
16167 4961 Acked Read 6 Sent Done Done false
16168 4962 Acked Read 9 Sent Done Done false
16169 4963 Acked Write 3 Sent Done Done false
16170 4964 Acked Read 6 Sent Done Done false
16171 4965 Acked Flush 0 Sent Done Done false
16172 4966 Acked Write 6 Sent Done Done false
16173 4967 Acked Read 10 Sent Done Done false
16174 4968 Acked Read 2 Sent Done Done false
16175 4969 Acked Write 10 Sent Done Done false
16176 4970 Acked Write 2 Sent Done Done false
16177 4971 Acked Read 7 Sent Done Done false
16178 4972 Acked Read 4 Sent Done Done false
16179 4973 Acked Read 5 Sent Done Done false
16180 4974 Acked Write 10 Sent Done Done false
16181 4975 Acked Write 6 Sent Done Done false
16182 4976 Acked Write 7 Sent Done Done false
16183 4977 Acked Read 7 Sent Done Done false
16184 4978 Acked Read 3 Sent Done Done false
16185 4979 Acked Write 6 Sent Done Done false
16186 4980 Acked Read 2 Sent Done Done false
16187 4981 Acked Write 4 Sent Done Done false
16188 4982 Acked Read 5 Sent Done Done false
16189 4983 Acked Write 3 Sent Done Done false
16190 4984 Acked Read 10 Sent Done Done false
16191 4985 Acked Read 1 Sent Done Done false
16192 4986 Acked Read 9 Sent Done Done false
16193 4987 Acked Read 5 Sent Done Done false
16194 4988 Acked Read 9 Sent Done Done false
16195 4989 Acked Flush 0 Sent Done Done false
16196 4990 Acked Write 4 Sent Done Done false
16197 4991 Acked Flush 0 Sent Done Done false
16198 4992 Acked Read 1 Sent Done Done false
16199 4993 Acked Write 1 Sent Done Done false
16200 4994 Acked Write 4 Sent Done Done false
16201 4995 Acked Write 1 Sent Done Done false
16202 4996 Acked Write 4 Sent Done Done false
16203 4997 Acked Write 9 Sent Done Done false
16204 4998 Acked Flush 0 Sent Done Done false
16205 4999 Acked Write 5 Sent Done Done false
16206 5000 Acked Read 4 Sent Done Done false
16207 5001 Acked Read 1 Sent Done Done false
16208 5002 Acked Read 4 Sent Done Done false
16209 5003 Acked Flush 0 Sent Done Done false
16210 5004 Acked Write 5 Sent Done Done false
16211 5005 Acked Read 7 Sent Done Done false
16212 5006 Acked Write 10 Sent Done Done false
16213 5007 Acked Write 8 Sent Done Done false
16214 5008 Acked Read 1 Sent Done Done false
16215 5009 Acked Read 2 Sent Done Done false
16216 5010 Acked Write 4 Sent Done Done false
16217 5011 Acked Write 3 Sent Done Done false
16218 5012 Acked Read 1 Sent Done Done false
16219 5013 Acked Read 2 Sent Done Done false
16220 5014 Acked Write 3 Sent Done Done false
16221 5015 Acked Write 10 Sent Done Done false
16222 5016 Acked Read 7 Sent Done Done false
16223 5017 Acked Read 4 Sent Done Done false
16224 5018 Acked Read 3 Sent Done Done false
16225 5019 Acked Write 5 Sent Done Done false
16226 5020 Acked Write 5 Sent Done Done false
16227 5021 Acked Write 6 Sent Done Done false
16228 5022 Acked Flush 0 Sent Done Done false
16229 5023 Acked Write 5 Sent Done Done false
16230 5024 Acked Read 3 Sent Done Done false
16231 5025 Acked Read 1 Sent Done Done false
16232 5026 Acked Read 9 Sent Done Done false
16233 5027 Acked Read 3 Sent Done Done false
16234 5028 Acked Read 5 Sent Done Done false
16235 5029 Acked Read 4 Sent Done Done false
16236 5030 Acked Read 9 Sent Done Done false
16237 5031 Acked Read 5 Sent Done Done false
16238 5032 Acked Read 1 Sent Done Done false
16239 5033 Acked Read 9 Sent Done Done false
16240 5034 Acked Read 5 Sent Done Done false
16241 5035 Acked Read 10 Sent Done Done false
16242 5036 Acked Write 3 Sent Done Done false
16243 5037 Acked Read 9 Sent Done Done false
16244 5038 Acked Write 6 Sent Done Done false
16245 5039 Acked Write 3 Sent Done Done false
16246 5040 Acked Read 9 Sent Done Done false
16247 5041 Acked Read 8 Sent Done Done false
16248 5042 Acked Write 8 Sent Done Done false
16249 5043 Acked Write 7 Sent Done Done false
16250 5044 Acked Read 1 Sent Done Done false
16251 5045 Acked Read 9 Sent Done Done false
16252 5046 Acked Write 7 Sent Done Done false
16253 5047 Acked Read 5 Sent Done Done false
16254 5048 Acked Read 6 Sent Done Done false
16255 5049 Acked Read 9 Sent Done Done false
16256 5050 Acked Write 8 Sent Done Done false
16257 5051 Acked Write 6 Sent Done Done false
16258 5052 Acked Write 5 Sent Done Done false
16259 5053 Acked Flush 0 Sent Done Done false
16260 5054 Acked Write 5 Sent Done Done false
16261 5055 Acked Read 4 Sent Done Done false
16262 5056 Acked Read 9 Sent Done Done false
16263 5057 Acked Read 3 Sent Done Done false
16264 5058 Acked Write 5 Sent Done Done false
16265 5059 Acked Read 9 Sent Done Done false
16266 5060 Acked Flush 0 Sent Done Done false
16267 5061 Acked Read 7 Sent Done Done false
16268 5062 Acked Write 1 Sent Done Done false
16269 5063 Acked Write 9 Sent Done Done false
16270 5064 Acked Read 8 Sent Done Done false
16271 5065 Acked Read 10 Sent Done Done false
16272 5066 Acked Read 3 Sent Done Done false
16273 5067 Acked Write 9 Sent Done Done false
16274 5068 Acked Write 10 Sent Done Done false
16275 5069 Acked Flush 0 Sent Done Done false
16276 5070 Acked Write 6 Sent Done Done false
16277 5071 Acked Write 3 Sent Done Done false
16278 5072 Acked Flush 0 Sent Done Done false
16279 5073 Acked Read 5 Sent Done Done false
16280 5074 Acked Read 9 Sent Done Done false
16281 5075 Acked Read 8 Sent Done Done false
16282 5076 Acked Read 10 Sent Done Done false
16283 5077 Acked Read 2 Sent Done Done false
16284 5078 Acked Write 10 Sent Done Done false
16285 5079 Acked Read 1 Sent Done Done false
16286 5080 Acked Write 2 Sent Done Done false
16287 5081 Acked Flush 0 Sent Done Done false
16288 5082 Acked Read 6 Sent Done Done false
16289 5083 Acked Write 10 Sent Done Done false
16290 5084 Acked Write 10 Sent Done Done false
16291 5085 Acked Write 5 Sent Done Done false
16292 5086 Acked Read 8 Sent Done Done false
16293 5087 Acked Write 10 Sent Done Done false
16294 5088 Acked Write 4 Sent Done Done false
16295 5089 Acked Write 3 Sent Done Done false
16296 5090 Acked Write 7 Sent Done Done false
16297 5091 Acked Read 5 Sent Done Done false
16298 5092 Acked Read 3 Sent Done Done false
16299 5093 Acked Flush 0 Sent Done Done false
16300 5094 Acked Read 2 Sent Done Done false
16301 5095 Acked Write 3 Sent Done Done false
16302 5096 Acked Write 2 Sent Done Done false
16303 5097 Acked Read 6 Sent Done Done false
16304 5098 Acked Write 7 Sent Done Done false
16305 5099 Acked Write 5 Sent Done Done false
16306 5100 Acked Read 9 Sent Done Done false
16307 5101 Acked Read 3 Sent Done Done false
16308 5102 Acked Write 10 Sent Done Done false
16309 5103 Acked Read 3 Sent Done Done false
16310 5104 Acked Write 3 Sent Done Done false
16311 5105 Acked Write 7 Sent Done Done false
16312 5106 Acked Read 5 Sent Done Done false
16313 5107 Acked Write 9 Sent Done Done false
16314 5108 Acked Read 5 Sent Done Done false
16315 5109 Acked Write 6 Sent Done Done false
16316 5110 Acked Flush 0 Sent Done Done false
16317 5111 Acked Read 9 Sent Done Done false
16318 5112 Acked Read 10 Sent Done Done false
16319 5113 Acked Read 8 Sent Done Done false
16320 5114 Acked Write 5 Sent Done Done false
16321 5115 Acked Write 8 Sent Done Done false
16322 5116 Acked Read 3 Sent Done Done false
16323 5117 Acked Write 10 Sent Done Done false
16324 5118 Acked Flush 0 Sent Done Done false
16325 5119 Acked Write 7 Sent Done Done false
16326 5120 Acked Read 8 Sent Done Done false
16327 5121 Acked Read 8 Sent Done Done false
16328 5122 Acked Read 9 Sent Done Done false
16329 5123 Acked Read 6 Sent Done Done false
16330 5124 Acked Read 5 Sent Done Done false
16331 5125 Acked Read 10 Sent Done Done false
16332 5126 Acked Read 4 Sent Done Done false
16333 5127 Acked Read 3 Sent Done Done false
16334 5128 Acked Write 10 Sent Done Done false
16335 5129 Acked Flush 0 Sent Done Done false
16336 5130 Acked Write 5 Sent Done Done false
16337 5131 Acked Read 3 Sent Done Done false
16338 5132 Acked Write 4 Sent Done Done false
16339 5133 Acked Write 9 Sent Done Done false
16340 5134 Acked Read 10 Sent Done Done false
16341 5135 Acked Write 6 Sent Done Done false
16342 5136 Acked Write 3 Sent Done Done false
16343 5137 Acked Read 7 Sent Done Done false
16344 5138 Acked Write 10 Sent Done Done false
16345 5139 Acked Write 3 Sent Done Done false
16346 5140 Acked Read 9 Sent Done Done false
16347 5141 Acked Read 1 Sent Done Done false
16348 5142 Acked Write 6 Sent Done Done false
16349 5143 Acked Read 4 Sent Done Done false
16350 5144 Acked Read 1 Sent Done Done false
16351 5145 Acked Write 3 Sent Done Done false
16352 5146 Acked Write 3 Sent Done Done false
16353 5147 Acked Read 6 Sent Done Done false
16354 5148 Acked Write 5 Sent Done Done false
16355 5149 Acked Read 8 Sent Done Done false
16356 5150 Acked Read 6 Sent Done Done false
16357 5151 Acked Write 9 Sent Done Done false
16358 5152 Acked Read 10 Sent Done Done false
16359 5153 Acked Read 5 Sent Done Done false
16360 5154 Acked Read 9 Sent Done Done false
16361 5155 Acked Read 7 Sent Done Done false
16362 5156 Acked Read 3 Sent Done Done false
16363 5157 Acked Write 9 Sent Done Done false
16364 5158 Acked Read 10 Sent Done Done false
16365 5159 Acked Write 6 Sent Done Done false
16366 5160 Acked Flush 0 Sent Done Done false
16367 5161 Acked Write 7 Sent Done Done false
16368 5162 Acked Write 3 Sent Done Done false
16369 5163 Acked Read 1 Sent Done Done false
16370 5164 Acked Write 2 Sent Done Done false
16371 5165 Acked Read 8 Sent Done Done false
16372 5166 Acked Write 8 Sent Done Done false
16373 5167 Acked Write 1 Sent Done Done false
16374 5168 Acked Write 8 Sent Done Done false
16375 5169 Acked Write 3 Sent Done Done false
16376 5170 Acked Write 7 Sent Done Done false
16377 5171 Acked Read 9 Sent Done Done false
16378 5172 Acked Read 4 Sent Done Done false
16379 5173 Acked Read 2 Sent Done Done false
16380 5174 Acked Write 5 Sent Done Done false
16381 5175 Acked Read 5 Sent Done Done false
16382 5176 Acked Flush 0 Sent Done Done false
16383 5177 Acked Write 9 Sent Done Done false
16384 5178 Acked Read 10 Sent Done Done false
16385 5179 Acked Write 8 Sent Done Done false
16386 5180 Acked Read 10 Sent Done Done false
16387 5181 Acked Read 6 Sent Done Done false
16388 5182 Acked Write 4 Sent Done Done false
16389 5183 Acked Write 10 Sent Done Done false
16390 5184 Acked Read 9 Sent Done Done false
16391 5185 Acked Write 10 Sent Done Done false
16392 5186 Acked Write 5 Sent Done Done false
16393 5187 Acked Write 6 Sent Done Done false
16394 5188 Acked Read 9 Sent Done Done false
16395 5189 Acked Write 3 Sent Done Done false
16396 5190 Acked Write 7 Sent Done Done false
16397 5191 Acked Write 4 Sent Done Done false
16398 5192 Acked Read 2 Sent Done Done false
16399 5193 Acked Read 3 Sent Done Done false
16400 5194 Acked Read 5 Sent Done Done false
16401 5195 Acked Read 10 Sent Done Done false
16402 5196 Acked Read 6 Sent Done Done false
16403 5197 Acked Read 5 Sent Done Done false
16404 5198 Acked Read 4 Sent Done Done false
16405 5199 Acked Read 5 Sent Done Done false
16406 5200 Acked Read 10 Sent Done Done false
16407 5201 Acked Write 10 Sent Done Done false
16408 5202 Acked Read 7 Sent Done Done false
16409 5203 Acked Read 3 Sent Done Done false
16410 5204 Acked Write 9 Sent Done Done false
16411 5205 Acked Read 3 Sent Done Done false
16412 5206 Acked Write 9 Sent Done Done false
16413 5207 Acked Read 5 Sent Done Done false
16414 5208 Acked Read 7 Sent Done Done false
16415 5209 Acked Write 1 Sent Done Done false
16416 5210 Acked Read 1 Sent Done Done false
16417 5211 Acked Flush 0 Sent Done Done false
16418 5212 Acked Read 9 Sent Done Done false
16419 5213 Acked Write 9 Sent Done Done false
16420 5214 Acked Write 5 Sent Done Done false
16421 5215 Acked Write 5 Sent Done Done false
16422 5216 Acked Read 4 Sent Done Done false
16423 5217 Acked Write 9 Sent Done Done false
16424 5218 Acked Read 9 Sent Done Done false
16425 5219 Acked Read 6 Sent Done Done false
16426 5220 Acked Read 3 Sent Done Done false
16427 5221 Acked Write 6 Sent Done Done false
16428 5222 Acked Read 5 Sent Done Done false
16429 5223 Acked Read 1 Sent Done Done false
16430 5224 Acked Read 2 Sent Done Done false
16431 5225 Acked Write 1 Sent Done Done false
16432 STATES DS:0 DS:1 DS:2 TOTAL
16433 Sent 301 0 0 301
16434 Done 0 301 301 602
16435 Skipped 0 0 0 0
16436 Error 0 0 0 0
16437 Last Flush: Some(JobId(4924)) Some(JobId(5211)) Some(JobId(5211))
16438 Downstairs last five completed: 4924 4923 4922 4921 4920
16439 Upstairs last five completed: 5225 5224 5223 5222 5221
16440 CLIENT: Up:0 ds:301 act:2
164412025-02-25T20:50:12.692ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
164422025-02-25T20:50:13.398ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
164432025-02-25T20:50:14.399ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
164442025-02-25T20:50:15.401ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
16445 [8810] got start action
16446 [8810] I am going to restart
16447 Starting downstairs at port 8810
16448 Make output file at "/tmp/dsc/downstairs-8810.txt"
16449 [8810][0] reports Starting
16450 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
16451 [8810][0] reports Running
16452 ----------------------------------------------------------------
16453 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16454 States: Offline Active Active
16455 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16456 4925 Acked Read 3 Sent Done Done false
16457 4926 Acked Read 7 Sent Done Done false
16458 4927 Acked Read 2 Sent Done Done false
16459 4928 Acked Read 1 Sent Done Done false
16460 4929 Acked Read 8 Sent Done Done false
16461 4930 Acked Write 1 Sent Done Done false
16462 4931 Acked Write 9 Sent Done Done false
16463 4932 Acked Read 3 Sent Done Done false
16464 4933 Acked Read 8 Sent Done Done false
16465 4934 Acked Read 2 Sent Done Done false
16466 4935 Acked Write 6 Sent Done Done false
16467 4936 Acked Read 3 Sent Done Done false
16468 4937 Acked Flush 0 Sent Done Done false
16469 4938 Acked Flush 0 Sent Done Done false
16470 4939 Acked Flush 0 Sent Done Done false
16471 4940 Acked Read 3 Sent Done Done false
16472 4941 Acked Write 10 Sent Done Done false
16473 4942 Acked Write 1 Sent Done Done false
16474 4943 Acked Write 5 Sent Done Done false
16475 4944 Acked Read 9 Sent Done Done false
16476 4945 Acked Write 2 Sent Done Done false
16477 4946 Acked Read 10 Sent Done Done false
16478 4947 Acked Read 8 Sent Done Done false
16479 4948 Acked Flush 0 Sent Done Done false
16480 4949 Acked Flush 0 Sent Done Done false
16481 4950 Acked Write 5 Sent Done Done false
16482 4951 Acked Write 2 Sent Done Done false
16483 4952 Acked Read 5 Sent Done Done false
16484 4953 Acked Read 5 Sent Done Done false
16485 4954 Acked Write 8 Sent Done Done false
16486 4955 Acked Read 7 Sent Done Done false
16487 4956 Acked Write 4 Sent Done Done false
16488 4957 Acked Write 6 Sent Done Done false
16489 4958 Acked Write 6 Sent Done Done false
16490 4959 Acked Read 6 Sent Done Done false
16491 4960 Acked Write 5 Sent Done Done false
16492 4961 Acked Read 6 Sent Done Done false
16493 4962 Acked Read 9 Sent Done Done false
16494 4963 Acked Write 3 Sent Done Done false
16495 4964 Acked Read 6 Sent Done Done false
16496 4965 Acked Flush 0 Sent Done Done false
16497 4966 Acked Write 6 Sent Done Done false
16498 4967 Acked Read 10 Sent Done Done false
16499 4968 Acked Read 2 Sent Done Done false
16500 4969 Acked Write 10 Sent Done Done false
16501 4970 Acked Write 2 Sent Done Done false
16502 4971 Acked Read 7 Sent Done Done false
16503 4972 Acked Read 4 Sent Done Done false
16504 4973 Acked Read 5 Sent Done Done false
16505 4974 Acked Write 10 Sent Done Done false
16506 4975 Acked Write 6 Sent Done Done false
16507 4976 Acked Write 7 Sent Done Done false
16508 4977 Acked Read 7 Sent Done Done false
16509 4978 Acked Read 3 Sent Done Done false
16510 4979 Acked Write 6 Sent Done Done false
16511 4980 Acked Read 2 Sent Done Done false
16512 4981 Acked Write 4 Sent Done Done false
16513 4982 Acked Read 5 Sent Done Done false
16514 4983 Acked Write 3 Sent Done Done false
16515 4984 Acked Read 10 Sent Done Done false
16516 4985 Acked Read 1 Sent Done Done false
16517 4986 Acked Read 9 Sent Done Done false
16518 4987 Acked Read 5 Sent Done Done false
16519 4988 Acked Read 9 Sent Done Done false
16520 4989 Acked Flush 0 Sent Done Done false
16521 4990 Acked Write 4 Sent Done Done false
16522 4991 Acked Flush 0 Sent Done Done false
16523 4992 Acked Read 1 Sent Done Done false
16524 4993 Acked Write 1 Sent Done Done false
16525 4994 Acked Write 4 Sent Done Done false
16526 4995 Acked Write 1 Sent Done Done false
16527 4996 Acked Write 4 Sent Done Done false
16528 4997 Acked Write 9 Sent Done Done false
16529 4998 Acked Flush 0 Sent Done Done false
16530 4999 Acked Write 5 Sent Done Done false
16531 5000 Acked Read 4 Sent Done Done false
16532 5001 Acked Read 1 Sent Done Done false
16533 5002 Acked Read 4 Sent Done Done false
16534 5003 Acked Flush 0 Sent Done Done false
16535 5004 Acked Write 5 Sent Done Done false
16536 5005 Acked Read 7 Sent Done Done false
16537 5006 Acked Write 10 Sent Done Done false
16538 5007 Acked Write 8 Sent Done Done false
16539 5008 Acked Read 1 Sent Done Done false
16540 5009 Acked Read 2 Sent Done Done false
16541 5010 Acked Write 4 Sent Done Done false
16542 5011 Acked Write 3 Sent Done Done false
16543 5012 Acked Read 1 Sent Done Done false
16544 5013 Acked Read 2 Sent Done Done false
16545 5014 Acked Write 3 Sent Done Done false
16546 5015 Acked Write 10 Sent Done Done false
16547 5016 Acked Read 7 Sent Done Done false
16548 5017 Acked Read 4 Sent Done Done false
16549 5018 Acked Read 3 Sent Done Done false
16550 5019 Acked Write 5 Sent Done Done false
16551 5020 Acked Write 5 Sent Done Done false
16552 5021 Acked Write 6 Sent Done Done false
16553 5022 Acked Flush 0 Sent Done Done false
16554 5023 Acked Write 5 Sent Done Done false
16555 5024 Acked Read 3 Sent Done Done false
16556 5025 Acked Read 1 Sent Done Done false
16557 5026 Acked Read 9 Sent Done Done false
16558 5027 Acked Read 3 Sent Done Done false
16559 5028 Acked Read 5 Sent Done Done false
16560 5029 Acked Read 4 Sent Done Done false
16561 5030 Acked Read 9 Sent Done Done false
16562 5031 Acked Read 5 Sent Done Done false
16563 5032 Acked Read 1 Sent Done Done false
16564 5033 Acked Read 9 Sent Done Done false
16565 5034 Acked Read 5 Sent Done Done false
16566 5035 Acked Read 10 Sent Done Done false
16567 5036 Acked Write 3 Sent Done Done false
16568 5037 Acked Read 9 Sent Done Done false
16569 5038 Acked Write 6 Sent Done Done false
16570 5039 Acked Write 3 Sent Done Done false
16571 5040 Acked Read 9 Sent Done Done false
16572 5041 Acked Read 8 Sent Done Done false
16573 5042 Acked Write 8 Sent Done Done false
16574 5043 Acked Write 7 Sent Done Done false
16575 5044 Acked Read 1 Sent Done Done false
16576 5045 Acked Read 9 Sent Done Done false
16577 5046 Acked Write 7 Sent Done Done false
16578 5047 Acked Read 5 Sent Done Done false
16579 5048 Acked Read 6 Sent Done Done false
16580 5049 Acked Read 9 Sent Done Done false
16581 5050 Acked Write 8 Sent Done Done false
16582 5051 Acked Write 6 Sent Done Done false
16583 5052 Acked Write 5 Sent Done Done false
16584 5053 Acked Flush 0 Sent Done Done false
16585 5054 Acked Write 5 Sent Done Done false
16586 5055 Acked Read 4 Sent Done Done false
16587 5056 Acked Read 9 Sent Done Done false
16588 5057 Acked Read 3 Sent Done Done false
16589 5058 Acked Write 5 Sent Done Done false
16590 5059 Acked Read 9 Sent Done Done false
16591 5060 Acked Flush 0 Sent Done Done false
16592 5061 Acked Read 7 Sent Done Done false
16593 5062 Acked Write 1 Sent Done Done false
16594 5063 Acked Write 9 Sent Done Done false
16595 5064 Acked Read 8 Sent Done Done false
16596 5065 Acked Read 10 Sent Done Done false
16597 5066 Acked Read 3 Sent Done Done false
16598 5067 Acked Write 9 Sent Done Done false
16599 5068 Acked Write 10 Sent Done Done false
16600 5069 Acked Flush 0 Sent Done Done false
16601 5070 Acked Write 6 Sent Done Done false
16602 5071 Acked Write 3 Sent Done Done false
16603 5072 Acked Flush 0 Sent Done Done false
16604 5073 Acked Read 5 Sent Done Done false
16605 5074 Acked Read 9 Sent Done Done false
16606 5075 Acked Read 8 Sent Done Done false
16607 5076 Acked Read 10 Sent Done Done false
16608 5077 Acked Read 2 Sent Done Done false
16609 5078 Acked Write 10 Sent Done Done false
16610 5079 Acked Read 1 Sent Done Done false
16611 5080 Acked Write 2 Sent Done Done false
16612 5081 Acked Flush 0 Sent Done Done false
16613 5082 Acked Read 6 Sent Done Done false
16614 5083 Acked Write 10 Sent Done Done false
16615 5084 Acked Write 10 Sent Done Done false
16616 5085 Acked Write 5 Sent Done Done false
16617 5086 Acked Read 8 Sent Done Done false
16618 5087 Acked Write 10 Sent Done Done false
16619 5088 Acked Write 4 Sent Done Done false
16620 5089 Acked Write 3 Sent Done Done false
16621 5090 Acked Write 7 Sent Done Done false
16622 5091 Acked Read 5 Sent Done Done false
16623 5092 Acked Read 3 Sent Done Done false
16624 5093 Acked Flush 0 Sent Done Done false
16625 5094 Acked Read 2 Sent Done Done false
16626 5095 Acked Write 3 Sent Done Done false
16627 5096 Acked Write 2 Sent Done Done false
16628 5097 Acked Read 6 Sent Done Done false
16629 5098 Acked Write 7 Sent Done Done false
16630 5099 Acked Write 5 Sent Done Done false
16631 5100 Acked Read 9 Sent Done Done false
16632 5101 Acked Read 3 Sent Done Done false
16633 5102 Acked Write 10 Sent Done Done false
16634 5103 Acked Read 3 Sent Done Done false
16635 5104 Acked Write 3 Sent Done Done false
16636 5105 Acked Write 7 Sent Done Done false
16637 5106 Acked Read 5 Sent Done Done false
16638 5107 Acked Write 9 Sent Done Done false
16639 5108 Acked Read 5 Sent Done Done false
16640 5109 Acked Write 6 Sent Done Done false
16641 5110 Acked Flush 0 Sent Done Done false
16642 5111 Acked Read 9 Sent Done Done false
16643 5112 Acked Read 10 Sent Done Done false
16644 5113 Acked Read 8 Sent Done Done false
16645 5114 Acked Write 5 Sent Done Done false
16646 5115 Acked Write 8 Sent Done Done false
16647 5116 Acked Read 3 Sent Done Done false
16648 5117 Acked Write 10 Sent Done Done false
16649 5118 Acked Flush 0 Sent Done Done false
16650 5119 Acked Write 7 Sent Done Done false
16651 5120 Acked Read 8 Sent Done Done false
16652 5121 Acked Read 8 Sent Done Done false
16653 5122 Acked Read 9 Sent Done Done false
16654 5123 Acked Read 6 Sent Done Done false
16655 5124 Acked Read 5 Sent Done Done false
16656 5125 Acked Read 10 Sent Done Done false
16657 5126 Acked Read 4 Sent Done Done false
16658 5127 Acked Read 3 Sent Done Done false
16659 5128 Acked Write 10 Sent Done Done false
16660 5129 Acked Flush 0 Sent Done Done false
16661 5130 Acked Write 5 Sent Done Done false
16662 5131 Acked Read 3 Sent Done Done false
16663 5132 Acked Write 4 Sent Done Done false
16664 5133 Acked Write 9 Sent Done Done false
16665 5134 Acked Read 10 Sent Done Done false
16666 5135 Acked Write 6 Sent Done Done false
16667 5136 Acked Write 3 Sent Done Done false
16668 5137 Acked Read 7 Sent Done Done false
16669 5138 Acked Write 10 Sent Done Done false
16670 5139 Acked Write 3 Sent Done Done false
16671 5140 Acked Read 9 Sent Done Done false
16672 5141 Acked Read 1 Sent Done Done false
16673 5142 Acked Write 6 Sent Done Done false
16674 5143 Acked Read 4 Sent Done Done false
16675 5144 Acked Read 1 Sent Done Done false
16676 5145 Acked Write 3 Sent Done Done false
16677 5146 Acked Write 3 Sent Done Done false
16678 5147 Acked Read 6 Sent Done Done false
16679 5148 Acked Write 5 Sent Done Done false
16680 5149 Acked Read 8 Sent Done Done false
16681 5150 Acked Read 6 Sent Done Done false
16682 5151 Acked Write 9 Sent Done Done false
16683 5152 Acked Read 10 Sent Done Done false
16684 5153 Acked Read 5 Sent Done Done false
16685 5154 Acked Read 9 Sent Done Done false
16686 5155 Acked Read 7 Sent Done Done false
16687 5156 Acked Read 3 Sent Done Done false
16688 5157 Acked Write 9 Sent Done Done false
16689 5158 Acked Read 10 Sent Done Done false
16690 5159 Acked Write 6 Sent Done Done false
16691 5160 Acked Flush 0 Sent Done Done false
16692 5161 Acked Write 7 Sent Done Done false
16693 5162 Acked Write 3 Sent Done Done false
16694 5163 Acked Read 1 Sent Done Done false
16695 5164 Acked Write 2 Sent Done Done false
16696 5165 Acked Read 8 Sent Done Done false
16697 5166 Acked Write 8 Sent Done Done false
16698 5167 Acked Write 1 Sent Done Done false
16699 5168 Acked Write 8 Sent Done Done false
16700 5169 Acked Write 3 Sent Done Done false
16701 5170 Acked Write 7 Sent Done Done false
16702 5171 Acked Read 9 Sent Done Done false
16703 5172 Acked Read 4 Sent Done Done false
16704 5173 Acked Read 2 Sent Done Done false
16705 5174 Acked Write 5 Sent Done Done false
16706 5175 Acked Read 5 Sent Done Done false
16707 5176 Acked Flush 0 Sent Done Done false
16708 5177 Acked Write 9 Sent Done Done false
16709 5178 Acked Read 10 Sent Done Done false
16710 5179 Acked Write 8 Sent Done Done false
16711 5180 Acked Read 10 Sent Done Done false
16712 5181 Acked Read 6 Sent Done Done false
16713 5182 Acked Write 4 Sent Done Done false
16714 5183 Acked Write 10 Sent Done Done false
16715 5184 Acked Read 9 Sent Done Done false
16716 5185 Acked Write 10 Sent Done Done false
16717 5186 Acked Write 5 Sent Done Done false
16718 5187 Acked Write 6 Sent Done Done false
16719 5188 Acked Read 9 Sent Done Done false
16720 5189 Acked Write 3 Sent Done Done false
16721 5190 Acked Write 7 Sent Done Done false
16722 5191 Acked Write 4 Sent Done Done false
16723 5192 Acked Read 2 Sent Done Done false
16724 5193 Acked Read 3 Sent Done Done false
16725 5194 Acked Read 5 Sent Done Done false
16726 5195 Acked Read 10 Sent Done Done false
16727 5196 Acked Read 6 Sent Done Done false
16728 5197 Acked Read 5 Sent Done Done false
16729 5198 Acked Read 4 Sent Done Done false
16730 5199 Acked Read 5 Sent Done Done false
16731 5200 Acked Read 10 Sent Done Done false
16732 5201 Acked Write 10 Sent Done Done false
16733 5202 Acked Read 7 Sent Done Done false
16734 5203 Acked Read 3 Sent Done Done false
16735 5204 Acked Write 9 Sent Done Done false
16736 5205 Acked Read 3 Sent Done Done false
16737 5206 Acked Write 9 Sent Done Done false
16738 5207 Acked Read 5 Sent Done Done false
16739 5208 Acked Read 7 Sent Done Done false
16740 5209 Acked Write 1 Sent Done Done false
16741 5210 Acked Read 1 Sent Done Done false
16742 5211 Acked Flush 0 Sent Done Done false
16743 5212 Acked Read 9 Sent Done Done false
16744 5213 Acked Write 9 Sent Done Done false
16745 5214 Acked Write 5 Sent Done Done false
16746 5215 Acked Write 5 Sent Done Done false
16747 5216 Acked Read 4 Sent Done Done false
16748 5217 Acked Write 9 Sent Done Done false
16749 5218 Acked Read 9 Sent Done Done false
16750 5219 Acked Read 6 Sent Done Done false
16751 5220 Acked Read 3 Sent Done Done false
16752 5221 Acked Write 6 Sent Done Done false
16753 5222 Acked Read 5 Sent Done Done false
16754 5223 Acked Read 1 Sent Done Done false
16755 5224 Acked Read 2 Sent Done Done false
16756 5225 Acked Write 1 Sent Done Done false
16757 STATES DS:0 DS:1 DS:2 TOTAL
16758 Sent 301 0 0 301
16759 Done 0 301 301 602
16760 Skipped 0 0 0 0
16761 Error 0 0 0 0
16762 Last Flush: Some(JobId(4924)) Some(JobId(5211)) Some(JobId(5211))
16763 Downstairs last five completed: 4924 4923 4922 4921 4920
16764 Upstairs last five completed: 5225 5224 5223 5222 5221
16765 CLIENT: Up:0 ds:301 act:2
167662025-02-25T20:50:16.694ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167672025-02-25T20:50:17.405ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167682025-02-25T20:50:17.932ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167692025-02-25T20:50:17.932ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56450) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167702025-02-25T20:50:17.933ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167712025-02-25T20:50:17.934ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167722025-02-25T20:50:17.934ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167732025-02-25T20:50:17.934ZINFOcrucible: send last flush ID to this DS: Some(JobId(4924)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167742025-02-25T20:50:17.934ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167752025-02-25T20:50:17.934ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4924)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167762025-02-25T20:50:17.934ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4924)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
16777 ----------------------------------------------------------------
16778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16779 Downstairs last five completed: 5226 5225 5224 5223 5222
16780 Upstairs last five completed: 5226 5225 5224 5223 5222
16781 CLIENT: Up:0 ds:0 act:3
16782 Replay: All jobs finished
16783 Main task has work to do, go find it
16784 got dsc Stop(0)
16785 stop 0
16786 Feb 25 20:50:20.195 INFO request completed, [8810] Got stop action so:false kr:false
16787 latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 65d9e09e-e371-46c7-9978-2ceccf02d123, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
167882025-02-25T20:50:20.196ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167892025-02-25T20:50:20.196ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
167902025-02-25T20:50:20.196ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
16791 Feb 25 20:50:20.196 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2522c83e-4747-4d31-92e6-55f48b9fc2d1, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
167922025-02-25T20:50:20.196ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
16793 [8810] Exited with: ExitStatus(unix_wait_status(9))
16794 [8810][0] reports Exit
16795 Feb 25 20:50:24.198 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: 98844764-5ec0-4d05-b9b1-a30a991a1b94, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
16796 001/300 Read block 2850 len 40960
16797 002/300 Write block 2221 len 36864 data: 4 5 5 4 3 3 2 2 4
16798 003/300 Read block 3676 len 24576
16799 004/300 Read block 702 len 12288
16800 005/300 Read block 676 len 36864
16801 006/300 Read block 664 len 12288
16802 007/300 Read block 4765 len 40960
16803 008/300 Read block 2668 len 12288
16804 009/300 Read block 1060 len 36864
16805 010/300 Flush
16806 011/300 Write block 4447 len 16384 data: 3 2 2 2
16807 012/300 Read block 4079 len 16384
16808 013/300 Read block 4323 len 28672
16809 014/300 Write block 4633 len 20480 data: 3 4 5 5 4
16810 015/300 Read block 4398 len 28672
16811 016/300 Flush
16812 017/300 Write block 2184 len 36864 data: 2 3 3 5 6 6 5 5 5
16813 018/300 Read block 400 len 28672
16814 019/300 Write block 238 len 16384 data: 4 4 3 3
16815 020/300 Read block 37 len 36864
16816 021/300 Write block 4689 len 8192 data: 5 6
16817 022/300 Write block 653 len 32768 data: 2 3 3 4 4 4 4 5
16818 023/300 Read block 2228 len 16384
16819 024/300 Read block 4273 len 24576
16820 025/300 Write block 4270 len 4096 data: 3
16821 026/300 Read block 2842 len 16384
16822 027/300 Read block 1763 len 24576
16823 028/300 Write block 4472 len 12288 data: 3 4 4
16824 029/300 Write block 4029 len 40960 data: 4 3 3 3 3 3 3 3 4 4
16825 030/300 Read block 4165 len 28672
16826 031/300 Write block 873 len 28672 data: 2 2 2 2 2 2 2
16827 032/300 Flush
16828 033/300 Write block 2083 len 28672 data: 3 3 4 4 4 4 4
16829 034/300 Read block 978 len 32768
16830 035/300 Write block 2596 len 20480 data: 4 3 3 3 3
16831 036/300 Write block 2450 len 12288 data: 8 7 6
16832 037/300 Write block 61 len 12288 data: 5 4 3
16833 038/300 Write block 2285 len 4096 data: 4
16834 039/300 Read block 1002 len 8192
16835 040/300 Write block 726 len 12288 data: 7 6 7
16836 041/300 Write block 2895 len 28672 data: 5 5 5 5 5 5 5
16837 042/300 Write block 2217 len 20480 data: 4 3 4 4 5
16838 043/300 Read block 4229 len 16384
16839 044/300 Read block 4962 len 8192
16840 045/300 Write block 1245 len 24576 data: 3 3 4 4 4 3
16841 046/300 Read block 3052 len 20480
16842 047/300 Read block 3105 len 28672
16843 048/300 Write block 1260 len 20480 data: 5 5 5 5 5
16844 049/300 Read block 1189 len 36864
16845 050/300 Read block 2859 len 32768
16846 051/300 Read block 1453 len 28672
16847 052/300 Read block 2923 len 40960
16848 053/300 Read block 2789 len 24576
16849 054/300 Read block 1868 len 40960
16850 055/300 Write block 1033 len 8192 data: 4 4
16851 056/300 Read block 2326 len 12288
16852 057/300 Read block 2528 len 40960
16853 058/300 Write block 1072 len 16384 data: 3 5 4 3
16854 059/300 Write block 620 len 8192 data: 4 4
16855 060/300 Read block 2689 len 40960
16856 061/300 Write block 2457 len 32768 data: 4 4 4 3 2 2 2 3
16857 062/300 Write block 982 len 16384 data: 5 6 7 8
16858 063/300 Write block 1851 len 4096 data: 4
16859 064/300 Read block 3317 len 20480
16860 065/300 Read block 3348 len 40960
16861 066/300 Write block 4360 len 20480 data: 4 4 4 5 4
16862 067/300 Write block 3109 len 32768 data: 2 2 3 5 5 4 4 5
16863 068/300 Read block 779 len 36864
16864 069/300 Read block 1397 len 12288
16865 070/300 Read block 3645 len 28672
16866 071/300 Write block 3955 len 20480 data: 3 5 4 5 4
16867 072/300 Read block 4807 len 12288
16868 073/300 Write block 4436 len 8192 data: 4 4
16869 074/300 Read block 3962 len 24576
16870 075/300 Write block 1988 len 40960 data: 4 4 4 5 5 5 5 2 3 3
16871 076/300 Write block 1182 len 20480 data: 4 4 4 4 5
16872 077/300 Read block 1441 len 4096
16873 078/300 Write block 2039 len 4096 data: 3
16874 079/300 Write block 1966 len 40960 data: 4 4 4 3 4 3 3 3 4 3
16875 080/300 Flush
16876 081/300 Read block 1300 len 20480
16877 082/300 Flush
16878 083/300 Flush
16879 084/300 Flush
16880 085/300 Read block 550 len 40960
16881 086/300 Write block 2893 len 4096 data: 5
16882 087/300 Write block 2732 len 16384 data: 2 3 3 3
16883 088/300 Write block 4273 len 36864 data: 2 2 2 2 2 3 4 4 3
16884 089/300 Read block 4514 len 20480
16885 090/300 Flush
16886 091/300 Read block 4678 len 40960
16887 092/300 Write block 4881 len 4096 data: 5
16888 093/300 Read block 3815 len 32768
16889 094/300 Write block 2121 len 32768 data: 2 3 3 4 4 4 4 4
16890 095/300 Write block 1059 len 12288 data: 5 5 5
16891 096/300 Write block 2239 len 36864 data: 3 3 3 3 5 8 6 6 5
16892 097/300 Read block 3490 len 32768
16893 098/300 Write block 4382 len 24576 data: 3 3 3 2 3 4
16894 099/300 Read block 2663 len 24576
16895 100/300 Write block 4798 len 32768 data: 8 8 5 3 2 2 3 4
16896 101/300 Write block 553 len 40960 data: 4 4 4 4 2 2 2 2 3 2
16897 102/300 Read block 4486 len 16384
16898 103/300 Read block 4632 len 16384
16899 104/300 Read block 4424 len 8192
16900 105/300 Read block 20 len 8192
16901 106/300 Write block 632 len 16384 data: 3 2 2 2
16902 107/300 Write block 3489 len 8192 data: 4 5
16903 108/300 Read block 926 len 40960
16904 109/300 Write block 3150 len 36864 data: 3 3 4 4 5 5 6 6 6
16905 110/300 Write block 1807 len 28672 data: 4 2 2 2 2 4 5
16906 111/300 Read block 3390 len 28672
16907 112/300 Write block 4892 len 20480 data: 5 6 5 5 7
16908 113/300 Write block 2274 len 20480 data: 4 4 3 3 3
16909 114/300 Read block 3883 len 12288
16910 115/300 Read block 786 len 32768
16911 116/300 Write block 3074 len 20480 data: 4 5 3 3 3
16912 117/300 Read block 1010 len 24576
16913 118/300 Read block 3950 len 4096
16914 119/300 Read block 2436 len 24576
16915 120/300 Read block 590 len 32768
16916 121/300 Read block 3352 len 28672
16917 122/300 Write block 440 len 36864 data: 6 5 5 5 3 3 3 3 3
16918 123/300 Write block 2808 len 40960 data: 3 3 2 2 2 3 2 2 2 2
16919 124/300 Write block 1985 len 4096 data: 3
16920 125/300 Write block 4558 len 8192 data: 2 2
16921 126/300 Read block 3657 len 32768
16922 127/300 Write block 525 len 36864 data: 3 3 3 3 4 4 5 5 4
16923 128/300 Read block 890 len 8192
16924 129/300 Read block 805 len 20480
16925 130/300 Write block 35 len 12288 data: 4 3 3
16926 131/300 Read block 2048 len 4096
16927 132/300 Write block 2781 len 16384 data: 2 2 3 2
16928 133/300 Write block 3160 len 4096 data: 4
16929 134/300 Read block 937 len 24576
16930 135/300 Read block 554 len 28672
16931 136/300 Read block 1309 len 32768
16932 137/300 Write block 1865 len 12288 data: 4 3 3
16933 138/300 Read block 1555 len 40960
16934 139/300 Write block 1977 len 36864 data: 2 2 3 3 4 4 4 3 4
16935 140/300 Read block 3494 len 12288
16936 141/300 Write block 2336 len 32768 data: 3 5 6 6 6 7 7 7
16937 142/300 Read block 2461 len 8192
16938 143/300 Read block 2073 len 40960
16939 144/300 Read block 1889 len 12288
16940 145/300 Flush
16941 146/300 Write block 3814 len 24576 data: 3 2 2 4 4 5
16942 147/300 Read block 1197 len 40960
16943 148/300 Read block 4213 len 12288
16944 149/300 Read block 1030 len 16384
16945 150/300 Write block 1893 len 32768 data: 4 4 4 3 3 4 4 4
16946 151/300 Flush
16947 152/300 Read block 1557 len 36864
16948 153/300 Read block 2987 len 12288
16949 154/300 Write block 2053 len 12288 data: 4 3 4
16950 155/300 Write block 2309 len 8192 data: 2 3
16951 156/300 Read block 383 len 16384
16952 157/300 Write block 3329 len 12288 data: 4 4 4
16953 158/300 Read block 1565 len 12288
16954 159/300 Write block 4651 len 28672 data: 2 3 3 3 3 3 4
16955 160/300 Write block 1330 len 24576 data: 4 3 3 3 3 3
16956 161/300 Read block 2754 len 8192
16957 162/300 Write block 2792 len 40960 data: 3 3 3 3 3 4 3 4 7 6
16958 163/300 Write block 3009 len 20480 data: 8 7 7 7 6
16959 164/300 Flush
16960 165/300 Read block 2710 len 16384
16961 166/300 Write block 724 len 32768 data: 5 7 8 7 8 8 6 5
16962 167/300 Write block 3656 len 32768 data: 5 5 7 6 6 5 5 5
16963 168/300 Write block 4277 len 40960 data: 3 4 5 5 4 3 3 3 3 3
16964 169/300 Read block 3706 len 8192
16965 170/300 Read block 3232 len 8192
16966 171/300 Read block 3629 len 8192
16967 172/300 Read block 550 len 8192
16968 173/300 Read block 3463 len 40960
16969 174/300 Write block 1633 len 32768 data: 4 5 5 4 3 3 3 2
16970 175/300 Write block 1917 len 16384 data: 3 4 3 3
16971 176/300 Write block 3783 len 16384 data: 3 3 3 4
16972 177/300 Write block 3626 len 36864 data: 4 5 5 7 7 5 4 6 6
16973 178/300 Read block 4441 len 20480
16974 179/300 Write block 1683 len 16384 data: 3 3 2 2
16975 180/300 Write block 1980 len 40960 data: 4 5 5 5 4 5 3 4 5 5
16976 181/300 Write block 2949 len 4096 data: 2
16977 182/300 Read block 4298 len 8192
16978 183/300 Write block 3903 len 28672 data: 4 4 4 3 3 3 2
16979 184/300 Read block 5 len 20480
16980 185/300 Read block 1329 len 4096
16981 186/300 Read block 1253 len 16384
16982 187/300 Flush
16983 188/300 Read block 614 len 4096
16984 189/300 Read block 2070 len 12288
16985 190/300 Read block 4475 len 16384
16986 191/300 Write block 2894 len 32768 data: 5 6 6 6 6 6 6 6
16987 192/300 Write block 430 len 32768 data: 4 4 3 2 3 3 3 5
16988 193/300 Read block 3056 len 28672
16989 194/300 Write block 243 len 12288 data: 2 3 3
16990 195/300 Read block 2879 len 36864
16991 196/300 Write block 1605 len 36864 data: 4 5 5 4 3 4 4 5 6
16992 197/300 Read block 2788 len 28672
16993 198/300 Flush
16994 199/300 Flush
16995 200/300 Read block 2631 len 40960
16996 201/300 Flush
16997 202/300 Read block 2352 len 28672
16998 203/300 Read block 4472 len 40960
16999 204/300 Read block 925 len 12288
17000 205/300 Write block 704 len 28672 data: 5 6 7 6 4 4 4
17001 206/300 Flush
17002 207/300 Read block 3046 len 12288
17003 208/300 Write block 3595 len 12288 data: 4 5 4
17004 209/300 Read block 2967 len 16384
17005 210/300 Read block 4837 len 24576
17006 211/300 Read block 2931 len 32768
17007 212/300 Write block 3254 len 12288 data: 5 5 3
17008 213/300 Write block 2409 len 36864 data: 2 2 2 3 4 4 4 2 2
17009 214/300 Write block 221 len 36864 data: 5 5 4 4 3 5 5 5 3
17010 215/300 Write block 3586 len 24576 data: 3 3 3 3 2 2
17011 216/300 Write block 4 len 40960 data: 2 2 3 3 3 4 4 4 3 3
17012 217/300 Flush
17013 218/300 Read block 2538 len 8192
17014 219/300 Read block 1570 len 32768
17015 220/300 Flush
17016 221/300 Write block 2555 len 12288 data: 4 4 4
17017 222/300 Read block 4765 len 20480
17018 223/300 Read block 3636 len 28672
17019 224/300 Write block 3684 len 40960 data: 3 3 3 3 3 2 3 3 3 3
17020 225/300 Flush
17021 226/300 Write block 345 len 28672 data: 4 4 4 4 4 4 4
17022 227/300 Write block 4881 len 40960 data: 6 4 5 5 5 4 5 4 5 5
17023 228/300 Read block 4954 len 16384
17024 229/300 Write block 2618 len 40960 data: 4 5 5 5 5 6 5 6 6 5
17025 230/300 Read block 2471 len 12288
17026 231/300 Read block 3798 len 40960
17027 232/300 Read block 4009 len 12288
17028 233/300 Read block 2329 len 24576
17029 234/300 Write block 4958 len 40960 data: 2 2 2 2 2 2 4 4 4 5
17030 235/300 Write block 3971 len 36864 data: 5 5 5 5 5 4 5 4 5
17031 236/300 Read block 3512 len 24576
17032 237/300 Flush
17033 238/300 Read block 4668 len 28672
17034 239/300 Read block 4104 len 32768
17035 240/300 Write block 2574 len 16384 data: 4 2 2 2
17036 241/300 Flush
17037 242/300 Write block 2530 len 40960 data: 4 4 5 5 6 4 3 3 4 4
17038 243/300 Write block 2075 len 8192 data: 3 4
17039 244/300 Write block 3631 len 20480 data: 6 5 7 7 7
17040 245/300 Read block 683 len 28672
17041 246/300 Read block 4575 len 28672
17042 247/300 Read block 337 len 12288
17043 248/300 Write block 2000 len 8192 data: 5 4
17044 249/300 Write block 1076 len 12288 data: 3 2 2
17045 250/300 Read block 542 len 16384
17046 251/300 Read block 298 len 40960
17047 252/300 Read block 701 len 16384
17048 253/300 Write block 3028 len 40960 data: 4 4 4 3 3 2 2 3 2 2
17049 254/300 Write block 3213 len 16384 data: 3 2 2 3
17050 255/300 Read block 142 len 24576
17051 256/300 Write block 1914 len 20480 data: 4 4 3 4 5
17052 257/300 Read block 4098 len 24576
17053 258/300 Read block 449 len 28672
17054 259/300 Read block 799 len 24576
17055 260/300 Read block 2736 len 16384
17056 261/300 Write block 2464 len 4096 data: 4
17057 262/300 Read block 539 len 32768
17058 263/300 Read block 1871 len 24576
17059 264/300 Read block 48 len 36864
17060 265/300 Read block 3043 len 40960
17061 266/300 Read block 4628 len 16384
17062 267/300 Read block 301 len 28672
17063 268/300 Read block 1291 len 32768
17064 269/300 Read block 3818 len 8192
17065 270/300 Write block 555 len 24576 data: 5 5 3 3 3 3
17066 271/300 Flush
17067 272/300 Write block 1312 len 36864 data: 2 2 2 2 2 3 4 5 7
17068 273/300 Write block 3380 len 32768 data: 5 6 5 5 5 5 5 3
17069 274/300 Write block 3110 len 4096 data: 3
17070 275/300 Write block 3944 len 16384 data: 4 5 6 5
17071 276/300 Read block 2357 len 36864
17072 277/300 Write block 4452 len 12288 data: 2 2 5
17073 278/300 Write block 4459 len 16384 data: 5 4 6 6
17074 279/300 Read block 2491 len 24576
17075 280/300 Read block 1565 len 28672
17076 281/300 Read block 370 len 32768
17077 282/300 Write block 3609 len 20480 data: 2 2 4 5 5
17078 283/300 Read block 1662 len 8192
17079 284/300 Write block 3914 len 32768 data: 3 4 4 4 3 3 3 3
17080 285/300 Read block 868 len 32768
17081 286/300 Write block 3990 len 28672 data: 5 5 5 4 4 5 4
17082 287/300 Write block 1351 len 4096 data: 2
17083 288/300 Flush
17084 289/300 Write block 1671 len 20480 data: 5 5 5 4 4
17085 290/300 Read block 399 len 24576
17086 291/300 Write block 46 len 8192 data: 3 4
17087 292/300 Read block 2405 len 32768
17088 293/300 Flush
17089 294/300 Read block 2700 len 12288
17090 295/300 Write block 2625 len 8192 data: 7 7
17091 296/300 Flush
17092 297/300 Write block 4169 len 8192 data: 7 7
17093 298/300 Write block 1270 len 8192 data: 3 3
17094 299/300 Write block 4790 len 4096 data: 4
17095 300/300 Write block 2758 len 12288 data: 6 4 4
17096 301/300 Read block 3368 len 4096
17097 Main task has work to do, go find it
17098 got dsc Start(0)
17099 start 0
17100 Feb 25 20:50:24.448 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/0, method: POST, req_id: 014a8bc1-49ec-4976-9951-06ba49971098, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
17101 Replay: started 0, returned:Ok(())
17102 ----------------------------------------------------------------
17103 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17104 States: Offline Active Active
17105 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17106 5227 Acked Read 10 Sent Done Done false
17107 5228 Acked Write 9 Sent Done Done false
17108 5229 Acked Read 6 Sent Done Done false
17109 5230 Acked Read 3 Sent Done Done false
17110 5231 Acked Read 9 Sent Done Done false
17111 5232 Acked Read 3 Sent Done Done false
17112 5233 Acked Read 10 Sent Done Done false
17113 5234 Acked Read 3 Sent Done Done false
17114 5235 Acked Read 9 Sent Done Done false
17115 5236 Acked Flush 0 Sent Done Done false
17116 5237 Acked Write 4 Sent Done Done false
17117 5238 Acked Read 4 Sent Done Done false
17118 5239 Acked Read 7 Sent Done Done false
17119 5240 Acked Write 5 Sent Done Done false
17120 5241 Acked Read 7 Sent Done Done false
17121 5242 Acked Flush 0 Sent Done Done false
17122 5243 Acked Write 9 Sent Done Done false
17123 5244 Acked Read 7 Sent Done Done false
17124 5245 Acked Write 4 Sent Done Done false
17125 5246 Acked Read 9 Sent Done Done false
17126 5247 Acked Write 2 Sent Done Done false
17127 5248 Acked Write 8 Sent Done Done false
17128 5249 Acked Read 4 Sent Done Done false
17129 5250 Acked Read 6 Sent Done Done false
17130 5251 Acked Write 1 Sent Done Done false
17131 5252 Acked Read 4 Sent Done Done false
17132 5253 Acked Read 6 Sent Done Done false
17133 5254 Acked Write 3 Sent Done Done false
17134 5255 Acked Write 10 Sent Done Done false
17135 5256 Acked Read 7 Sent Done Done false
17136 5257 Acked Write 7 Sent Done Done false
17137 5258 Acked Flush 0 Sent Done Done false
17138 5259 Acked Write 7 Sent Done Done false
17139 5260 Acked Read 8 Sent Done Done false
17140 5261 Acked Write 5 Sent Done Done false
17141 5262 Acked Write 3 Sent Done Done false
17142 5263 Acked Write 3 Sent Done Done false
17143 5264 Acked Write 1 Sent Done Done false
17144 5265 Acked Read 2 Sent Done Done false
17145 5266 Acked Write 3 Sent Done Done false
17146 5267 Acked Write 7 Sent Done Done false
17147 5268 Acked Write 5 Sent Done Done false
17148 5269 Acked Read 4 Sent Done Done false
17149 5270 Acked Read 2 Sent Done Done false
17150 5271 Acked Write 6 Sent Done Done false
17151 5272 Acked Read 5 Sent Done Done false
17152 5273 Acked Read 7 Sent Done Done false
17153 5274 Acked Write 5 Sent Done Done false
17154 5275 Acked Read 9 Sent Done Done false
17155 5276 Acked Read 8 Sent Done Done false
17156 5277 Acked Read 7 Sent Done Done false
17157 5278 Acked Read 10 Sent Done Done false
17158 5279 Acked Read 6 Sent Done Done false
17159 5280 Acked Read 10 Sent Done Done false
17160 5281 Acked Write 2 Sent Done Done false
17161 5282 Acked Read 3 Sent Done Done false
17162 5283 Acked Read 10 Sent Done Done false
17163 5284 Acked Write 4 Sent Done Done false
17164 5285 Acked Write 2 Sent Done Done false
17165 5286 Acked Read 10 Sent Done Done false
17166 5287 Acked Write 8 Sent Done Done false
17167 5288 Acked Write 4 Sent Done Done false
17168 5289 Acked Write 1 Sent Done Done false
17169 5290 Acked Read 5 Sent Done Done false
17170 5291 Acked Read 10 Sent Done Done false
17171 5292 Acked Write 5 Sent Done Done false
17172 5293 Acked Write 8 Sent Done Done false
17173 5294 Acked Read 9 Sent Done Done false
17174 5295 Acked Read 3 Sent Done Done false
17175 5296 Acked Read 7 Sent Done Done false
17176 5297 Acked Write 5 Sent Done Done false
17177 5298 Acked Read 3 Sent Done Done false
17178 5299 Acked Write 2 Sent Done Done false
17179 5300 Acked Read 6 Sent Done Done false
17180 5301 Acked Write 10 Sent Done Done false
17181 5302 Acked Write 5 Sent Done Done false
17182 5303 Acked Read 1 Sent Done Done false
17183 5304 Acked Write 1 Sent Done Done false
17184 5305 Acked Write 10 Sent Done Done false
17185 5306 Acked Flush 0 Sent Done Done false
17186 5307 Acked Read 5 Sent Done Done false
17187 5308 Acked Flush 0 Sent Done Done false
17188 5309 Acked Flush 0 Sent Done Done false
17189 5310 Acked Flush 0 Sent Done Done false
17190 5311 Acked Read 10 Sent Done Done false
17191 5312 Acked Write 1 Sent Done Done false
17192 5313 Acked Write 4 Sent Done Done false
17193 5314 Acked Write 9 Sent Done Done false
17194 5315 Acked Read 5 Sent Done Done false
17195 5316 Acked Flush 0 Sent Done Done false
17196 5317 Acked Read 10 Sent Done Done false
17197 5318 Acked Write 1 Sent Done Done false
17198 5319 Acked Read 8 Sent Done Done false
17199 5320 Acked Write 8 Sent Done Done false
17200 5321 Acked Write 3 Sent Done Done false
17201 5322 Acked Write 9 Sent Done Done false
17202 5323 Acked Read 8 Sent Done Done false
17203 5324 Acked Write 6 Sent Done Done false
17204 5325 Acked Read 6 Sent Done Done false
17205 5326 Acked Write 8 Sent Done Done false
17206 5327 Acked Write 10 Sent Done Done false
17207 5328 Acked Read 4 Sent Done Done false
17208 5329 Acked Read 4 Sent Done Done false
17209 5330 Acked Read 2 Sent Done Done false
17210 5331 Acked Read 2 Sent Done Done false
17211 5332 Acked Write 4 Sent Done Done false
17212 5333 Acked Write 2 Sent Done Done false
17213 5334 Acked Read 10 Sent Done Done false
17214 5335 Acked Write 9 Sent Done Done false
17215 5336 Acked Write 7 Sent Done Done false
17216 5337 Acked Read 7 Sent Done Done false
17217 5338 Acked Write 5 Sent Done Done false
17218 5339 Acked Write 5 Sent Done Done false
17219 5340 Acked Read 3 Sent Done Done false
17220 5341 Acked Read 8 Sent Done Done false
17221 5342 Acked Write 5 Sent Done Done false
17222 5343 Acked Read 6 Sent Done Done false
17223 5344 Acked Read 1 Sent Done Done false
17224 5345 Acked Read 6 Sent Done Done false
17225 5346 Acked Read 8 Sent Done Done false
17226 5347 Acked Read 7 Sent Done Done false
17227 5348 Acked Write 9 Sent Done Done false
17228 5349 Acked Write 10 Sent Done Done false
17229 5350 Acked Write 1 Sent Done Done false
17230 5351 Acked Write 2 Sent Done Done false
17231 5352 Acked Read 8 Sent Done Done false
17232 5353 Acked Write 9 Sent Done Done false
17233 5354 Acked Read 2 Sent Done Done false
17234 5355 Acked Read 5 Sent Done Done false
17235 5356 Acked Write 3 Sent Done Done false
17236 5357 Acked Read 1 Sent Done Done false
17237 5358 Acked Write 4 Sent Done Done false
17238 5359 Acked Write 1 Sent Done Done false
17239 5360 Acked Read 6 Sent Done Done false
17240 5361 Acked Read 7 Sent Done Done false
17241 5362 Acked Read 8 Sent Done Done false
17242 5363 Acked Write 3 Sent Done Done false
17243 5364 Acked Read 10 Sent Done Done false
17244 5365 Acked Write 9 Sent Done Done false
17245 5366 Acked Read 3 Sent Done Done false
17246 5367 Acked Write 8 Sent Done Done false
17247 5368 Acked Read 2 Sent Done Done false
17248 5369 Acked Read 10 Sent Done Done false
17249 5370 Acked Read 3 Sent Done Done false
17250 5371 Acked Flush 0 Sent Done Done false
17251 5372 Acked Write 6 Sent Done Done false
17252 5373 Acked Read 10 Sent Done Done false
17253 5374 Acked Read 3 Sent Done Done false
17254 5375 Acked Read 4 Sent Done Done false
17255 5376 Acked Write 8 Sent Done Done false
17256 5377 Acked Flush 0 Sent Done Done false
17257 5378 Acked Read 9 Sent Done Done false
17258 5379 Acked Read 3 Sent Done Done false
17259 5380 Acked Write 3 Sent Done Done false
17260 5381 Acked Write 2 Sent Done Done false
17261 5382 Acked Read 4 Sent Done Done false
17262 5383 Acked Write 3 Sent Done Done false
17263 5384 Acked Read 3 Sent Done Done false
17264 5385 Acked Write 7 Sent Done Done false
17265 5386 Acked Write 6 Sent Done Done false
17266 5387 Acked Read 2 Sent Done Done false
17267 5388 Acked Write 10 Sent Done Done false
17268 5389 Acked Write 5 Sent Done Done false
17269 5390 Acked Flush 0 Sent Done Done false
17270 5391 Acked Read 4 Sent Done Done false
17271 5392 Acked Write 8 Sent Done Done false
17272 5393 Acked Write 8 Sent Done Done false
17273 5394 Acked Write 10 Sent Done Done false
17274 5395 Acked Read 2 Sent Done Done false
17275 5396 Acked Read 2 Sent Done Done false
17276 5397 Acked Read 2 Sent Done Done false
17277 5398 Acked Read 2 Sent Done Done false
17278 5399 Acked Read 10 Sent Done Done false
17279 5400 Acked Write 8 Sent Done Done false
17280 5401 Acked Write 4 Sent Done Done false
17281 5402 Acked Write 4 Sent Done Done false
17282 5403 Acked Write 9 Sent Done Done false
17283 5404 Acked Read 5 Sent Done Done false
17284 5405 Acked Write 4 Sent Done Done false
17285 5406 Acked Write 10 Sent Done Done false
17286 5407 Acked Write 1 Sent Done Done false
17287 5408 Acked Read 2 Sent Done Done false
17288 5409 Acked Write 7 Sent Done Done false
17289 5410 Acked Read 5 Sent Done Done false
17290 5411 Acked Read 1 Sent Done Done false
17291 5412 Acked Read 4 Sent Done Done false
17292 5413 Acked Flush 0 Sent Done Done false
17293 5414 Acked Read 1 Sent Done Done false
17294 5415 Acked Read 3 Sent Done Done false
17295 5416 Acked Read 4 Sent Done Done false
17296 5417 Acked Write 8 Sent Done Done false
17297 5418 Acked Write 8 Sent Done Done false
17298 5419 Acked Read 7 Sent Done Done false
17299 5420 Acked Write 3 Sent Done Done false
17300 5421 Acked Read 9 Sent Done Done false
17301 5422 Acked Write 9 Sent Done Done false
17302 5423 Acked Read 7 Sent Done Done false
17303 5424 Acked Flush 0 Sent Done Done false
17304 5425 Acked Flush 0 Sent Done Done false
17305 5426 Acked Read 10 Sent Done Done false
17306 5427 Acked Flush 0 Sent Done Done false
17307 5428 Acked Read 7 Sent Done Done false
17308 5429 Acked Read 10 Sent Done Done false
17309 5430 Acked Read 3 Sent Done Done false
17310 5431 Acked Write 7 Sent Done Done false
17311 5432 Acked Flush 0 Sent Done Done false
17312 5433 Acked Read 3 Sent Done Done false
17313 5434 Acked Write 3 Sent Done Done false
17314 5435 Acked Read 4 Sent Done Done false
17315 5436 Acked Read 6 Sent Done Done false
17316 5437 Acked Read 8 Sent Done Done false
17317 5438 Acked Write 3 Sent Done Done false
17318 5439 Acked Write 9 Sent Done Done false
17319 5440 Acked Write 9 Sent Done Done false
17320 5441 Acked Write 6 Sent Done Done false
17321 5442 Acked Write 10 Sent Done Done false
17322 5443 Acked Flush 0 Sent Done Done false
17323 5444 Acked Read 2 Sent Done Done false
17324 5445 Acked Read 8 Sent Done Done false
17325 5446 Acked Flush 0 Sent Done Done false
17326 5447 Acked Write 3 Sent Done Done false
17327 5448 Acked Read 5 Sent Done Done false
17328 5449 Acked Read 7 Sent Done Done false
17329 5450 Acked Write 10 Sent Done Done false
17330 5451 Acked Flush 0 Sent Done Done false
17331 5452 Acked Write 7 Sent Done Done false
17332 5453 Acked Write 10 Sent Done Done false
17333 5454 Acked Read 4 Sent Done Done false
17334 5455 Acked Write 10 Sent Done Done false
17335 5456 Acked Read 3 Sent Done Done false
17336 5457 Acked Read 10 Sent Done Done false
17337 5458 Acked Read 3 Sent Done Done false
17338 5459 Acked Read 6 Sent Done Done false
17339 5460 Acked Write 10 Sent Done Done false
17340 5461 Acked Write 9 Sent Done Done false
17341 5462 Acked Read 6 Sent Done Done false
17342 5463 Acked Flush 0 Sent Done Done false
17343 5464 Acked Read 7 Sent Done Done false
17344 5465 Acked Read 8 Sent Done Done false
17345 5466 Acked Write 4 Sent Done Done false
17346 5467 Acked Flush 0 Sent Done Done false
17347 5468 Acked Write 10 Sent Done Done false
17348 5469 Acked Write 2 Sent Done Done false
17349 5470 Acked Write 5 Sent Done Done false
17350 5471 Acked Read 7 Sent Done Done false
17351 5472 Acked Read 7 Sent Done Done false
17352 5473 Acked Read 3 Sent Done Done false
17353 5474 Acked Write 2 Sent Done Done false
17354 5475 Acked Write 3 Sent Done Done false
17355 5476 Acked Read 4 Sent Done Done false
17356 5477 Acked Read 10 Sent Done Done false
17357 5478 Acked Read 4 Sent Done Done false
17358 5479 Acked Write 10 Sent Done Done false
17359 5480 Acked Write 4 Sent Done Done false
17360 5481 Acked Read 6 Sent Done Done false
17361 5482 Acked Write 5 Sent Done Done false
17362 5483 Acked Read 6 Sent Done Done false
17363 5484 Acked Read 7 Sent Done Done false
17364 5485 Acked Read 6 Sent Done Done false
17365 5486 Acked Read 4 Sent Done Done false
17366 5487 Acked Write 1 Sent Done Done false
17367 5488 Acked Read 8 Sent Done Done false
17368 5489 Acked Read 6 Sent Done Done false
17369 5490 Acked Read 9 Sent Done Done false
17370 5491 Acked Read 10 Sent Done Done false
17371 5492 Acked Read 4 Sent Done Done false
17372 5493 Acked Read 7 Sent Done Done false
17373 5494 Acked Read 8 Sent Done Done false
17374 5495 Acked Read 2 Sent Done Done false
17375 5496 Acked Write 6 Sent Done Done false
17376 5497 Acked Flush 0 Sent Done Done false
17377 5498 Acked Write 9 Sent Done Done false
17378 5499 Acked Write 8 Sent Done Done false
17379 5500 Acked Write 1 Sent Done Done false
17380 5501 Acked Write 4 Sent Done Done false
17381 5502 Acked Read 9 Sent Done Done false
17382 5503 Acked Write 3 Sent Done Done false
17383 5504 Acked Write 4 Sent Done Done false
17384 5505 Acked Read 6 Sent Done Done false
17385 5506 Acked Read 7 Sent Done Done false
17386 5507 Acked Read 8 Sent Done Done false
17387 5508 Acked Write 5 Sent Done Done false
17388 5509 Acked Read 2 Sent Done Done false
17389 5510 Acked Write 8 Sent Done Done false
17390 5511 Acked Read 8 Sent Done Done false
17391 5512 Acked Write 7 Sent Done Done false
17392 5513 Acked Write 1 Sent Done Done false
17393 5514 Acked Flush 0 Sent Done Done false
17394 5515 Acked Write 5 Sent Done Done false
17395 5516 Acked Read 6 Sent Done Done false
17396 5517 Acked Write 2 Sent Done Done false
17397 5518 Acked Read 8 Sent Done Done false
17398 5519 Acked Flush 0 Sent Done Done false
17399 5520 Acked Read 3 Sent Done Done false
17400 5521 Acked Write 2 Sent Done Done false
17401 5522 Acked Flush 0 Sent Done Done false
17402 5523 Acked Write 2 Sent Done Done false
17403 5524 Acked Write 2 Sent Done Done false
17404 5525 Acked Write 1 Sent Done Done false
17405 5526 Acked Write 3 Sent Done Done false
17406 5527 Acked Read 1 Sent Done Done false
17407 STATES DS:0 DS:1 DS:2 TOTAL
17408 Sent 301 0 0 301
17409 Done 0 301 301 602
17410 Skipped 0 0 0 0
17411 Error 0 0 0 0
17412 Last Flush: Some(JobId(5226)) Some(JobId(5522)) Some(JobId(5522))
17413 Downstairs last five completed: 5226 5225 5224 5223 5222
17414 Upstairs last five completed: 5527 5526 5525 5524 5523
17415 CLIENT: Up:0 ds:301 act:2
174162025-02-25T20:50:25.421ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
17417 [8810] got start action
17418 [8810] I am going to restart
17419 Starting downstairs at port 8810
17420 Make output file at "/tmp/dsc/downstairs-8810.txt"
17421 [8810][0] reports Starting
17422 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
17423 [8810][0] reports Running
174242025-02-25T20:50:26.422ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
174252025-02-25T20:50:27.425ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
174262025-02-25T20:50:28.426ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
17427 ----------------------------------------------------------------
17428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17429 States: Offline Active Active
17430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17431 5227 Acked Read 10 Sent Done Done false
17432 5228 Acked Write 9 Sent Done Done false
17433 5229 Acked Read 6 Sent Done Done false
17434 5230 Acked Read 3 Sent Done Done false
17435 5231 Acked Read 9 Sent Done Done false
17436 5232 Acked Read 3 Sent Done Done false
17437 5233 Acked Read 10 Sent Done Done false
17438 5234 Acked Read 3 Sent Done Done false
17439 5235 Acked Read 9 Sent Done Done false
17440 5236 Acked Flush 0 Sent Done Done false
17441 5237 Acked Write 4 Sent Done Done false
17442 5238 Acked Read 4 Sent Done Done false
17443 5239 Acked Read 7 Sent Done Done false
17444 5240 Acked Write 5 Sent Done Done false
17445 5241 Acked Read 7 Sent Done Done false
17446 5242 Acked Flush 0 Sent Done Done false
17447 5243 Acked Write 9 Sent Done Done false
17448 5244 Acked Read 7 Sent Done Done false
17449 5245 Acked Write 4 Sent Done Done false
17450 5246 Acked Read 9 Sent Done Done false
17451 5247 Acked Write 2 Sent Done Done false
17452 5248 Acked Write 8 Sent Done Done false
17453 5249 Acked Read 4 Sent Done Done false
17454 5250 Acked Read 6 Sent Done Done false
17455 5251 Acked Write 1 Sent Done Done false
17456 5252 Acked Read 4 Sent Done Done false
17457 5253 Acked Read 6 Sent Done Done false
17458 5254 Acked Write 3 Sent Done Done false
17459 5255 Acked Write 10 Sent Done Done false
17460 5256 Acked Read 7 Sent Done Done false
17461 5257 Acked Write 7 Sent Done Done false
17462 5258 Acked Flush 0 Sent Done Done false
17463 5259 Acked Write 7 Sent Done Done false
17464 5260 Acked Read 8 Sent Done Done false
17465 5261 Acked Write 5 Sent Done Done false
17466 5262 Acked Write 3 Sent Done Done false
17467 5263 Acked Write 3 Sent Done Done false
17468 5264 Acked Write 1 Sent Done Done false
17469 5265 Acked Read 2 Sent Done Done false
17470 5266 Acked Write 3 Sent Done Done false
17471 5267 Acked Write 7 Sent Done Done false
17472 5268 Acked Write 5 Sent Done Done false
17473 5269 Acked Read 4 Sent Done Done false
17474 5270 Acked Read 2 Sent Done Done false
17475 5271 Acked Write 6 Sent Done Done false
17476 5272 Acked Read 5 Sent Done Done false
17477 5273 Acked Read 7 Sent Done Done false
17478 5274 Acked Write 5 Sent Done Done false
17479 5275 Acked Read 9 Sent Done Done false
17480 5276 Acked Read 8 Sent Done Done false
17481 5277 Acked Read 7 Sent Done Done false
17482 5278 Acked Read 10 Sent Done Done false
17483 5279 Acked Read 6 Sent Done Done false
17484 5280 Acked Read 10 Sent Done Done false
17485 5281 Acked Write 2 Sent Done Done false
17486 5282 Acked Read 3 Sent Done Done false
17487 5283 Acked Read 10 Sent Done Done false
17488 5284 Acked Write 4 Sent Done Done false
17489 5285 Acked Write 2 Sent Done Done false
17490 5286 Acked Read 10 Sent Done Done false
17491 5287 Acked Write 8 Sent Done Done false
17492 5288 Acked Write 4 Sent Done Done false
17493 5289 Acked Write 1 Sent Done Done false
17494 5290 Acked Read 5 Sent Done Done false
17495 5291 Acked Read 10 Sent Done Done false
17496 5292 Acked Write 5 Sent Done Done false
17497 5293 Acked Write 8 Sent Done Done false
17498 5294 Acked Read 9 Sent Done Done false
17499 5295 Acked Read 3 Sent Done Done false
17500 5296 Acked Read 7 Sent Done Done false
17501 5297 Acked Write 5 Sent Done Done false
17502 5298 Acked Read 3 Sent Done Done false
17503 5299 Acked Write 2 Sent Done Done false
17504 5300 Acked Read 6 Sent Done Done false
17505 5301 Acked Write 10 Sent Done Done false
17506 5302 Acked Write 5 Sent Done Done false
17507 5303 Acked Read 1 Sent Done Done false
17508 5304 Acked Write 1 Sent Done Done false
17509 5305 Acked Write 10 Sent Done Done false
17510 5306 Acked Flush 0 Sent Done Done false
17511 5307 Acked Read 5 Sent Done Done false
17512 5308 Acked Flush 0 Sent Done Done false
17513 5309 Acked Flush 0 Sent Done Done false
17514 5310 Acked Flush 0 Sent Done Done false
17515 5311 Acked Read 10 Sent Done Done false
17516 5312 Acked Write 1 Sent Done Done false
17517 5313 Acked Write 4 Sent Done Done false
17518 5314 Acked Write 9 Sent Done Done false
17519 5315 Acked Read 5 Sent Done Done false
17520 5316 Acked Flush 0 Sent Done Done false
17521 5317 Acked Read 10 Sent Done Done false
17522 5318 Acked Write 1 Sent Done Done false
17523 5319 Acked Read 8 Sent Done Done false
17524 5320 Acked Write 8 Sent Done Done false
17525 5321 Acked Write 3 Sent Done Done false
17526 5322 Acked Write 9 Sent Done Done false
17527 5323 Acked Read 8 Sent Done Done false
17528 5324 Acked Write 6 Sent Done Done false
17529 5325 Acked Read 6 Sent Done Done false
17530 5326 Acked Write 8 Sent Done Done false
17531 5327 Acked Write 10 Sent Done Done false
17532 5328 Acked Read 4 Sent Done Done false
17533 5329 Acked Read 4 Sent Done Done false
17534 5330 Acked Read 2 Sent Done Done false
17535 5331 Acked Read 2 Sent Done Done false
17536 5332 Acked Write 4 Sent Done Done false
17537 5333 Acked Write 2 Sent Done Done false
17538 5334 Acked Read 10 Sent Done Done false
17539 5335 Acked Write 9 Sent Done Done false
17540 5336 Acked Write 7 Sent Done Done false
17541 5337 Acked Read 7 Sent Done Done false
17542 5338 Acked Write 5 Sent Done Done false
17543 5339 Acked Write 5 Sent Done Done false
17544 5340 Acked Read 3 Sent Done Done false
17545 5341 Acked Read 8 Sent Done Done false
17546 5342 Acked Write 5 Sent Done Done false
17547 5343 Acked Read 6 Sent Done Done false
17548 5344 Acked Read 1 Sent Done Done false
17549 5345 Acked Read 6 Sent Done Done false
17550 5346 Acked Read 8 Sent Done Done false
17551 5347 Acked Read 7 Sent Done Done false
17552 5348 Acked Write 9 Sent Done Done false
17553 5349 Acked Write 10 Sent Done Done false
17554 5350 Acked Write 1 Sent Done Done false
17555 5351 Acked Write 2 Sent Done Done false
17556 5352 Acked Read 8 Sent Done Done false
17557 5353 Acked Write 9 Sent Done Done false
17558 5354 Acked Read 2 Sent Done Done false
17559 5355 Acked Read 5 Sent Done Done false
17560 5356 Acked Write 3 Sent Done Done false
17561 5357 Acked Read 1 Sent Done Done false
17562 5358 Acked Write 4 Sent Done Done false
17563 5359 Acked Write 1 Sent Done Done false
17564 5360 Acked Read 6 Sent Done Done false
17565 5361 Acked Read 7 Sent Done Done false
17566 5362 Acked Read 8 Sent Done Done false
17567 5363 Acked Write 3 Sent Done Done false
17568 5364 Acked Read 10 Sent Done Done false
17569 5365 Acked Write 9 Sent Done Done false
17570 5366 Acked Read 3 Sent Done Done false
17571 5367 Acked Write 8 Sent Done Done false
17572 5368 Acked Read 2 Sent Done Done false
17573 5369 Acked Read 10 Sent Done Done false
17574 5370 Acked Read 3 Sent Done Done false
17575 5371 Acked Flush 0 Sent Done Done false
17576 5372 Acked Write 6 Sent Done Done false
17577 5373 Acked Read 10 Sent Done Done false
17578 5374 Acked Read 3 Sent Done Done false
17579 5375 Acked Read 4 Sent Done Done false
17580 5376 Acked Write 8 Sent Done Done false
17581 5377 Acked Flush 0 Sent Done Done false
17582 5378 Acked Read 9 Sent Done Done false
17583 5379 Acked Read 3 Sent Done Done false
17584 5380 Acked Write 3 Sent Done Done false
17585 5381 Acked Write 2 Sent Done Done false
17586 5382 Acked Read 4 Sent Done Done false
17587 5383 Acked Write 3 Sent Done Done false
17588 5384 Acked Read 3 Sent Done Done false
17589 5385 Acked Write 7 Sent Done Done false
17590 5386 Acked Write 6 Sent Done Done false
17591 5387 Acked Read 2 Sent Done Done false
17592 5388 Acked Write 10 Sent Done Done false
17593 5389 Acked Write 5 Sent Done Done false
17594 5390 Acked Flush 0 Sent Done Done false
17595 5391 Acked Read 4 Sent Done Done false
17596 5392 Acked Write 8 Sent Done Done false
17597 5393 Acked Write 8 Sent Done Done false
17598 5394 Acked Write 10 Sent Done Done false
17599 5395 Acked Read 2 Sent Done Done false
17600 5396 Acked Read 2 Sent Done Done false
17601 5397 Acked Read 2 Sent Done Done false
17602 5398 Acked Read 2 Sent Done Done false
17603 5399 Acked Read 10 Sent Done Done false
17604 5400 Acked Write 8 Sent Done Done false
17605 5401 Acked Write 4 Sent Done Done false
17606 5402 Acked Write 4 Sent Done Done false
17607 5403 Acked Write 9 Sent Done Done false
17608 5404 Acked Read 5 Sent Done Done false
17609 5405 Acked Write 4 Sent Done Done false
17610 5406 Acked Write 10 Sent Done Done false
17611 5407 Acked Write 1 Sent Done Done false
17612 5408 Acked Read 2 Sent Done Done false
17613 5409 Acked Write 7 Sent Done Done false
17614 5410 Acked Read 5 Sent Done Done false
17615 5411 Acked Read 1 Sent Done Done false
17616 5412 Acked Read 4 Sent Done Done false
17617 5413 Acked Flush 0 Sent Done Done false
17618 5414 Acked Read 1 Sent Done Done false
17619 5415 Acked Read 3 Sent Done Done false
17620 5416 Acked Read 4 Sent Done Done false
17621 5417 Acked Write 8 Sent Done Done false
17622 5418 Acked Write 8 Sent Done Done false
17623 5419 Acked Read 7 Sent Done Done false
17624 5420 Acked Write 3 Sent Done Done false
17625 5421 Acked Read 9 Sent Done Done false
17626 5422 Acked Write 9 Sent Done Done false
17627 5423 Acked Read 7 Sent Done Done false
17628 5424 Acked Flush 0 Sent Done Done false
17629 5425 Acked Flush 0 Sent Done Done false
17630 5426 Acked Read 10 Sent Done Done false
17631 5427 Acked Flush 0 Sent Done Done false
17632 5428 Acked Read 7 Sent Done Done false
17633 5429 Acked Read 10 Sent Done Done false
17634 5430 Acked Read 3 Sent Done Done false
17635 5431 Acked Write 7 Sent Done Done false
17636 5432 Acked Flush 0 Sent Done Done false
17637 5433 Acked Read 3 Sent Done Done false
17638 5434 Acked Write 3 Sent Done Done false
17639 5435 Acked Read 4 Sent Done Done false
17640 5436 Acked Read 6 Sent Done Done false
17641 5437 Acked Read 8 Sent Done Done false
17642 5438 Acked Write 3 Sent Done Done false
17643 5439 Acked Write 9 Sent Done Done false
17644 5440 Acked Write 9 Sent Done Done false
17645 5441 Acked Write 6 Sent Done Done false
17646 5442 Acked Write 10 Sent Done Done false
17647 5443 Acked Flush 0 Sent Done Done false
17648 5444 Acked Read 2 Sent Done Done false
17649 5445 Acked Read 8 Sent Done Done false
17650 5446 Acked Flush 0 Sent Done Done false
17651 5447 Acked Write 3 Sent Done Done false
17652 5448 Acked Read 5 Sent Done Done false
17653 5449 Acked Read 7 Sent Done Done false
17654 5450 Acked Write 10 Sent Done Done false
17655 5451 Acked Flush 0 Sent Done Done false
17656 5452 Acked Write 7 Sent Done Done false
17657 5453 Acked Write 10 Sent Done Done false
17658 5454 Acked Read 4 Sent Done Done false
17659 5455 Acked Write 10 Sent Done Done false
17660 5456 Acked Read 3 Sent Done Done false
17661 5457 Acked Read 10 Sent Done Done false
17662 5458 Acked Read 3 Sent Done Done false
17663 5459 Acked Read 6 Sent Done Done false
17664 5460 Acked Write 10 Sent Done Done false
17665 5461 Acked Write 9 Sent Done Done false
17666 5462 Acked Read 6 Sent Done Done false
17667 5463 Acked Flush 0 Sent Done Done false
17668 5464 Acked Read 7 Sent Done Done false
17669 5465 Acked Read 8 Sent Done Done false
17670 5466 Acked Write 4 Sent Done Done false
17671 5467 Acked Flush 0 Sent Done Done false
17672 5468 Acked Write 10 Sent Done Done false
17673 5469 Acked Write 2 Sent Done Done false
17674 5470 Acked Write 5 Sent Done Done false
17675 5471 Acked Read 7 Sent Done Done false
17676 5472 Acked Read 7 Sent Done Done false
17677 5473 Acked Read 3 Sent Done Done false
17678 5474 Acked Write 2 Sent Done Done false
17679 5475 Acked Write 3 Sent Done Done false
17680 5476 Acked Read 4 Sent Done Done false
17681 5477 Acked Read 10 Sent Done Done false
17682 5478 Acked Read 4 Sent Done Done false
17683 5479 Acked Write 10 Sent Done Done false
17684 5480 Acked Write 4 Sent Done Done false
17685 5481 Acked Read 6 Sent Done Done false
17686 5482 Acked Write 5 Sent Done Done false
17687 5483 Acked Read 6 Sent Done Done false
17688 5484 Acked Read 7 Sent Done Done false
17689 5485 Acked Read 6 Sent Done Done false
17690 5486 Acked Read 4 Sent Done Done false
17691 5487 Acked Write 1 Sent Done Done false
17692 5488 Acked Read 8 Sent Done Done false
17693 5489 Acked Read 6 Sent Done Done false
17694 5490 Acked Read 9 Sent Done Done false
17695 5491 Acked Read 10 Sent Done Done false
17696 5492 Acked Read 4 Sent Done Done false
17697 5493 Acked Read 7 Sent Done Done false
17698 5494 Acked Read 8 Sent Done Done false
17699 5495 Acked Read 2 Sent Done Done false
17700 5496 Acked Write 6 Sent Done Done false
17701 5497 Acked Flush 0 Sent Done Done false
17702 5498 Acked Write 9 Sent Done Done false
17703 5499 Acked Write 8 Sent Done Done false
17704 5500 Acked Write 1 Sent Done Done false
17705 5501 Acked Write 4 Sent Done Done false
17706 5502 Acked Read 9 Sent Done Done false
17707 5503 Acked Write 3 Sent Done Done false
17708 5504 Acked Write 4 Sent Done Done false
17709 5505 Acked Read 6 Sent Done Done false
17710 5506 Acked Read 7 Sent Done Done false
17711 5507 Acked Read 8 Sent Done Done false
17712 5508 Acked Write 5 Sent Done Done false
17713 5509 Acked Read 2 Sent Done Done false
17714 5510 Acked Write 8 Sent Done Done false
17715 5511 Acked Read 8 Sent Done Done false
17716 5512 Acked Write 7 Sent Done Done false
17717 5513 Acked Write 1 Sent Done Done false
17718 5514 Acked Flush 0 Sent Done Done false
17719 5515 Acked Write 5 Sent Done Done false
17720 5516 Acked Read 6 Sent Done Done false
17721 5517 Acked Write 2 Sent Done Done false
17722 5518 Acked Read 8 Sent Done Done false
17723 5519 Acked Flush 0 Sent Done Done false
17724 5520 Acked Read 3 Sent Done Done false
17725 5521 Acked Write 2 Sent Done Done false
17726 5522 Acked Flush 0 Sent Done Done false
17727 5523 Acked Write 2 Sent Done Done false
17728 5524 Acked Write 2 Sent Done Done false
17729 5525 Acked Write 1 Sent Done Done false
17730 5526 Acked Write 3 Sent Done Done false
17731 5527 Acked Read 1 Sent Done Done false
17732 STATES DS:0 DS:1 DS:2 TOTAL
17733 Sent 301 0 0 301
17734 Done 0 301 301 602
17735 Skipped 0 0 0 0
17736 Error 0 0 0 0
17737 Last Flush: Some(JobId(5226)) Some(JobId(5522)) Some(JobId(5522))
17738 Downstairs last five completed: 5226 5225 5224 5223 5222
17739 Upstairs last five completed: 5527 5526 5525 5524 5523
17740 CLIENT: Up:0 ds:301 act:2
177412025-02-25T20:50:29.427ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177422025-02-25T20:50:30.197ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177432025-02-25T20:50:30.197ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44278) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177442025-02-25T20:50:30.199ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177452025-02-25T20:50:30.199ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177462025-02-25T20:50:30.199ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177472025-02-25T20:50:30.199ZINFOcrucible: send last flush ID to this DS: Some(JobId(5226)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177482025-02-25T20:50:30.199ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177492025-02-25T20:50:30.199ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5226)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177502025-02-25T20:50:30.200ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5226)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
17751 ----------------------------------------------------------------
17752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17753 Downstairs last five completed: 5528 5527 5526 5525 5524
17754 Upstairs last five completed: 5528 5527 5526 5525 5524
17755 CLIENT: Up:0 ds:0 act:3
17756 Replay: All jobs finished
17757 Main task has work to do, go find it
17758 got dsc Stop(2)
17759 stop 2
17760 Feb 25 20:50:32.454 INFO request completed, latency_us:[8830] Got stop action so:false kr:false
17761 488, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 66d7181d-7ebe-449c-a00c-21a38503f1d2, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
177622025-02-25T20:50:32.455ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177632025-02-25T20:50:32.455ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
177642025-02-25T20:50:32.455ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
17765 Feb 25 20:50:32.455 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/2, method: GET, req_id: 59b1f7dd-0f73-479d-af0d-7c4c462fe02c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
177662025-02-25T20:50:32.455ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
17767 [8830] Exited with: ExitStatus(unix_wait_status(9))
17768 [8830][2] reports Exit
17769 Feb 25 20:50:36.457 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0c46c875-040b-4bf6-934c-5d40dc6d52bb, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
17770 001/300 Read block 1121 len 24576
17771 002/300 Write block 3596 len 40960 data: 6 5 3 3 4 4 3 2 2 2
17772 003/300 Read block 4238 len 24576
17773 004/300 Read block 944 len 32768
17774 005/300 Read block 3928 len 12288
17775 006/300 Read block 2224 len 36864
17776 007/300 Read block 2816 len 36864
17777 008/300 Write block 4753 len 16384 data: 4 4 5 5
17778 009/300 Write block 4191 len 36864 data: 2 2 2 2 2 2 2 2 2
17779 010/300 Read block 2032 len 16384
17780 011/300 Read block 4296 len 4096
17781 012/300 Write block 2306 len 36864 data: 2 2 2 3 4 3 3 3 4
17782 013/300 Read block 4897 len 36864
17783 014/300 Read block 3097 len 16384
17784 015/300 Write block 3778 len 8192 data: 4 4
17785 016/300 Read block 4621 len 8192
17786 017/300 Read block 2881 len 32768
17787 018/300 Read block 2919 len 12288
17788 019/300 Read block 1982 len 8192
17789 020/300 Write block 568 len 8192 data: 2 2
17790 021/300 Read block 2978 len 20480
17791 022/300 Write block 1478 len 40960 data: 7 6 6 6 4 2 3 2 2 2
17792 023/300 Write block 115 len 28672 data: 2 2 2 3 3 3 3
17793 024/300 Write block 4154 len 4096 data: 3
17794 025/300 Read block 3557 len 24576
17795 026/300 Write block 2017 len 24576 data: 4 4 5 4 5 4
17796 027/300 Write block 2359 len 36864 data: 5 2 2 3 5 5 5 5 5
17797 028/300 Write block 3698 len 32768 data: 3 3 3 5 5 4 5 4
17798 029/300 Read block 2344 len 8192
17799 030/300 Write block 3759 len 4096 data: 6
17800 031/300 Flush
17801 032/300 Write block 2225 len 16384 data: 4 4 3 3
17802 033/300 Read block 725 len 20480
17803 034/300 Read block 1194 len 36864
17804 035/300 Write block 4763 len 36864 data: 3 2 2 2 2 3 3 3 4
17805 036/300 Write block 3217 len 36864 data: 3 3 3 3 3 2 2 2 2
17806 037/300 Read block 1033 len 24576
17807 038/300 Read block 2899 len 40960
17808 039/300 Write block 4115 len 40960 data: 3 3 3 4 4 5 5 5 6 5
17809 040/300 Read block 4788 len 40960
17810 041/300 Read block 4982 len 20480
17811 042/300 Write block 2620 len 32768 data: 6 6 6 7 6 8 8 6
17812 043/300 Read block 1274 len 28672
17813 044/300 Read block 661 len 4096
17814 045/300 Read block 1838 len 8192
17815 046/300 Write block 1256 len 32768 data: 4 5 5 4 6 6 6 6
17816 047/300 Read block 2437 len 4096
17817 048/300 Write block 1095 len 40960 data: 6 6 5 3 3 4 4 5 5 6
17818 049/300 Flush
17819 050/300 Read block 1745 len 40960
17820 051/300 Write block 272 len 24576 data: 5 5 5 4 4 4
17821 052/300 Write block 3733 len 32768 data: 3 3 3 3 3 3 3 3
17822 053/300 Read block 1415 len 12288
17823 054/300 Flush
17824 055/300 Read block 253 len 28672
17825 056/300 Read block 937 len 32768
17826 057/300 Write block 4517 len 16384 data: 3 4 4 4
17827 058/300 Read block 3795 len 24576
17828 059/300 Read block 3668 len 24576
17829 060/300 Read block 22 len 40960
17830 061/300 Read block 2233 len 40960
17831 062/300 Read block 3895 len 40960
17832 063/300 Flush
17833 064/300 Write block 3732 len 24576 data: 4 4 4 4 4 4
17834 065/300 Read block 2180 len 24576
17835 066/300 Write block 16 len 16384 data: 3 2 2 2
17836 067/300 Read block 2419 len 12288
17837 068/300 Write block 2208 len 12288 data: 6 5 4
17838 069/300 Read block 1705 len 36864
17839 070/300 Write block 4600 len 16384 data: 4 4 5 5
17840 071/300 Read block 4571 len 32768
17841 072/300 Read block 3705 len 24576
17842 073/300 Write block 403 len 20480 data: 2 2 2 4 4
17843 074/300 Read block 1393 len 40960
17844 075/300 Write block 3480 len 12288 data: 4 4 4
17845 076/300 Write block 3443 len 8192 data: 4 4
17846 077/300 Read block 2774 len 16384
17847 078/300 Read block 4131 len 12288
17848 079/300 Read block 4246 len 4096
17849 080/300 Read block 3261 len 20480
17850 081/300 Write block 3688 len 20480 data: 4 3 4 4 4
17851 082/300 Read block 1777 len 12288
17852 083/300 Read block 4110 len 36864
17853 084/300 Read block 3004 len 12288
17854 085/300 Write block 2963 len 12288 data: 5 4 4
17855 086/300 Read block 4608 len 36864
17856 087/300 Read block 1801 len 12288
17857 088/300 Write block 4554 len 32768 data: 2 2 2 2 3 3 2 2
17858 089/300 Read block 1850 len 28672
17859 090/300 Read block 4779 len 12288
17860 091/300 Write block 4550 len 8192 data: 3 2
17861 092/300 Read block 3046 len 28672
17862 093/300 Write block 4741 len 20480 data: 3 3 3 3 4
17863 094/300 Read block 3004 len 16384
17864 095/300 Read block 326 len 20480
17865 096/300 Read block 2574 len 16384
17866 097/300 Read block 2701 len 20480
17867 098/300 Read block 3780 len 28672
17868 099/300 Read block 4938 len 36864
17869 100/300 Write block 2083 len 24576 data: 4 4 5 5 5 5
17870 101/300 Read block 4097 len 4096
17871 102/300 Write block 3088 len 8192 data: 4 4
17872 103/300 Write block 1266 len 24576 data: 4 4 2 2 4 4
17873 104/300 Read block 4426 len 16384
17874 105/300 Write block 1388 len 28672 data: 5 5 5 5 5 6 4
17875 106/300 Read block 3311 len 32768
17876 107/300 Write block 4773 len 4096 data: 3
17877 108/300 Write block 4770 len 32768 data: 4 5 3 4 3 3 3 4
17878 109/300 Read block 902 len 4096
17879 110/300 Read block 3967 len 8192
17880 111/300 Write block 1369 len 8192 data: 5 4
17881 112/300 Read block 947 len 28672
17882 113/300 Read block 1642 len 24576
17883 114/300 Read block 2812 len 16384
17884 115/300 Read block 2104 len 36864
17885 116/300 Read block 4771 len 12288
17886 117/300 Flush
17887 118/300 Read block 936 len 32768
17888 119/300 Read block 884 len 4096
17889 120/300 Read block 467 len 4096
17890 121/300 Read block 4345 len 20480
17891 122/300 Read block 3554 len 40960
17892 123/300 Write block 1037 len 8192 data: 5 4
17893 124/300 Write block 512 len 20480 data: 4 3 4 4 4
17894 125/300 Read block 3789 len 32768
17895 126/300 Write block 3603 len 28672 data: 3 3 3 2 2 2 3
17896 127/300 Write block 190 len 36864 data: 5 5 4 4 5 4 5 4 4
17897 128/300 Write block 4147 len 36864 data: 2 2 4 4 4 4 3 4 3
17898 129/300 Read block 3189 len 8192
17899 130/300 Read block 1646 len 24576
17900 131/300 Read block 1578 len 12288
17901 132/300 Read block 2445 len 16384
17902 133/300 Read block 3844 len 4096
17903 134/300 Read block 2272 len 8192
17904 135/300 Read block 4962 len 40960
17905 136/300 Read block 4830 len 24576
17906 137/300 Read block 4356 len 16384
17907 138/300 Write block 2654 len 20480 data: 3 2 3 4 3
17908 139/300 Read block 4396 len 4096
17909 140/300 Write block 2513 len 28672 data: 4 5 4 4 5 6 6
17910 141/300 Write block 1527 len 40960 data: 4 5 6 6 7 7 6 6 6 5
17911 142/300 Read block 616 len 24576
17912 143/300 Write block 2737 len 4096 data: 4
17913 144/300 Read block 2042 len 20480
17914 145/300 Read block 797 len 32768
17915 146/300 Write block 2249 len 32768 data: 5 5 4 4 3 4 5 6
17916 147/300 Write block 1310 len 32768 data: 3 3 3 3 3 3 3 4
17917 148/300 Write block 4493 len 36864 data: 2 2 2 2 2 3 4 3 3
17918 149/300 Read block 1352 len 16384
17919 150/300 Write block 162 len 12288 data: 4 5 4
17920 151/300 Read block 893 len 12288
17921 152/300 Read block 331 len 12288
17922 153/300 Flush
17923 154/300 Flush
17924 155/300 Write block 1025 len 12288 data: 7 5 5
17925 156/300 Flush
17926 157/300 Write block 2606 len 40960 data: 5 5 5 4 4 4 3 2 2 4
17927 158/300 Read block 3304 len 12288
17928 159/300 Read block 233 len 20480
17929 160/300 Write block 4126 len 16384 data: 5 4 4 4
17930 161/300 Read block 215 len 28672
17931 162/300 Write block 4992 len 16384 data: 5 6 5 5
17932 163/300 Write block 2004 len 32768 data: 4 4 3 5 6 8 8 5
17933 164/300 Write block 2154 len 28672 data: 3 3 3 4 4 6 6
17934 165/300 Read block 1350 len 20480
17935 166/300 Read block 4888 len 12288
17936 167/300 Write block 1561 len 8192 data: 6 7
17937 168/300 Write block 2238 len 20480 data: 3 4 4 4 4
17938 169/300 Read block 3964 len 16384
17939 170/300 Write block 1335 len 20480 data: 4 4 4 4 4
17940 171/300 Read block 4400 len 36864
17941 172/300 Read block 1746 len 28672
17942 173/300 Write block 3832 len 12288 data: 5 5 5
17943 174/300 Read block 815 len 36864
17944 175/300 Flush
17945 176/300 Write block 1617 len 12288 data: 7 5 4
17946 177/300 Write block 1165 len 20480 data: 3 4 4 4 4
17947 178/300 Read block 82 len 4096
17948 179/300 Write block 3867 len 8192 data: 2 2
17949 180/300 Read block 976 len 32768
17950 181/300 Write block 437 len 8192 data: 6 5
17951 182/300 Write block 458 len 32768 data: 2 2 2 2 2 2 2 2
17952 183/300 Read block 2776 len 24576
17953 184/300 Flush
17954 185/300 Write block 4258 len 12288 data: 2 3 3
17955 186/300 Read block 1288 len 40960
17956 187/300 Read block 1393 len 4096
17957 188/300 Write block 3088 len 4096 data: 5
17958 189/300 Read block 4988 len 28672
17959 190/300 Read block 2965 len 4096
17960 191/300 Read block 1032 len 24576
17961 192/300 Flush
17962 193/300 Read block 1490 len 40960
17963 194/300 Write block 3295 len 40960 data: 5 4 4 5 7 7 7 8 9 8
17964 195/300 Read block 4235 len 20480
17965 196/300 Read block 4407 len 16384
17966 197/300 Read block 3873 len 4096
17967 198/300 Write block 4229 len 40960 data: 5 4 5 5 6 7 7 4 4 4
17968 199/300 Write block 2194 len 12288 data: 5 4 3
17969 200/300 Write block 831 len 16384 data: 2 2 2 3
17970 201/300 Write block 3777 len 4096 data: 4
17971 202/300 Write block 3433 len 12288 data: 4 5 5
17972 203/300 Read block 2942 len 8192
17973 204/300 Write block 1050 len 36864 data: 4 4 6 5 4 4 5 4 3
17974 205/300 Read block 4267 len 16384
17975 206/300 Read block 2842 len 16384
17976 207/300 Read block 1047 len 36864
17977 208/300 Write block 3904 len 16384 data: 5 5 4 4
17978 209/300 Write block 4145 len 8192 data: 4 3
17979 210/300 Read block 1753 len 20480
17980 211/300 Write block 590 len 16384 data: 4 4 3 4
17981 212/300 Write block 1784 len 32768 data: 5 5 5 4 4 5 6 5
17982 213/300 Read block 4255 len 24576
17983 214/300 Read block 2469 len 32768
17984 215/300 Read block 3029 len 36864
17985 216/300 Write block 3885 len 4096 data: 7
17986 217/300 Read block 1486 len 40960
17987 218/300 Write block 4838 len 8192 data: 4 5
17988 219/300 Read block 3676 len 32768
17989 220/300 Read block 4728 len 32768
17990 221/300 Write block 1027 len 16384 data: 6 4 3 3
17991 222/300 Flush
17992 223/300 Write block 3115 len 28672 data: 5 6 4 4 4 4 5
17993 224/300 Write block 3404 len 28672 data: 2 2 2 2 2 3 3
17994 225/300 Write block 227 len 12288 data: 6 6 4
17995 226/300 Write block 2252 len 32768 data: 5 4 5 6 7 5 6 7
17996 227/300 Write block 4116 len 32768 data: 4 4 5 5 6 6 6 7
17997 228/300 Read block 703 len 8192
17998 229/300 Flush
17999 230/300 Write block 1821 len 36864 data: 3 3 4 4 5 6 6 6 5
18000 231/300 Read block 4587 len 40960
18001 232/300 Read block 3168 len 40960
18002 233/300 Read block 919 len 36864
18003 234/300 Read block 298 len 12288
18004 235/300 Read block 2098 len 4096
18005 236/300 Write block 3570 len 32768 data: 4 5 5 5 6 4 3 3
18006 237/300 Read block 1627 len 16384
18007 238/300 Read block 1762 len 32768
18008 239/300 Write block 2707 len 12288 data: 6 7 5
18009 240/300 Read block 415 len 24576
18010 241/300 Read block 4003 len 32768
18011 242/300 Write block 3399 len 28672 data: 2 2 2 2 2 3 3
18012 243/300 Write block 1886 len 40960 data: 4 4 4 5 5 4 4 5 5 5
18013 244/300 Read block 3724 len 4096
18014 245/300 Read block 3815 len 28672
18015 246/300 Read block 4782 len 24576
18016 247/300 Read block 1041 len 8192
18017 248/300 Write block 4918 len 20480 data: 4 4 6 6 5
18018 249/300 Read block 672 len 32768
18019 250/300 Write block 616 len 20480 data: 4 5 5 5 5
18020 251/300 Write block 3554 len 40960 data: 2 4 5 5 4 4 4 5 4 4
18021 252/300 Write block 3722 len 12288 data: 7 7 5
18022 253/300 Read block 4097 len 16384
18023 254/300 Flush
18024 255/300 Write block 80 len 4096 data: 5
18025 256/300 Write block 1590 len 16384 data: 3 2 2 2
18026 257/300 Flush
18027 258/300 Write block 2981 len 36864 data: 3 5 5 5 5 7 8 10 7
18028 259/300 Read block 1999 len 24576
18029 260/300 Write block 4542 len 24576 data: 2 2 2 2 3 3
18030 261/300 Write block 1576 len 4096 data: 2
18031 262/300 Write block 745 len 12288 data: 2 2 2
18032 263/300 Write block 2341 len 40960 data: 8 8 8 4 4 3 3 5 4 5
18033 264/300 Write block 4841 len 40960 data: 3 3 3 3 3 4 5 5 4 3
18034 265/300 Read block 2336 len 36864
18035 266/300 Write block 3259 len 4096 data: 2
18036 267/300 Read block 1190 len 28672
18037 268/300 Read block 3491 len 32768
18038 269/300 Read block 593 len 16384
18039 270/300 Write block 601 len 28672 data: 3 3 3 3 4 3 3
18040 271/300 Write block 3129 len 28672 data: 4 4 4 4 4 5 5
18041 272/300 Read block 796 len 40960
18042 273/300 Read block 553 len 20480
18043 274/300 Write block 4128 len 16384 data: 5 5 4 4
18044 275/300 Write block 52 len 20480 data: 4 3 3 4 4
18045 276/300 Write block 4954 len 12288 data: 4 3 3
18046 277/300 Write block 2545 len 20480 data: 5 5 4 4 5
18047 278/300 Flush
18048 279/300 Write block 1944 len 32768 data: 3 3 4 5 6 5 5 5
18049 280/300 Read block 3880 len 28672
18050 281/300 Write block 334 len 16384 data: 6 6 6 4
18051 282/300 Read block 3129 len 28672
18052 283/300 Write block 1595 len 36864 data: 3 3 3 5 5 6 4 4 5
18053 284/300 Read block 2690 len 36864
18054 285/300 Read block 4062 len 12288
18055 286/300 Flush
18056 287/300 Write block 1868 len 12288 data: 3 3 4
18057 288/300 Read block 509 len 8192
18058 289/300 Read block 2918 len 16384
18059 290/300 Flush
18060 291/300 Read block 2954 len 36864
18061 292/300 Write block 2831 len 20480 data: 3 4 4 4 4
18062 293/300 Write block 4883 len 32768 data: 6 6 6 5 6 5 6 6
18063 294/300 Write block 3953 len 12288 data: 4 3 4
18064 295/300 Read block 3565 len 12288
18065 296/300 Write block 4740 len 24576 data: 4 4 4 4 4 5
18066 297/300 Read block 1821 len 24576
18067 298/300 Read block 1809 len 36864
18068 299/300 Read block 2255 len 40960
18069 300/300 Read block 1032 len 24576
18070 301/300 Write block 421 len 40960 data: 4 6 6 6 7 7 6 4 3 5
18071 Main task has work to do, go find it
18072 got dsc Start(2)
18073 start 2
18074 Feb 25 20:50:36.711 INFO request completed, latency_us: 267, response_code: 204, uri: /start/cid/2, method: POST, req_id: a5833e28-a4b8-4561-916a-1c8e7b3ce411, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
18075 Replay: started 2, returned:Ok(())
18076 ----------------------------------------------------------------
18077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18078 States: Active Active Offline
18079 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18080 5529 Acked Read 6 Done Done Sent false
18081 5530 Acked Write 10 Done Done Sent false
18082 5531 Acked Read 6 Done Done Sent false
18083 5532 Acked Read 8 Done Done Sent false
18084 5533 Acked Read 3 Done Done Sent false
18085 5534 Acked Read 9 Done Done Sent false
18086 5535 Acked Read 9 Done Done Sent false
18087 5536 Acked Write 4 Done Done Sent false
18088 5537 Acked Write 9 Done Done Sent false
18089 5538 Acked Read 4 Done Done Sent false
18090 5539 Acked Read 1 Done Done Sent false
18091 5540 Acked Write 9 Done Done Sent false
18092 5541 Acked Read 9 Done Done Sent false
18093 5542 Acked Read 4 Done Done Sent false
18094 5543 Acked Write 2 Done Done Sent false
18095 5544 Acked Read 2 Done Done Sent false
18096 5545 Acked Read 8 Done Done Sent false
18097 5546 Acked Read 3 Done Done Sent false
18098 5547 Acked Read 2 Done Done Sent false
18099 5548 Acked Write 2 Done Done Sent false
18100 5549 Acked Read 5 Done Done Sent false
18101 5550 Acked Write 10 Done Done Sent false
18102 5551 Acked Write 7 Done Done Sent false
18103 5552 Acked Write 1 Done Done Sent false
18104 5553 Acked Read 6 Done Done Sent false
18105 5554 Acked Write 6 Done Done Sent false
18106 5555 Acked Write 9 Done Done Sent false
18107 5556 Acked Write 8 Done Done Sent false
18108 5557 Acked Read 2 Done Done Sent false
18109 5558 Acked Write 1 Done Done Sent false
18110 5559 Acked Flush 0 Done Done Sent false
18111 5560 Acked Write 4 Done Done Sent false
18112 5561 Acked Read 5 Done Done Sent false
18113 5562 Acked Read 9 Done Done Sent false
18114 5563 Acked Write 9 Done Done Sent false
18115 5564 Acked Write 9 Done Done Sent false
18116 5565 Acked Read 6 Done Done Sent false
18117 5566 Acked Read 10 Done Done Sent false
18118 5567 Acked Write 10 Done Done Sent false
18119 5568 Acked Read 10 Done Done Sent false
18120 5569 Acked Read 5 Done Done Sent false
18121 5570 Acked Write 8 Done Done Sent false
18122 5571 Acked Read 7 Done Done Sent false
18123 5572 Acked Read 1 Done Done Sent false
18124 5573 Acked Read 2 Done Done Sent false
18125 5574 Acked Write 8 Done Done Sent false
18126 5575 Acked Read 1 Done Done Sent false
18127 5576 Acked Write 10 Done Done Sent false
18128 5577 Acked Flush 0 Done Done Sent false
18129 5578 Acked Read 10 Done Done Sent false
18130 5579 Acked Write 6 Done Done Sent false
18131 5580 Acked Write 8 Done Done Sent false
18132 5581 Acked Read 3 Done Done Sent false
18133 5582 Acked Flush 0 Done Done Sent false
18134 5583 Acked Read 7 Done Done Sent false
18135 5584 Acked Read 8 Done Done Sent false
18136 5585 Acked Write 4 Done Done Sent false
18137 5586 Acked Read 6 Done Done Sent false
18138 5587 Acked Read 6 Done Done Sent false
18139 5588 Acked Read 10 Done Done Sent false
18140 5589 Acked Read 10 Done Done Sent false
18141 5590 Acked Read 10 Done Done Sent false
18142 5591 Acked Flush 0 Done Done Sent false
18143 5592 Acked Write 6 Done Done Sent false
18144 5593 Acked Read 6 Done Done Sent false
18145 5594 Acked Write 4 Done Done Sent false
18146 5595 Acked Read 3 Done Done Sent false
18147 5596 Acked Write 3 Done Done Sent false
18148 5597 Acked Read 9 Done Done Sent false
18149 5598 Acked Write 4 Done Done Sent false
18150 5599 Acked Read 8 Done Done Sent false
18151 5600 Acked Read 6 Done Done Sent false
18152 5601 Acked Write 5 Done Done Sent false
18153 5602 Acked Read 10 Done Done Sent false
18154 5603 Acked Write 3 Done Done Sent false
18155 5604 Acked Write 2 Done Done Sent false
18156 5605 Acked Read 4 Done Done Sent false
18157 5606 Acked Read 3 Done Done Sent false
18158 5607 Acked Read 1 Done Done Sent false
18159 5608 Acked Read 5 Done Done Sent false
18160 5609 Acked Write 5 Done Done Sent false
18161 5610 Acked Read 3 Done Done Sent false
18162 5611 Acked Read 9 Done Done Sent false
18163 5612 Acked Read 3 Done Done Sent false
18164 5613 Acked Write 3 Done Done Sent false
18165 5614 Acked Read 9 Done Done Sent false
18166 5615 Acked Read 3 Done Done Sent false
18167 5616 Acked Write 8 Done Done Sent false
18168 5617 Acked Read 7 Done Done Sent false
18169 5618 Acked Read 3 Done Done Sent false
18170 5619 Acked Write 2 Done Done Sent false
18171 5620 Acked Read 7 Done Done Sent false
18172 5621 Acked Write 5 Done Done Sent false
18173 5622 Acked Read 4 Done Done Sent false
18174 5623 Acked Read 5 Done Done Sent false
18175 5624 Acked Read 4 Done Done Sent false
18176 5625 Acked Read 5 Done Done Sent false
18177 5626 Acked Read 7 Done Done Sent false
18178 5627 Acked Read 9 Done Done Sent false
18179 5628 Acked Write 6 Done Done Sent false
18180 5629 Acked Read 1 Done Done Sent false
18181 5630 Acked Write 2 Done Done Sent false
18182 5631 Acked Write 6 Done Done Sent false
18183 5632 Acked Read 4 Done Done Sent false
18184 5633 Acked Write 7 Done Done Sent false
18185 5634 Acked Read 8 Done Done Sent false
18186 5635 Acked Write 1 Done Done Sent false
18187 5636 Acked Write 8 Done Done Sent false
18188 5637 Acked Read 1 Done Done Sent false
18189 5638 Acked Read 2 Done Done Sent false
18190 5639 Acked Write 2 Done Done Sent false
18191 5640 Acked Read 7 Done Done Sent false
18192 5641 Acked Read 6 Done Done Sent false
18193 5642 Acked Read 4 Done Done Sent false
18194 5643 Acked Read 9 Done Done Sent false
18195 5644 Acked Read 3 Done Done Sent false
18196 5645 Acked Flush 0 Done Done Sent false
18197 5646 Acked Read 8 Done Done Sent false
18198 5647 Acked Read 1 Done Done Sent false
18199 5648 Acked Read 1 Done Done Sent false
18200 5649 Acked Read 5 Done Done Sent false
18201 5650 Acked Read 10 Done Done Sent false
18202 5651 Acked Write 2 Done Done Sent false
18203 5652 Acked Write 5 Done Done Sent false
18204 5653 Acked Read 8 Done Done Sent false
18205 5654 Acked Write 7 Done Done Sent false
18206 5655 Acked Write 9 Done Done Sent false
18207 5656 Acked Write 9 Done Done Sent false
18208 5657 Acked Read 2 Done Done Sent false
18209 5658 Acked Read 6 Done Done Sent false
18210 5659 Acked Read 3 Done Done Sent false
18211 5660 Acked Read 4 Done Done Sent false
18212 5661 Acked Read 1 Done Done Sent false
18213 5662 Acked Read 2 Done Done Sent false
18214 5663 Acked Read 10 Done Done Sent false
18215 5664 Acked Read 6 Done Done Sent false
18216 5665 Acked Read 4 Done Done Sent false
18217 5666 Acked Write 5 Done Done Sent false
18218 5667 Acked Read 1 Done Done Sent false
18219 5668 Acked Write 7 Done Done Sent false
18220 5669 Acked Write 10 Done Done Sent false
18221 5670 Acked Read 6 Done Done Sent false
18222 5671 Acked Write 1 Done Done Sent false
18223 5672 Acked Read 5 Done Done Sent false
18224 5673 Acked Read 8 Done Done Sent false
18225 5674 Acked Write 8 Done Done Sent false
18226 5675 Acked Write 8 Done Done Sent false
18227 5676 Acked Write 9 Done Done Sent false
18228 5677 Acked Read 4 Done Done Sent false
18229 5678 Acked Write 3 Done Done Sent false
18230 5679 Acked Read 3 Done Done Sent false
18231 5680 Acked Read 3 Done Done Sent false
18232 5681 Acked Flush 0 Done Done Sent false
18233 5682 Acked Flush 0 Done Done Sent false
18234 5683 Acked Write 3 Done Done Sent false
18235 5684 Acked Flush 0 Done Done Sent false
18236 5685 Acked Write 10 Done Done Sent false
18237 5686 Acked Read 3 Done Done Sent false
18238 5687 Acked Read 5 Done Done Sent false
18239 5688 Acked Write 4 Done Done Sent false
18240 5689 Acked Read 7 Done Done Sent false
18241 5690 Acked Write 4 Done Done Sent false
18242 5691 Acked Write 8 Done Done Sent false
18243 5692 Acked Write 7 Done Done Sent false
18244 5693 Acked Read 5 Done Done Sent false
18245 5694 Acked Read 3 Done Done Sent false
18246 5695 Acked Write 2 Done Done Sent false
18247 5696 Acked Write 5 Done Done Sent false
18248 5697 Acked Read 4 Done Done Sent false
18249 5698 Acked Write 5 Done Done Sent false
18250 5699 Acked Read 9 Done Done Sent false
18251 5700 Acked Read 7 Done Done Sent false
18252 5701 Acked Write 3 Done Done Sent false
18253 5702 Acked Read 9 Done Done Sent false
18254 5703 Acked Flush 0 Done Done Sent false
18255 5704 Acked Write 3 Done Done Sent false
18256 5705 Acked Write 5 Done Done Sent false
18257 5706 Acked Read 1 Done Done Sent false
18258 5707 Acked Write 2 Done Done Sent false
18259 5708 Acked Read 8 Done Done Sent false
18260 5709 Acked Write 2 Done Done Sent false
18261 5710 Acked Write 8 Done Done Sent false
18262 5711 Acked Read 6 Done Done Sent false
18263 5712 Acked Flush 0 Done Done Sent false
18264 5713 Acked Write 3 Done Done Sent false
18265 5714 Acked Read 10 Done Done Sent false
18266 5715 Acked Read 1 Done Done Sent false
18267 5716 Acked Write 1 Done Done Sent false
18268 5717 Acked Read 7 Done Done Sent false
18269 5718 Acked Read 1 Done Done Sent false
18270 5719 Acked Read 6 Done Done Sent false
18271 5720 Acked Flush 0 Done Done Sent false
18272 5721 Acked Read 10 Done Done Sent false
18273 5722 Acked Write 10 Done Done Sent false
18274 5723 Acked Read 5 Done Done Sent false
18275 5724 Acked Read 4 Done Done Sent false
18276 5725 Acked Read 1 Done Done Sent false
18277 5726 Acked Write 10 Done Done Sent false
18278 5727 Acked Write 3 Done Done Sent false
18279 5728 Acked Write 4 Done Done Sent false
18280 5729 Acked Write 1 Done Done Sent false
18281 5730 Acked Write 3 Done Done Sent false
18282 5731 Acked Read 2 Done Done Sent false
18283 5732 Acked Write 9 Done Done Sent false
18284 5733 Acked Read 4 Done Done Sent false
18285 5734 Acked Read 4 Done Done Sent false
18286 5735 Acked Read 9 Done Done Sent false
18287 5736 Acked Write 4 Done Done Sent false
18288 5737 Acked Write 2 Done Done Sent false
18289 5738 Acked Read 5 Done Done Sent false
18290 5739 Acked Write 4 Done Done Sent false
18291 5740 Acked Write 8 Done Done Sent false
18292 5741 Acked Read 6 Done Done Sent false
18293 5742 Acked Read 8 Done Done Sent false
18294 5743 Acked Read 9 Done Done Sent false
18295 5744 Acked Write 1 Done Done Sent false
18296 5745 Acked Read 10 Done Done Sent false
18297 5746 Acked Write 2 Done Done Sent false
18298 5747 Acked Read 8 Done Done Sent false
18299 5748 Acked Read 8 Done Done Sent false
18300 5749 Acked Write 4 Done Done Sent false
18301 5750 Acked Flush 0 Done Done Sent false
18302 5751 Acked Write 7 Done Done Sent false
18303 5752 Acked Write 7 Done Done Sent false
18304 5753 Acked Write 3 Done Done Sent false
18305 5754 Acked Write 8 Done Done Sent false
18306 5755 Acked Write 8 Done Done Sent false
18307 5756 Acked Read 2 Done Done Sent false
18308 5757 Acked Flush 0 Done Done Sent false
18309 5758 Acked Write 9 Done Done Sent false
18310 5759 Acked Read 10 Done Done Sent false
18311 5760 Acked Read 10 Done Done Sent false
18312 5761 Acked Read 9 Done Done Sent false
18313 5762 Acked Read 3 Done Done Sent false
18314 5763 Acked Read 1 Done Done Sent false
18315 5764 Acked Write 8 Done Done Sent false
18316 5765 Acked Read 4 Done Done Sent false
18317 5766 Acked Read 8 Done Done Sent false
18318 5767 Acked Write 3 Done Done Sent false
18319 5768 Acked Read 6 Done Done Sent false
18320 5769 Acked Read 8 Done Done Sent false
18321 5770 Acked Write 7 Done Done Sent false
18322 5771 Acked Write 10 Done Done Sent false
18323 5772 Acked Read 1 Done Done Sent false
18324 5773 Acked Read 7 Done Done Sent false
18325 5774 Acked Read 6 Done Done Sent false
18326 5775 Acked Read 2 Done Done Sent false
18327 5776 Acked Write 5 Done Done Sent false
18328 5777 Acked Read 8 Done Done Sent false
18329 5778 Acked Write 5 Done Done Sent false
18330 5779 Acked Write 10 Done Done Sent false
18331 5780 Acked Write 3 Done Done Sent false
18332 5781 Acked Read 4 Done Done Sent false
18333 5782 Acked Flush 0 Done Done Sent false
18334 5783 Acked Write 1 Done Done Sent false
18335 5784 Acked Write 4 Done Done Sent false
18336 5785 Acked Flush 0 Done Done Sent false
18337 5786 Acked Write 9 Done Done Sent false
18338 5787 Acked Read 6 Done Done Sent false
18339 5788 Acked Write 6 Done Done Sent false
18340 5789 Acked Write 1 Done Done Sent false
18341 5790 Acked Write 3 Done Done Sent false
18342 5791 Acked Write 10 Done Done Sent false
18343 5792 Acked Write 10 Done Done Sent false
18344 5793 Acked Read 9 Done Done Sent false
18345 5794 Acked Write 1 Done Done Sent false
18346 5795 Acked Read 7 Done Done Sent false
18347 5796 Acked Read 8 Done Done Sent false
18348 5797 Acked Read 4 Done Done Sent false
18349 5798 Acked Write 7 Done Done Sent false
18350 5799 Acked Write 7 Done Done Sent false
18351 5800 Acked Read 10 Done Done Sent false
18352 5801 Acked Read 5 Done Done Sent false
18353 5802 Acked Write 4 Done Done Sent false
18354 5803 Acked Write 5 Done Done Sent false
18355 5804 Acked Write 3 Done Done Sent false
18356 5805 Acked Write 5 Done Done Sent false
18357 5806 Acked Flush 0 Done Done Sent false
18358 5807 Acked Write 8 Done Done Sent false
18359 5808 Acked Read 7 Done Done Sent false
18360 5809 Acked Write 4 Done Done Sent false
18361 5810 Acked Read 7 Done Done Sent false
18362 5811 Acked Write 9 Done Done Sent false
18363 5812 Acked Read 9 Done Done Sent false
18364 5813 Acked Read 3 Done Done Sent false
18365 5814 Acked Flush 0 Done Done Sent false
18366 5815 Acked Write 3 Done Done Sent false
18367 5816 Acked Read 2 Done Done Sent false
18368 5817 Acked Read 4 Done Done Sent false
18369 5818 Acked Flush 0 Done Done Sent false
18370 5819 Acked Read 9 Done Done Sent false
18371 5820 Acked Write 5 Done Done Sent false
18372 5821 Acked Write 8 Done Done Sent false
18373 5822 Acked Write 3 Done Done Sent false
18374 5823 Acked Read 3 Done Done Sent false
18375 5824 Acked Write 6 Done Done Sent false
18376 5825 Acked Read 6 Done Done Sent false
18377 5826 Acked Read 9 Done Done Sent false
18378 5827 Acked Read 10 Done Done Sent false
18379 5828 Acked Read 6 Done Done Sent false
18380 5829 Acked Write 10 Done Done Sent false
18381 STATES DS:0 DS:1 DS:2 TOTAL
18382 Sent 0 0 301 301
18383 Done 301 301 0 602
18384 Skipped 0 0 0 0
18385 Error 0 0 0 0
18386 Last Flush: Some(JobId(5818)) Some(JobId(5818)) Some(JobId(5528))
18387 Downstairs last five completed: 5528 5527 5526 5525 5524
18388 Upstairs last five completed: 5829 5828 5827 5826 5825
18389 CLIENT: Up:0 ds:301 act:2
183902025-02-25T20:50:37.441ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
183912025-02-25T20:50:38.443ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
183922025-02-25T20:50:39.445ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
18393 [8830] got start action
18394 [8830] I am going to restart
18395 Starting downstairs at port 8830
18396 Make output file at "/tmp/dsc/downstairs-8830.txt"
18397 [8830][2] reports Starting
18398 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
18399 [8830][2] reports Running
18400 ----------------------------------------------------------------
18401 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18402 States: Active Active Offline
18403 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18404 5529 Acked Read 6 Done Done Sent false
18405 5530 Acked Write 10 Done Done Sent false
18406 5531 Acked Read 6 Done Done Sent false
18407 5532 Acked Read 8 Done Done Sent false
18408 5533 Acked Read 3 Done Done Sent false
18409 5534 Acked Read 9 Done Done Sent false
18410 5535 Acked Read 9 Done Done Sent false
18411 5536 Acked Write 4 Done Done Sent false
18412 5537 Acked Write 9 Done Done Sent false
18413 5538 Acked Read 4 Done Done Sent false
18414 5539 Acked Read 1 Done Done Sent false
18415 5540 Acked Write 9 Done Done Sent false
18416 5541 Acked Read 9 Done Done Sent false
18417 5542 Acked Read 4 Done Done Sent false
18418 5543 Acked Write 2 Done Done Sent false
18419 5544 Acked Read 2 Done Done Sent false
18420 5545 Acked Read 8 Done Done Sent false
18421 5546 Acked Read 3 Done Done Sent false
18422 5547 Acked Read 2 Done Done Sent false
18423 5548 Acked Write 2 Done Done Sent false
18424 5549 Acked Read 5 Done Done Sent false
18425 5550 Acked Write 10 Done Done Sent false
18426 5551 Acked Write 7 Done Done Sent false
18427 5552 Acked Write 1 Done Done Sent false
18428 5553 Acked Read 6 Done Done Sent false
18429 5554 Acked Write 6 Done Done Sent false
18430 5555 Acked Write 9 Done Done Sent false
18431 5556 Acked Write 8 Done Done Sent false
18432 5557 Acked Read 2 Done Done Sent false
18433 5558 Acked Write 1 Done Done Sent false
18434 5559 Acked Flush 0 Done Done Sent false
18435 5560 Acked Write 4 Done Done Sent false
18436 5561 Acked Read 5 Done Done Sent false
18437 5562 Acked Read 9 Done Done Sent false
18438 5563 Acked Write 9 Done Done Sent false
18439 5564 Acked Write 9 Done Done Sent false
18440 5565 Acked Read 6 Done Done Sent false
18441 5566 Acked Read 10 Done Done Sent false
18442 5567 Acked Write 10 Done Done Sent false
18443 5568 Acked Read 10 Done Done Sent false
18444 5569 Acked Read 5 Done Done Sent false
18445 5570 Acked Write 8 Done Done Sent false
18446 5571 Acked Read 7 Done Done Sent false
18447 5572 Acked Read 1 Done Done Sent false
18448 5573 Acked Read 2 Done Done Sent false
18449 5574 Acked Write 8 Done Done Sent false
18450 5575 Acked Read 1 Done Done Sent false
18451 5576 Acked Write 10 Done Done Sent false
18452 5577 Acked Flush 0 Done Done Sent false
18453 5578 Acked Read 10 Done Done Sent false
18454 5579 Acked Write 6 Done Done Sent false
18455 5580 Acked Write 8 Done Done Sent false
18456 5581 Acked Read 3 Done Done Sent false
18457 5582 Acked Flush 0 Done Done Sent false
18458 5583 Acked Read 7 Done Done Sent false
18459 5584 Acked Read 8 Done Done Sent false
18460 5585 Acked Write 4 Done Done Sent false
18461 5586 Acked Read 6 Done Done Sent false
18462 5587 Acked Read 6 Done Done Sent false
18463 5588 Acked Read 10 Done Done Sent false
18464 5589 Acked Read 10 Done Done Sent false
18465 5590 Acked Read 10 Done Done Sent false
18466 5591 Acked Flush 0 Done Done Sent false
18467 5592 Acked Write 6 Done Done Sent false
18468 5593 Acked Read 6 Done Done Sent false
18469 5594 Acked Write 4 Done Done Sent false
18470 5595 Acked Read 3 Done Done Sent false
18471 5596 Acked Write 3 Done Done Sent false
18472 5597 Acked Read 9 Done Done Sent false
18473 5598 Acked Write 4 Done Done Sent false
18474 5599 Acked Read 8 Done Done Sent false
18475 5600 Acked Read 6 Done Done Sent false
18476 5601 Acked Write 5 Done Done Sent false
18477 5602 Acked Read 10 Done Done Sent false
18478 5603 Acked Write 3 Done Done Sent false
18479 5604 Acked Write 2 Done Done Sent false
18480 5605 Acked Read 4 Done Done Sent false
18481 5606 Acked Read 3 Done Done Sent false
18482 5607 Acked Read 1 Done Done Sent false
18483 5608 Acked Read 5 Done Done Sent false
18484 5609 Acked Write 5 Done Done Sent false
18485 5610 Acked Read 3 Done Done Sent false
18486 5611 Acked Read 9 Done Done Sent false
18487 5612 Acked Read 3 Done Done Sent false
18488 5613 Acked Write 3 Done Done Sent false
18489 5614 Acked Read 9 Done Done Sent false
18490 5615 Acked Read 3 Done Done Sent false
18491 5616 Acked Write 8 Done Done Sent false
18492 5617 Acked Read 7 Done Done Sent false
18493 5618 Acked Read 3 Done Done Sent false
18494 5619 Acked Write 2 Done Done Sent false
18495 5620 Acked Read 7 Done Done Sent false
18496 5621 Acked Write 5 Done Done Sent false
18497 5622 Acked Read 4 Done Done Sent false
18498 5623 Acked Read 5 Done Done Sent false
18499 5624 Acked Read 4 Done Done Sent false
18500 5625 Acked Read 5 Done Done Sent false
18501 5626 Acked Read 7 Done Done Sent false
18502 5627 Acked Read 9 Done Done Sent false
18503 5628 Acked Write 6 Done Done Sent false
18504 5629 Acked Read 1 Done Done Sent false
18505 5630 Acked Write 2 Done Done Sent false
18506 5631 Acked Write 6 Done Done Sent false
18507 5632 Acked Read 4 Done Done Sent false
18508 5633 Acked Write 7 Done Done Sent false
18509 5634 Acked Read 8 Done Done Sent false
18510 5635 Acked Write 1 Done Done Sent false
18511 5636 Acked Write 8 Done Done Sent false
18512 5637 Acked Read 1 Done Done Sent false
18513 5638 Acked Read 2 Done Done Sent false
18514 5639 Acked Write 2 Done Done Sent false
18515 5640 Acked Read 7 Done Done Sent false
18516 5641 Acked Read 6 Done Done Sent false
18517 5642 Acked Read 4 Done Done Sent false
18518 5643 Acked Read 9 Done Done Sent false
18519 5644 Acked Read 3 Done Done Sent false
18520 5645 Acked Flush 0 Done Done Sent false
18521 5646 Acked Read 8 Done Done Sent false
18522 5647 Acked Read 1 Done Done Sent false
18523 5648 Acked Read 1 Done Done Sent false
18524 5649 Acked Read 5 Done Done Sent false
18525 5650 Acked Read 10 Done Done Sent false
18526 5651 Acked Write 2 Done Done Sent false
18527 5652 Acked Write 5 Done Done Sent false
18528 5653 Acked Read 8 Done Done Sent false
18529 5654 Acked Write 7 Done Done Sent false
18530 5655 Acked Write 9 Done Done Sent false
18531 5656 Acked Write 9 Done Done Sent false
18532 5657 Acked Read 2 Done Done Sent false
18533 5658 Acked Read 6 Done Done Sent false
18534 5659 Acked Read 3 Done Done Sent false
18535 5660 Acked Read 4 Done Done Sent false
18536 5661 Acked Read 1 Done Done Sent false
18537 5662 Acked Read 2 Done Done Sent false
18538 5663 Acked Read 10 Done Done Sent false
18539 5664 Acked Read 6 Done Done Sent false
18540 5665 Acked Read 4 Done Done Sent false
18541 5666 Acked Write 5 Done Done Sent false
18542 5667 Acked Read 1 Done Done Sent false
18543 5668 Acked Write 7 Done Done Sent false
18544 5669 Acked Write 10 Done Done Sent false
18545 5670 Acked Read 6 Done Done Sent false
18546 5671 Acked Write 1 Done Done Sent false
18547 5672 Acked Read 5 Done Done Sent false
18548 5673 Acked Read 8 Done Done Sent false
18549 5674 Acked Write 8 Done Done Sent false
18550 5675 Acked Write 8 Done Done Sent false
18551 5676 Acked Write 9 Done Done Sent false
18552 5677 Acked Read 4 Done Done Sent false
18553 5678 Acked Write 3 Done Done Sent false
18554 5679 Acked Read 3 Done Done Sent false
18555 5680 Acked Read 3 Done Done Sent false
18556 5681 Acked Flush 0 Done Done Sent false
18557 5682 Acked Flush 0 Done Done Sent false
18558 5683 Acked Write 3 Done Done Sent false
18559 5684 Acked Flush 0 Done Done Sent false
18560 5685 Acked Write 10 Done Done Sent false
18561 5686 Acked Read 3 Done Done Sent false
18562 5687 Acked Read 5 Done Done Sent false
18563 5688 Acked Write 4 Done Done Sent false
18564 5689 Acked Read 7 Done Done Sent false
18565 5690 Acked Write 4 Done Done Sent false
18566 5691 Acked Write 8 Done Done Sent false
18567 5692 Acked Write 7 Done Done Sent false
18568 5693 Acked Read 5 Done Done Sent false
18569 5694 Acked Read 3 Done Done Sent false
18570 5695 Acked Write 2 Done Done Sent false
18571 5696 Acked Write 5 Done Done Sent false
18572 5697 Acked Read 4 Done Done Sent false
18573 5698 Acked Write 5 Done Done Sent false
18574 5699 Acked Read 9 Done Done Sent false
18575 5700 Acked Read 7 Done Done Sent false
18576 5701 Acked Write 3 Done Done Sent false
18577 5702 Acked Read 9 Done Done Sent false
18578 5703 Acked Flush 0 Done Done Sent false
18579 5704 Acked Write 3 Done Done Sent false
18580 5705 Acked Write 5 Done Done Sent false
18581 5706 Acked Read 1 Done Done Sent false
18582 5707 Acked Write 2 Done Done Sent false
18583 5708 Acked Read 8 Done Done Sent false
18584 5709 Acked Write 2 Done Done Sent false
18585 5710 Acked Write 8 Done Done Sent false
18586 5711 Acked Read 6 Done Done Sent false
18587 5712 Acked Flush 0 Done Done Sent false
18588 5713 Acked Write 3 Done Done Sent false
18589 5714 Acked Read 10 Done Done Sent false
18590 5715 Acked Read 1 Done Done Sent false
18591 5716 Acked Write 1 Done Done Sent false
18592 5717 Acked Read 7 Done Done Sent false
18593 5718 Acked Read 1 Done Done Sent false
18594 5719 Acked Read 6 Done Done Sent false
18595 5720 Acked Flush 0 Done Done Sent false
18596 5721 Acked Read 10 Done Done Sent false
18597 5722 Acked Write 10 Done Done Sent false
18598 5723 Acked Read 5 Done Done Sent false
18599 5724 Acked Read 4 Done Done Sent false
18600 5725 Acked Read 1 Done Done Sent false
18601 5726 Acked Write 10 Done Done Sent false
18602 5727 Acked Write 3 Done Done Sent false
18603 5728 Acked Write 4 Done Done Sent false
18604 5729 Acked Write 1 Done Done Sent false
18605 5730 Acked Write 3 Done Done Sent false
18606 5731 Acked Read 2 Done Done Sent false
18607 5732 Acked Write 9 Done Done Sent false
18608 5733 Acked Read 4 Done Done Sent false
18609 5734 Acked Read 4 Done Done Sent false
18610 5735 Acked Read 9 Done Done Sent false
18611 5736 Acked Write 4 Done Done Sent false
18612 5737 Acked Write 2 Done Done Sent false
18613 5738 Acked Read 5 Done Done Sent false
18614 5739 Acked Write 4 Done Done Sent false
18615 5740 Acked Write 8 Done Done Sent false
18616 5741 Acked Read 6 Done Done Sent false
18617 5742 Acked Read 8 Done Done Sent false
18618 5743 Acked Read 9 Done Done Sent false
18619 5744 Acked Write 1 Done Done Sent false
18620 5745 Acked Read 10 Done Done Sent false
18621 5746 Acked Write 2 Done Done Sent false
18622 5747 Acked Read 8 Done Done Sent false
18623 5748 Acked Read 8 Done Done Sent false
18624 5749 Acked Write 4 Done Done Sent false
18625 5750 Acked Flush 0 Done Done Sent false
18626 5751 Acked Write 7 Done Done Sent false
18627 5752 Acked Write 7 Done Done Sent false
18628 5753 Acked Write 3 Done Done Sent false
18629 5754 Acked Write 8 Done Done Sent false
18630 5755 Acked Write 8 Done Done Sent false
18631 5756 Acked Read 2 Done Done Sent false
18632 5757 Acked Flush 0 Done Done Sent false
18633 5758 Acked Write 9 Done Done Sent false
18634 5759 Acked Read 10 Done Done Sent false
18635 5760 Acked Read 10 Done Done Sent false
18636 5761 Acked Read 9 Done Done Sent false
18637 5762 Acked Read 3 Done Done Sent false
18638 5763 Acked Read 1 Done Done Sent false
18639 5764 Acked Write 8 Done Done Sent false
18640 5765 Acked Read 4 Done Done Sent false
18641 5766 Acked Read 8 Done Done Sent false
18642 5767 Acked Write 3 Done Done Sent false
18643 5768 Acked Read 6 Done Done Sent false
18644 5769 Acked Read 8 Done Done Sent false
18645 5770 Acked Write 7 Done Done Sent false
18646 5771 Acked Write 10 Done Done Sent false
18647 5772 Acked Read 1 Done Done Sent false
18648 5773 Acked Read 7 Done Done Sent false
18649 5774 Acked Read 6 Done Done Sent false
18650 5775 Acked Read 2 Done Done Sent false
18651 5776 Acked Write 5 Done Done Sent false
18652 5777 Acked Read 8 Done Done Sent false
18653 5778 Acked Write 5 Done Done Sent false
18654 5779 Acked Write 10 Done Done Sent false
18655 5780 Acked Write 3 Done Done Sent false
18656 5781 Acked Read 4 Done Done Sent false
18657 5782 Acked Flush 0 Done Done Sent false
18658 5783 Acked Write 1 Done Done Sent false
18659 5784 Acked Write 4 Done Done Sent false
18660 5785 Acked Flush 0 Done Done Sent false
18661 5786 Acked Write 9 Done Done Sent false
18662 5787 Acked Read 6 Done Done Sent false
18663 5788 Acked Write 6 Done Done Sent false
18664 5789 Acked Write 1 Done Done Sent false
18665 5790 Acked Write 3 Done Done Sent false
18666 5791 Acked Write 10 Done Done Sent false
18667 5792 Acked Write 10 Done Done Sent false
18668 5793 Acked Read 9 Done Done Sent false
18669 5794 Acked Write 1 Done Done Sent false
18670 5795 Acked Read 7 Done Done Sent false
18671 5796 Acked Read 8 Done Done Sent false
18672 5797 Acked Read 4 Done Done Sent false
18673 5798 Acked Write 7 Done Done Sent false
18674 5799 Acked Write 7 Done Done Sent false
18675 5800 Acked Read 10 Done Done Sent false
18676 5801 Acked Read 5 Done Done Sent false
18677 5802 Acked Write 4 Done Done Sent false
18678 5803 Acked Write 5 Done Done Sent false
18679 5804 Acked Write 3 Done Done Sent false
18680 5805 Acked Write 5 Done Done Sent false
18681 5806 Acked Flush 0 Done Done Sent false
18682 5807 Acked Write 8 Done Done Sent false
18683 5808 Acked Read 7 Done Done Sent false
18684 5809 Acked Write 4 Done Done Sent false
18685 5810 Acked Read 7 Done Done Sent false
18686 5811 Acked Write 9 Done Done Sent false
18687 5812 Acked Read 9 Done Done Sent false
18688 5813 Acked Read 3 Done Done Sent false
18689 5814 Acked Flush 0 Done Done Sent false
18690 5815 Acked Write 3 Done Done Sent false
18691 5816 Acked Read 2 Done Done Sent false
18692 5817 Acked Read 4 Done Done Sent false
18693 5818 Acked Flush 0 Done Done Sent false
18694 5819 Acked Read 9 Done Done Sent false
18695 5820 Acked Write 5 Done Done Sent false
18696 5821 Acked Write 8 Done Done Sent false
18697 5822 Acked Write 3 Done Done Sent false
18698 5823 Acked Read 3 Done Done Sent false
18699 5824 Acked Write 6 Done Done Sent false
18700 5825 Acked Read 6 Done Done Sent false
18701 5826 Acked Read 9 Done Done Sent false
18702 5827 Acked Read 10 Done Done Sent false
18703 5828 Acked Read 6 Done Done Sent false
18704 5829 Acked Write 10 Done Done Sent false
18705 STATES DS:0 DS:1 DS:2 TOTAL
18706 Sent 0 0 301 301
18707 Done 301 301 0 602
18708 Skipped 0 0 0 0
18709 Error 0 0 0 0
18710 Last Flush: Some(JobId(5818)) Some(JobId(5818)) Some(JobId(5528))
18711 Downstairs last five completed: 5528 5527 5526 5525 5524
18712 Upstairs last five completed: 5829 5828 5827 5826 5825
18713 CLIENT: Up:0 ds:301 act:2
187142025-02-25T20:50:41.448ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187152025-02-25T20:50:42.449ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187162025-02-25T20:50:42.456ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187172025-02-25T20:50:42.456ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54183) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187182025-02-25T20:50:42.458ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187192025-02-25T20:50:42.458ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187202025-02-25T20:50:42.458ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187212025-02-25T20:50:42.459ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187222025-02-25T20:50:42.459ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187232025-02-25T20:50:42.459ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
187242025-02-25T20:50:42.459ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
18725 ----------------------------------------------------------------
18726 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18727 Downstairs last five completed: 5830 5829 5828 5827 5826
18728 Upstairs last five completed: 5830 5829 5828 5827 5826
18729 CLIENT: Up:0 ds:0 act:3
18730 Replay: All jobs finished
18731 Main task has work to do, go find it
18732 got dsc Stop(2)
18733 stop 2
18734 Feb 25 20:50:44.718 INFO request completed, latency_us: 351, response_code: 204, [8830] Got stop action so:false kr:false
18735 uri: /stop/cid/2, method: POST, req_id: 9f32a33b-2891-40c4-b776-3a462bfceb77, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
18736 Feb 25 20:50:44.718{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:50:44.718826273Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
18737 INFO request completed, latency_us: 157, response_code: 200, uri: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:50:44.718883399Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
18738 /state/cid/2, method: GET{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:50:44.718919367Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
18739 , req_id: a45eb852-0b15-486b-8d6e-87a0bc508bae, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
187402025-02-25T20:50:44.719ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
18741 [8830] Exited with: ExitStatus(unix_wait_status(9))
18742 [8830][2] reports Exit
18743 Feb 25 20:50:48.721 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1a28eb12-0e1b-4478-abe7-2ea0c88cab09, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
18744 001/300 Read block 1997 len 40960
18745 002/300 Read block 79 len 24576
18746 003/300 Write block 3792 len 16384 data: 2 2 2 2
18747 004/300 Read block 3276 len 20480
18748 005/300 Write block 3524 len 36864 data: 3 4 4 3 2 3 3 4 4
18749 006/300 Write block 65 len 4096 data: 3
18750 007/300 Write block 2585 len 8192 data: 5 5
18751 008/300 Read block 4922 len 28672
18752 009/300 Read block 4352 len 8192
18753 010/300 Read block 3144 len 12288
18754 011/300 Read block 1206 len 32768
18755 012/300 Write block 4416 len 24576 data: 4 4 6 5 6 8
18756 013/300 Write block 1803 len 40960 data: 3 4 4 5 5 3 3 3 3 5
18757 014/300 Write block 264 len 28672 data: 3 3 3 3 5 6 5
18758 015/300 Write block 198 len 16384 data: 5 4 4 4
18759 016/300 Read block 1076 len 24576
18760 017/300 Write block 404 len 36864 data: 3 3 5 5 4 5 5 5 6
18761 018/300 Read block 3812 len 4096
18762 019/300 Write block 4866 len 32768 data: 2 2 2 2 3 4 5 4
18763 020/300 Read block 3061 len 40960
18764 021/300 Write block 1521 len 20480 data: 3 3 3 4 4
18765 022/300 Flush
18766 023/300 Read block 1311 len 16384
18767 024/300 Read block 2015 len 20480
18768 025/300 Write block 4694 len 32768 data: 6 5 5 5 5 4 5 3
18769 026/300 Write block 1890 len 36864 data: 6 5 5 6 6 6 4 4 5
18770 027/300 Write block 2844 len 20480 data: 5 6 4 4 4
18771 028/300 Read block 652 len 12288
18772 029/300 Write block 1641 len 12288 data: 2 2 2
18773 030/300 Write block 793 len 12288 data: 3 3 4
18774 031/300 Write block 2899 len 12288 data: 7 7 7
18775 032/300 Write block 2773 len 4096 data: 7
18776 033/300 Write block 1382 len 28672 data: 2 2 2 3 3 4 6
18777 034/300 Read block 640 len 4096
18778 035/300 Write block 3431 len 36864 data: 3 4 5 6 6 6 5 3 4
18779 036/300 Write block 675 len 12288 data: 9 6 4
18780 037/300 Write block 4955 len 24576 data: 4 4 3 3 3 3
18781 038/300 Write block 3246 len 40960 data: 6 4 3 3 4 4 6 6 6 6
18782 039/300 Read block 2657 len 32768
18783 040/300 Read block 3093 len 24576
18784 041/300 Flush
18785 042/300 Write block 1953 len 40960 data: 4 5 4 4 4 5 5 4 5 5
18786 043/300 Write block 3417 len 36864 data: 7 7 6 6 6 7 6 6 5
18787 044/300 Read block 2390 len 16384
18788 045/300 Write block 485 len 4096 data: 5
18789 046/300 Write block 2449 len 12288 data: 7 9 8
18790 047/300 Read block 2671 len 4096
18791 048/300 Write block 2293 len 16384 data: 2 2 2 2
18792 049/300 Write block 1245 len 20480 data: 4 4 5 5 5
18793 050/300 Write block 2063 len 28672 data: 3 3 3 3 3 2 2
18794 051/300 Read block 2280 len 8192
18795 052/300 Write block 1569 len 12288 data: 2 4 3
18796 053/300 Read block 1408 len 8192
18797 054/300 Flush
18798 055/300 Read block 1491 len 8192
18799 056/300 Flush
18800 057/300 Write block 2628 len 4096 data: 5
18801 058/300 Write block 252 len 20480 data: 5 5 5 3 4
18802 059/300 Write block 4605 len 36864 data: 4 4 2 2 3 3 4 4 5
18803 060/300 Read block 977 len 12288
18804 061/300 Read block 3579 len 8192
18805 062/300 Flush
18806 063/300 Read block 340 len 28672
18807 064/300 Write block 421 len 4096 data: 5
18808 065/300 Write block 4504 len 16384 data: 2 3 3 4
18809 066/300 Read block 1015 len 40960
18810 067/300 Write block 4525 len 16384 data: 4 6 6 7
18811 068/300 Write block 296 len 24576 data: 2 3 2 3 2 2
18812 069/300 Read block 1468 len 12288
18813 070/300 Write block 64 len 40960 data: 3 4 2 2 2 3 4 4 4 4
18814 071/300 Write block 225 len 16384 data: 4 6 7 7
18815 072/300 Write block 1258 len 40960 data: 6 5 7 7 7 7 6 4 5 5
18816 073/300 Read block 2172 len 24576
18817 074/300 Read block 1534 len 16384
18818 075/300 Read block 908 len 8192
18819 076/300 Flush
18820 077/300 Write block 3307 len 16384 data: 3 3 3 5
18821 078/300 Read block 2187 len 36864
18822 079/300 Flush
18823 080/300 Read block 4915 len 40960
18824 081/300 Write block 3865 len 28672 data: 3 3 3 3 2 2 5
18825 082/300 Flush
18826 083/300 Write block 1479 len 12288 data: 7 7 7
18827 084/300 Write block 2224 len 8192 data: 5 5
18828 085/300 Write block 3765 len 20480 data: 6 6 4 2 4
18829 086/300 Read block 3898 len 16384
18830 087/300 Read block 4742 len 16384
18831 088/300 Write block 3791 len 40960 data: 3 3 3 3 3 3 4 5 5 5
18832 089/300 Read block 4393 len 36864
18833 090/300 Write block 2980 len 36864 data: 3 4 6 6 6 6 8 9 11
18834 091/300 Flush
18835 092/300 Read block 2185 len 36864
18836 093/300 Write block 1058 len 4096 data: 4
18837 094/300 Write block 3093 len 8192 data: 3 3
18838 095/300 Read block 178 len 4096
18839 096/300 Read block 2965 len 8192
18840 097/300 Read block 2050 len 24576
18841 098/300 Write block 1416 len 4096 data: 6
18842 099/300 Read block 3724 len 12288
18843 100/300 Read block 116 len 16384
18844 101/300 Read block 2177 len 24576
18845 102/300 Read block 3400 len 24576
18846 103/300 Write block 178 len 4096 data: 4
18847 104/300 Write block 1585 len 8192 data: 9 8
18848 105/300 Read block 3435 len 8192
18849 106/300 Write block 3265 len 36864 data: 4 5 4 5 4 5 2 3 3
18850 107/300 Read block 3413 len 32768
18851 108/300 Read block 3455 len 36864
18852 109/300 Read block 790 len 36864
18853 110/300 Write block 1370 len 24576 data: 5 4 4 4 2 2
18854 111/300 Flush
18855 112/300 Read block 677 len 24576
18856 113/300 Read block 1855 len 32768
18857 114/300 Write block 1654 len 36864 data: 3 3 3 4 4 4 4 4 4
18858 115/300 Write block 498 len 16384 data: 4 4 4 5
18859 116/300 Flush
18860 117/300 Read block 4242 len 4096
18861 118/300 Read block 1314 len 8192
18862 119/300 Read block 4302 len 16384
18863 120/300 Read block 612 len 4096
18864 121/300 Write block 2227 len 20480 data: 4 4 5 4 4
18865 122/300 Read block 2227 len 12288
18866 123/300 Flush
18867 124/300 Read block 3721 len 12288
18868 125/300 Read block 1542 len 32768
18869 126/300 Read block 3042 len 32768
18870 127/300 Read block 3773 len 24576
18871 128/300 Read block 3909 len 8192
18872 129/300 Write block 4673 len 24576 data: 3 3 3 3 3 3
18873 130/300 Read block 1763 len 40960
18874 131/300 Flush
18875 132/300 Read block 1335 len 28672
18876 133/300 Read block 1230 len 28672
18877 134/300 Write block 321 len 4096 data: 5
18878 135/300 Read block 259 len 28672
18879 136/300 Write block 3434 len 4096 data: 7
18880 137/300 Flush
18881 138/300 Write block 4955 len 36864 data: 5 5 4 4 4 4 3 3 3
18882 139/300 Write block 2921 len 24576 data: 7 7 5 4 3 3
18883 140/300 Write block 424 len 32768 data: 7 8 8 7 5 4 6 5
18884 141/300 Flush
18885 142/300 Write block 668 len 4096 data: 6
18886 143/300 Read block 1411 len 16384
18887 144/300 Read block 4059 len 28672
18888 145/300 Flush
18889 146/300 Read block 2268 len 8192
18890 147/300 Read block 1247 len 40960
18891 148/300 Read block 358 len 16384
18892 149/300 Read block 1497 len 28672
18893 150/300 Read block 2650 len 12288
18894 151/300 Write block 155 len 32768 data: 3 2 2 2 2 3 4 5
18895 152/300 Read block 3237 len 40960
18896 153/300 Write block 2154 len 36864 data: 4 4 4 5 5 7 7 5 5
18897 154/300 Read block 3357 len 24576
18898 155/300 Read block 296 len 4096
18899 156/300 Write block 3213 len 4096 data: 4
18900 157/300 Write block 3919 len 36864 data: 4 4 4 3 3 2 2 2 2
18901 158/300 Read block 3478 len 28672
18902 159/300 Write block 3269 len 40960 data: 5 6 3 4 4 3 3 5 5 6
18903 160/300 Write block 3755 len 20480 data: 4 4 4 5 7
18904 161/300 Read block 1873 len 4096
18905 162/300 Write block 2579 len 40960 data: 3 3 3 4 4 5 6 6 4 4
18906 163/300 Read block 3132 len 24576
18907 164/300 Read block 2985 len 20480
18908 165/300 Write block 4983 len 40960 data: 2 3 3 3 4 4 5 4 5 6
18909 166/300 Read block 1022 len 12288
18910 167/300 Write block 4555 len 12288 data: 3 3 3
18911 168/300 Read block 565 len 4096
18912 169/300 Read block 3210 len 28672
18913 170/300 Flush
18914 171/300 Write block 3660 len 28672 data: 7 6 6 6 4 4 4
18915 172/300 Write block 3189 len 40960 data: 2 2 2 2 2 2 3 3 3 5
18916 173/300 Write block 3954 len 24576 data: 4 5 6 5 6 5
18917 174/300 Write block 1458 len 24576 data: 7 7 6 6 7 5
18918 175/300 Write block 828 len 12288 data: 2 2 2
18919 176/300 Read block 2104 len 12288
18920 177/300 Read block 479 len 4096
18921 178/300 Write block 4278 len 16384 data: 5 6 6 5
18922 179/300 Read block 4949 len 4096
18923 180/300 Write block 2662 len 28672 data: 4 4 3 4 3 3 3
18924 181/300 Write block 2961 len 4096 data: 6
18925 182/300 Read block 1577 len 32768
18926 183/300 Write block 2548 len 4096 data: 5
18927 184/300 Read block 132 len 12288
18928 185/300 Read block 2397 len 4096
18929 186/300 Read block 4262 len 32768
18930 187/300 Flush
18931 188/300 Write block 3041 len 28672 data: 2 3 3 3 2 2 3
18932 189/300 Read block 2038 len 36864
18933 190/300 Read block 1185 len 4096
18934 191/300 Write block 3358 len 36864 data: 2 2 2 2 2 2 3 3 3
18935 192/300 Read block 1955 len 24576
18936 193/300 Flush
18937 194/300 Read block 707 len 4096
18938 195/300 Write block 4099 len 36864 data: 5 6 5 5 5 5 3 3 3
18939 196/300 Write block 1591 len 28672 data: 3 3 3 2 4 4 4
18940 197/300 Write block 1091 len 24576 data: 4 5 5 5 7 7
18941 198/300 Read block 3763 len 36864
18942 199/300 Read block 3337 len 28672
18943 200/300 Write block 2283 len 36864 data: 4 4 5 4 4 3 3 2 2
18944 201/300 Read block 4202 len 12288
18945 202/300 Read block 3323 len 16384
18946 203/300 Write block 3586 len 24576 data: 4 4 4 4 3 3
18947 204/300 Read block 819 len 4096
18948 205/300 Write block 2850 len 24576 data: 3 3 3 4 3 3
18949 206/300 Write block 4394 len 40960 data: 5 5 4 4 5 5 6 4 4 4
18950 207/300 Read block 4958 len 20480
18951 208/300 Read block 1979 len 16384
18952 209/300 Flush
18953 210/300 Read block 642 len 20480
18954 211/300 Write block 171 len 16384 data: 4 4 6 6
18955 212/300 Flush
18956 213/300 Write block 3632 len 28672 data: 6 8 8 8 7 6 7
18957 214/300 Read block 1779 len 36864
18958 215/300 Write block 690 len 40960 data: 3 5 4 5 4 4 3 3 3 3
18959 216/300 Write block 993 len 16384 data: 5 5 5 5
18960 217/300 Read block 4841 len 40960
18961 218/300 Write block 3066 len 8192 data: 3 3
18962 219/300 Flush
18963 220/300 Write block 2099 len 4096 data: 4
18964 221/300 Write block 2543 len 16384 data: 4 4 6 6
18965 222/300 Flush
18966 223/300 Flush
18967 224/300 Write block 1363 len 24576 data: 6 6 5 4 5 4
18968 225/300 Read block 1679 len 12288
18969 226/300 Read block 3879 len 28672
18970 227/300 Write block 2590 len 24576 data: 3 3 3 3 3 3
18971 228/300 Read block 4106 len 24576
18972 229/300 Write block 2320 len 28672 data: 5 4 5 4 4 5 4
18973 230/300 Write block 4806 len 28672 data: 4 5 5 4 3 3 3
18974 231/300 Write block 4736 len 28672 data: 3 3 3 3 5 5 5
18975 232/300 Write block 2958 len 32768 data: 7 6 6 7 5 6 5 5
18976 233/300 Write block 2122 len 4096 data: 4
18977 234/300 Read block 3048 len 28672
18978 235/300 Read block 2416 len 28672
18979 236/300 Read block 1292 len 4096
18980 237/300 Read block 1152 len 40960
18981 238/300 Write block 3866 len 16384 data: 4 4 4 3
18982 239/300 Read block 3975 len 32768
18983 240/300 Read block 1917 len 16384
18984 241/300 Read block 3941 len 36864
18985 242/300 Read block 4169 len 32768
18986 243/300 Read block 1402 len 40960
18987 244/300 Write block 1522 len 12288 data: 4 4 5
18988 245/300 Write block 2029 len 32768 data: 5 5 6 6 5 3 4 5
18989 246/300 Read block 188 len 36864
18990 247/300 Read block 3136 len 16384
18991 248/300 Write block 4135 len 24576 data: 3 3 3 3 4 4
18992 249/300 Read block 1667 len 16384
18993 250/300 Read block 906 len 24576
18994 251/300 Read block 3749 len 28672
18995 252/300 Write block 3103 len 28672 data: 5 5 6 5 4 2 3
18996 253/300 Read block 3905 len 24576
18997 254/300 Read block 4622 len 24576
18998 255/300 Write block 2335 len 36864 data: 4 4 6 7 7 7 9 9 9
18999 256/300 Write block 2400 len 4096 data: 3
19000 257/300 Read block 2838 len 16384
19001 258/300 Read block 3674 len 36864
19002 259/300 Flush
19003 260/300 Write block 3267 len 16384 data: 5 6 6 7
19004 261/300 Read block 3724 len 24576
19005 262/300 Read block 4924 len 32768
19006 263/300 Read block 1646 len 24576
19007 264/300 Write block 4808 len 8192 data: 6 5
19008 265/300 Write block 3419 len 20480 data: 7 7 7 8 7
19009 266/300 Read block 3242 len 12288
19010 267/300 Write block 1468 len 20480 data: 2 2 2 2 2
19011 268/300 Write block 956 len 36864 data: 3 4 5 5 6 5 4 3 2
19012 269/300 Read block 1553 len 32768
19013 270/300 Read block 1877 len 4096
19014 271/300 Read block 4414 len 12288
19015 272/300 Read block 3651 len 36864
19016 273/300 Read block 3260 len 32768
19017 274/300 Write block 2713 len 32768 data: 3 2 2 2 2 4 5 5
19018 275/300 Write block 2048 len 12288 data: 3 3 3
19019 276/300 Read block 2199 len 16384
19020 277/300 Read block 3144 len 28672
19021 278/300 Write block 4130 len 20480 data: 5 5 3 4 4
19022 279/300 Read block 1705 len 40960
19023 280/300 Read block 1603 len 16384
19024 281/300 Write block 3974 len 24576 data: 6 6 5 6 5 6
19025 282/300 Read block 2491 len 40960
19026 283/300 Write block 4936 len 36864 data: 4 4 4 5 5 5 5 5 4
19027 284/300 Flush
19028 285/300 Write block 803 len 20480 data: 2 3 3 3 3
19029 286/300 Read block 3694 len 40960
19030 287/300 Read block 1715 len 24576
19031 288/300 Read block 1504 len 16384
19032 289/300 Write block 402 len 28672 data: 3 3 4 4 6 6 5
19033 290/300 Write block 2237 len 8192 data: 3 4
19034 291/300 Read block 198 len 4096
19035 292/300 Read block 676 len 24576
19036 293/300 Read block 1504 len 4096
19037 294/300 Flush
19038 295/300 Read block 3273 len 20480
19039 296/300 Write block 4639 len 8192 data: 3 3
19040 297/300 Read block 4805 len 8192
19041 298/300 Write block 966 len 12288 data: 3 3 3
19042 299/300 Read block 886 len 28672
19043 300/300 Write block 3343 len 32768 data: 2 3 3 5 4 5 5 5
19044 301/300 Flush
19045 Main task has work to do, go find it
19046 got dsc Start(2)
19047 start 2
19048 Feb 25 20:50:48.964 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: b7d8b723-639b-42fc-9c07-0a133a9b9f5a, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
19049 Replay: started 2, returned:Ok(())
19050 ----------------------------------------------------------------
19051 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19052 States: Active Active Offline
19053 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19054 5831 Acked Read 10 Done Done Sent false
19055 5832 Acked Read 6 Done Done Sent false
19056 5833 Acked Write 4 Done Done Sent false
19057 5834 Acked Read 5 Done Done Sent false
19058 5835 Acked Write 9 Done Done Sent false
19059 5836 Acked Write 1 Done Done Sent false
19060 5837 Acked Write 2 Done Done Sent false
19061 5838 Acked Read 7 Done Done Sent false
19062 5839 Acked Read 2 Done Done Sent false
19063 5840 Acked Read 3 Done Done Sent false
19064 5841 Acked Read 8 Done Done Sent false
19065 5842 Acked Write 6 Done Done Sent false
19066 5843 Acked Write 10 Done Done Sent false
19067 5844 Acked Write 7 Done Done Sent false
19068 5845 Acked Write 4 Done Done Sent false
19069 5846 Acked Read 6 Done Done Sent false
19070 5847 Acked Write 9 Done Done Sent false
19071 5848 Acked Read 1 Done Done Sent false
19072 5849 Acked Write 8 Done Done Sent false
19073 5850 Acked Read 10 Done Done Sent false
19074 5851 Acked Write 5 Done Done Sent false
19075 5852 Acked Flush 0 Done Done Sent false
19076 5853 Acked Read 4 Done Done Sent false
19077 5854 Acked Read 5 Done Done Sent false
19078 5855 Acked Write 8 Done Done Sent false
19079 5856 Acked Write 9 Done Done Sent false
19080 5857 Acked Write 5 Done Done Sent false
19081 5858 Acked Read 3 Done Done Sent false
19082 5859 Acked Write 3 Done Done Sent false
19083 5860 Acked Write 3 Done Done Sent false
19084 5861 Acked Write 3 Done Done Sent false
19085 5862 Acked Write 1 Done Done Sent false
19086 5863 Acked Write 7 Done Done Sent false
19087 5864 Acked Read 1 Done Done Sent false
19088 5865 Acked Write 9 Done Done Sent false
19089 5866 Acked Write 3 Done Done Sent false
19090 5867 Acked Write 6 Done Done Sent false
19091 5868 Acked Write 10 Done Done Sent false
19092 5869 Acked Read 8 Done Done Sent false
19093 5870 Acked Read 6 Done Done Sent false
19094 5871 Acked Flush 0 Done Done Sent false
19095 5872 Acked Write 10 Done Done Sent false
19096 5873 Acked Write 9 Done Done Sent false
19097 5874 Acked Read 4 Done Done Sent false
19098 5875 Acked Write 1 Done Done Sent false
19099 5876 Acked Write 3 Done Done Sent false
19100 5877 Acked Read 1 Done Done Sent false
19101 5878 Acked Write 4 Done Done Sent false
19102 5879 Acked Write 5 Done Done Sent false
19103 5880 Acked Write 7 Done Done Sent false
19104 5881 Acked Read 2 Done Done Sent false
19105 5882 Acked Write 3 Done Done Sent false
19106 5883 Acked Read 2 Done Done Sent false
19107 5884 Acked Flush 0 Done Done Sent false
19108 5885 Acked Read 2 Done Done Sent false
19109 5886 Acked Flush 0 Done Done Sent false
19110 5887 Acked Write 1 Done Done Sent false
19111 5888 Acked Write 5 Done Done Sent false
19112 5889 Acked Write 9 Done Done Sent false
19113 5890 Acked Read 3 Done Done Sent false
19114 5891 Acked Read 2 Done Done Sent false
19115 5892 Acked Flush 0 Done Done Sent false
19116 5893 Acked Read 7 Done Done Sent false
19117 5894 Acked Write 1 Done Done Sent false
19118 5895 Acked Write 4 Done Done Sent false
19119 5896 Acked Read 10 Done Done Sent false
19120 5897 Acked Write 4 Done Done Sent false
19121 5898 Acked Write 6 Done Done Sent false
19122 5899 Acked Read 3 Done Done Sent false
19123 5900 Acked Write 10 Done Done Sent false
19124 5901 Acked Write 4 Done Done Sent false
19125 5902 Acked Write 10 Done Done Sent false
19126 5903 Acked Read 6 Done Done Sent false
19127 5904 Acked Read 4 Done Done Sent false
19128 5905 Acked Read 2 Done Done Sent false
19129 5906 Acked Flush 0 Done Done Sent false
19130 5907 Acked Write 4 Done Done Sent false
19131 5908 Acked Read 9 Done Done Sent false
19132 5909 Acked Flush 0 Done Done Sent false
19133 5910 Acked Read 10 Done Done Sent false
19134 5911 Acked Write 7 Done Done Sent false
19135 5912 Acked Flush 0 Done Done Sent false
19136 5913 Acked Write 3 Done Done Sent false
19137 5914 Acked Write 2 Done Done Sent false
19138 5915 Acked Write 5 Done Done Sent false
19139 5916 Acked Read 4 Done Done Sent false
19140 5917 Acked Read 4 Done Done Sent false
19141 5918 Acked Write 10 Done Done Sent false
19142 5919 Acked Read 9 Done Done Sent false
19143 5920 Acked Write 9 Done Done Sent false
19144 5921 Acked Flush 0 Done Done Sent false
19145 5922 Acked Read 9 Done Done Sent false
19146 5923 Acked Write 1 Done Done Sent false
19147 5924 Acked Write 2 Done Done Sent false
19148 5925 Acked Read 1 Done Done Sent false
19149 5926 Acked Read 2 Done Done Sent false
19150 5927 Acked Read 6 Done Done Sent false
19151 5928 Acked Write 1 Done Done Sent false
19152 5929 Acked Read 3 Done Done Sent false
19153 5930 Acked Read 4 Done Done Sent false
19154 5931 Acked Read 6 Done Done Sent false
19155 5932 Acked Read 6 Done Done Sent false
19156 5933 Acked Write 1 Done Done Sent false
19157 5934 Acked Write 2 Done Done Sent false
19158 5935 Acked Read 2 Done Done Sent false
19159 5936 Acked Write 9 Done Done Sent false
19160 5937 Acked Read 8 Done Done Sent false
19161 5938 Acked Read 9 Done Done Sent false
19162 5939 Acked Read 9 Done Done Sent false
19163 5940 Acked Write 6 Done Done Sent false
19164 5941 Acked Flush 0 Done Done Sent false
19165 5942 Acked Read 6 Done Done Sent false
19166 5943 Acked Read 8 Done Done Sent false
19167 5944 Acked Write 9 Done Done Sent false
19168 5945 Acked Write 4 Done Done Sent false
19169 5946 Acked Flush 0 Done Done Sent false
19170 5947 Acked Read 1 Done Done Sent false
19171 5948 Acked Read 2 Done Done Sent false
19172 5949 Acked Read 4 Done Done Sent false
19173 5950 Acked Read 1 Done Done Sent false
19174 5951 Acked Write 5 Done Done Sent false
19175 5952 Acked Read 3 Done Done Sent false
19176 5953 Acked Flush 0 Done Done Sent false
19177 5954 Acked Read 3 Done Done Sent false
19178 5955 Acked Read 8 Done Done Sent false
19179 5956 Acked Read 8 Done Done Sent false
19180 5957 Acked Read 6 Done Done Sent false
19181 5958 Acked Read 2 Done Done Sent false
19182 5959 Acked Write 6 Done Done Sent false
19183 5960 Acked Read 10 Done Done Sent false
19184 5961 Acked Flush 0 Done Done Sent false
19185 5962 Acked Read 7 Done Done Sent false
19186 5963 Acked Read 7 Done Done Sent false
19187 5964 Acked Write 1 Done Done Sent false
19188 5965 Acked Read 7 Done Done Sent false
19189 5966 Acked Write 1 Done Done Sent false
19190 5967 Acked Flush 0 Done Done Sent false
19191 5968 Acked Write 9 Done Done Sent false
19192 5969 Acked Write 6 Done Done Sent false
19193 5970 Acked Write 8 Done Done Sent false
19194 5971 Acked Flush 0 Done Done Sent false
19195 5972 Acked Write 1 Done Done Sent false
19196 5973 Acked Read 4 Done Done Sent false
19197 5974 Acked Read 7 Done Done Sent false
19198 5975 Acked Flush 0 Done Done Sent false
19199 5976 Acked Read 2 Done Done Sent false
19200 5977 Acked Read 10 Done Done Sent false
19201 5978 Acked Read 4 Done Done Sent false
19202 5979 Acked Read 7 Done Done Sent false
19203 5980 Acked Read 3 Done Done Sent false
19204 5981 Acked Write 8 Done Done Sent false
19205 5982 Acked Read 10 Done Done Sent false
19206 5983 Acked Write 9 Done Done Sent false
19207 5984 Acked Read 6 Done Done Sent false
19208 5985 Acked Read 1 Done Done Sent false
19209 5986 Acked Write 1 Done Done Sent false
19210 5987 Acked Write 9 Done Done Sent false
19211 5988 Acked Read 7 Done Done Sent false
19212 5989 Acked Write 10 Done Done Sent false
19213 5990 Acked Write 5 Done Done Sent false
19214 5991 Acked Read 1 Done Done Sent false
19215 5992 Acked Write 10 Done Done Sent false
19216 5993 Acked Read 6 Done Done Sent false
19217 5994 Acked Read 5 Done Done Sent false
19218 5995 Acked Write 10 Done Done Sent false
19219 5996 Acked Read 3 Done Done Sent false
19220 5997 Acked Write 3 Done Done Sent false
19221 5998 Acked Read 1 Done Done Sent false
19222 5999 Acked Read 7 Done Done Sent false
19223 6000 Acked Flush 0 Done Done Sent false
19224 6001 Acked Write 7 Done Done Sent false
19225 6002 Acked Write 10 Done Done Sent false
19226 6003 Acked Write 6 Done Done Sent false
19227 6004 Acked Write 6 Done Done Sent false
19228 6005 Acked Write 3 Done Done Sent false
19229 6006 Acked Read 3 Done Done Sent false
19230 6007 Acked Read 1 Done Done Sent false
19231 6008 Acked Write 4 Done Done Sent false
19232 6009 Acked Read 1 Done Done Sent false
19233 6010 Acked Write 7 Done Done Sent false
19234 6011 Acked Write 1 Done Done Sent false
19235 6012 Acked Read 8 Done Done Sent false
19236 6013 Acked Write 1 Done Done Sent false
19237 6014 Acked Read 3 Done Done Sent false
19238 6015 Acked Read 1 Done Done Sent false
19239 6016 Acked Read 8 Done Done Sent false
19240 6017 Acked Flush 0 Done Done Sent false
19241 6018 Acked Write 7 Done Done Sent false
19242 6019 Acked Read 9 Done Done Sent false
19243 6020 Acked Read 1 Done Done Sent false
19244 6021 Acked Write 9 Done Done Sent false
19245 6022 Acked Read 6 Done Done Sent false
19246 6023 Acked Flush 0 Done Done Sent false
19247 6024 Acked Read 1 Done Done Sent false
19248 6025 Acked Write 9 Done Done Sent false
19249 6026 Acked Write 7 Done Done Sent false
19250 6027 Acked Write 6 Done Done Sent false
19251 6028 Acked Read 9 Done Done Sent false
19252 6029 Acked Read 7 Done Done Sent false
19253 6030 Acked Write 9 Done Done Sent false
19254 6031 Acked Read 3 Done Done Sent false
19255 6032 Acked Read 4 Done Done Sent false
19256 6033 Acked Write 6 Done Done Sent false
19257 6034 Acked Read 1 Done Done Sent false
19258 6035 Acked Write 6 Done Done Sent false
19259 6036 Acked Write 10 Done Done Sent false
19260 6037 Acked Read 5 Done Done Sent false
19261 6038 Acked Read 4 Done Done Sent false
19262 6039 Acked Flush 0 Done Done Sent false
19263 6040 Acked Read 5 Done Done Sent false
19264 6041 Acked Write 4 Done Done Sent false
19265 6042 Acked Flush 0 Done Done Sent false
19266 6043 Acked Write 7 Done Done Sent false
19267 6044 Acked Read 9 Done Done Sent false
19268 6045 Acked Write 10 Done Done Sent false
19269 6046 Acked Write 4 Done Done Sent false
19270 6047 Acked Read 10 Done Done Sent false
19271 6048 Acked Write 2 Done Done Sent false
19272 6049 Acked Flush 0 Done Done Sent false
19273 6050 Acked Write 1 Done Done Sent false
19274 6051 Acked Write 4 Done Done Sent false
19275 6052 Acked Flush 0 Done Done Sent false
19276 6053 Acked Flush 0 Done Done Sent false
19277 6054 Acked Write 6 Done Done Sent false
19278 6055 Acked Read 3 Done Done Sent false
19279 6056 Acked Read 7 Done Done Sent false
19280 6057 Acked Write 6 Done Done Sent false
19281 6058 Acked Read 6 Done Done Sent false
19282 6059 Acked Write 7 Done Done Sent false
19283 6060 Acked Write 7 Done Done Sent false
19284 6061 Acked Write 7 Done Done Sent false
19285 6062 Acked Write 8 Done Done Sent false
19286 6063 Acked Write 1 Done Done Sent false
19287 6064 Acked Read 7 Done Done Sent false
19288 6065 Acked Read 7 Done Done Sent false
19289 6066 Acked Read 1 Done Done Sent false
19290 6067 Acked Read 10 Done Done Sent false
19291 6068 Acked Write 4 Done Done Sent false
19292 6069 Acked Read 8 Done Done Sent false
19293 6070 Acked Read 4 Done Done Sent false
19294 6071 Acked Read 9 Done Done Sent false
19295 6072 Acked Read 8 Done Done Sent false
19296 6073 Acked Read 10 Done Done Sent false
19297 6074 Acked Write 3 Done Done Sent false
19298 6075 Acked Write 8 Done Done Sent false
19299 6076 Acked Read 9 Done Done Sent false
19300 6077 Acked Read 4 Done Done Sent false
19301 6078 Acked Write 6 Done Done Sent false
19302 6079 Acked Read 4 Done Done Sent false
19303 6080 Acked Read 6 Done Done Sent false
19304 6081 Acked Read 7 Done Done Sent false
19305 6082 Acked Write 7 Done Done Sent false
19306 6083 Acked Read 6 Done Done Sent false
19307 6084 Acked Read 6 Done Done Sent false
19308 6085 Acked Write 9 Done Done Sent false
19309 6086 Acked Write 1 Done Done Sent false
19310 6087 Acked Read 4 Done Done Sent false
19311 6088 Acked Read 9 Done Done Sent false
19312 6089 Acked Flush 0 Done Done Sent false
19313 6090 Acked Write 4 Done Done Sent false
19314 6091 Acked Read 6 Done Done Sent false
19315 6092 Acked Read 8 Done Done Sent false
19316 6093 Acked Read 6 Done Done Sent false
19317 6094 Acked Write 2 Done Done Sent false
19318 6095 Acked Write 5 Done Done Sent false
19319 6096 Acked Read 3 Done Done Sent false
19320 6097 Acked Write 5 Done Done Sent false
19321 6098 Acked Write 9 Done Done Sent false
19322 6099 Acked Read 8 Done Done Sent false
19323 6100 Acked Read 1 Done Done Sent false
19324 6101 Acked Read 3 Done Done Sent false
19325 6102 Acked Read 9 Done Done Sent false
19326 6103 Acked Read 8 Done Done Sent false
19327 6104 Acked Write 8 Done Done Sent false
19328 6105 Acked Write 3 Done Done Sent false
19329 6106 Acked Read 4 Done Done Sent false
19330 6107 Acked Read 7 Done Done Sent false
19331 6108 Acked Write 5 Done Done Sent false
19332 6109 Acked Read 10 Done Done Sent false
19333 6110 Acked Read 4 Done Done Sent false
19334 6111 Acked Write 6 Done Done Sent false
19335 6112 Acked Read 10 Done Done Sent false
19336 6113 Acked Write 9 Done Done Sent false
19337 6114 Acked Flush 0 Done Done Sent false
19338 6115 Acked Write 5 Done Done Sent false
19339 6116 Acked Read 10 Done Done Sent false
19340 6117 Acked Read 6 Done Done Sent false
19341 6118 Acked Read 4 Done Done Sent false
19342 6119 Acked Write 7 Done Done Sent false
19343 6120 Acked Write 2 Done Done Sent false
19344 6121 Acked Read 1 Done Done Sent false
19345 6122 Acked Read 6 Done Done Sent false
19346 6123 Acked Read 1 Done Done Sent false
19347 6124 Acked Flush 0 Done Done Sent false
19348 6125 Acked Read 5 Done Done Sent false
19349 6126 Acked Write 2 Done Done Sent false
19350 6127 Acked Read 2 Done Done Sent false
19351 6128 Acked Write 3 Done Done Sent false
19352 6129 Acked Read 7 Done Done Sent false
19353 6130 Acked Write 8 Done Done Sent false
19354 6131 Acked Flush 0 Done Done Sent false
19355 STATES DS:0 DS:1 DS:2 TOTAL
19356 Sent 0 0 301 301
19357 Done 301 301 0 602
19358 Skipped 0 0 0 0
19359 Error 0 0 0 0
19360 Last Flush: Some(JobId(6131)) Some(JobId(6131)) Some(JobId(5830))
19361 Downstairs last five completed: 5830 5829 5828 5827 5826
19362 Upstairs last five completed: 6131 6130 6129 6128 6127
19363 CLIENT: Up:0 ds:301 act:2
193642025-02-25T20:50:49.467ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
19365 [8830] got start action
19366 [8830] I am going to restart
19367 Starting downstairs at port 8830
19368 Make output file at "/tmp/dsc/downstairs-8830.txt"
19369 [8830][2] reports Starting
19370 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223)
19371 [8830][2] reports Running
193722025-02-25T20:50:51.461ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
193732025-02-25T20:50:52.462ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
19374 ----------------------------------------------------------------
19375 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19376 States: Active Active Offline
19377 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19378 5831 Acked Read 10 Done Done Sent false
19379 5832 Acked Read 6 Done Done Sent false
19380 5833 Acked Write 4 Done Done Sent false
19381 5834 Acked Read 5 Done Done Sent false
19382 5835 Acked Write 9 Done Done Sent false
19383 5836 Acked Write 1 Done Done Sent false
19384 5837 Acked Write 2 Done Done Sent false
19385 5838 Acked Read 7 Done Done Sent false
19386 5839 Acked Read 2 Done Done Sent false
19387 5840 Acked Read 3 Done Done Sent false
19388 5841 Acked Read 8 Done Done Sent false
19389 5842 Acked Write 6 Done Done Sent false
19390 5843 Acked Write 10 Done Done Sent false
19391 5844 Acked Write 7 Done Done Sent false
19392 5845 Acked Write 4 Done Done Sent false
19393 5846 Acked Read 6 Done Done Sent false
19394 5847 Acked Write 9 Done Done Sent false
19395 5848 Acked Read 1 Done Done Sent false
19396 5849 Acked Write 8 Done Done Sent false
19397 5850 Acked Read 10 Done Done Sent false
19398 5851 Acked Write 5 Done Done Sent false
19399 5852 Acked Flush 0 Done Done Sent false
19400 5853 Acked Read 4 Done Done Sent false
19401 5854 Acked Read 5 Done Done Sent false
19402 5855 Acked Write 8 Done Done Sent false
19403 5856 Acked Write 9 Done Done Sent false
19404 5857 Acked Write 5 Done Done Sent false
19405 5858 Acked Read 3 Done Done Sent false
19406 5859 Acked Write 3 Done Done Sent false
19407 5860 Acked Write 3 Done Done Sent false
19408 5861 Acked Write 3 Done Done Sent false
19409 5862 Acked Write 1 Done Done Sent false
19410 5863 Acked Write 7 Done Done Sent false
19411 5864 Acked Read 1 Done Done Sent false
19412 5865 Acked Write 9 Done Done Sent false
19413 5866 Acked Write 3 Done Done Sent false
19414 5867 Acked Write 6 Done Done Sent false
19415 5868 Acked Write 10 Done Done Sent false
19416 5869 Acked Read 8 Done Done Sent false
19417 5870 Acked Read 6 Done Done Sent false
19418 5871 Acked Flush 0 Done Done Sent false
19419 5872 Acked Write 10 Done Done Sent false
19420 5873 Acked Write 9 Done Done Sent false
19421 5874 Acked Read 4 Done Done Sent false
19422 5875 Acked Write 1 Done Done Sent false
19423 5876 Acked Write 3 Done Done Sent false
19424 5877 Acked Read 1 Done Done Sent false
19425 5878 Acked Write 4 Done Done Sent false
19426 5879 Acked Write 5 Done Done Sent false
19427 5880 Acked Write 7 Done Done Sent false
19428 5881 Acked Read 2 Done Done Sent false
19429 5882 Acked Write 3 Done Done Sent false
19430 5883 Acked Read 2 Done Done Sent false
19431 5884 Acked Flush 0 Done Done Sent false
19432 5885 Acked Read 2 Done Done Sent false
19433 5886 Acked Flush 0 Done Done Sent false
19434 5887 Acked Write 1 Done Done Sent false
19435 5888 Acked Write 5 Done Done Sent false
19436 5889 Acked Write 9 Done Done Sent false
19437 5890 Acked Read 3 Done Done Sent false
19438 5891 Acked Read 2 Done Done Sent false
19439 5892 Acked Flush 0 Done Done Sent false
19440 5893 Acked Read 7 Done Done Sent false
19441 5894 Acked Write 1 Done Done Sent false
19442 5895 Acked Write 4 Done Done Sent false
19443 5896 Acked Read 10 Done Done Sent false
19444 5897 Acked Write 4 Done Done Sent false
19445 5898 Acked Write 6 Done Done Sent false
19446 5899 Acked Read 3 Done Done Sent false
19447 5900 Acked Write 10 Done Done Sent false
19448 5901 Acked Write 4 Done Done Sent false
19449 5902 Acked Write 10 Done Done Sent false
19450 5903 Acked Read 6 Done Done Sent false
19451 5904 Acked Read 4 Done Done Sent false
19452 5905 Acked Read 2 Done Done Sent false
19453 5906 Acked Flush 0 Done Done Sent false
19454 5907 Acked Write 4 Done Done Sent false
19455 5908 Acked Read 9 Done Done Sent false
19456 5909 Acked Flush 0 Done Done Sent false
19457 5910 Acked Read 10 Done Done Sent false
19458 5911 Acked Write 7 Done Done Sent false
19459 5912 Acked Flush 0 Done Done Sent false
19460 5913 Acked Write 3 Done Done Sent false
19461 5914 Acked Write 2 Done Done Sent false
19462 5915 Acked Write 5 Done Done Sent false
19463 5916 Acked Read 4 Done Done Sent false
19464 5917 Acked Read 4 Done Done Sent false
19465 5918 Acked Write 10 Done Done Sent false
19466 5919 Acked Read 9 Done Done Sent false
19467 5920 Acked Write 9 Done Done Sent false
19468 5921 Acked Flush 0 Done Done Sent false
19469 5922 Acked Read 9 Done Done Sent false
19470 5923 Acked Write 1 Done Done Sent false
19471 5924 Acked Write 2 Done Done Sent false
19472 5925 Acked Read 1 Done Done Sent false
19473 5926 Acked Read 2 Done Done Sent false
19474 5927 Acked Read 6 Done Done Sent false
19475 5928 Acked Write 1 Done Done Sent false
19476 5929 Acked Read 3 Done Done Sent false
19477 5930 Acked Read 4 Done Done Sent false
19478 5931 Acked Read 6 Done Done Sent false
19479 5932 Acked Read 6 Done Done Sent false
19480 5933 Acked Write 1 Done Done Sent false
19481 5934 Acked Write 2 Done Done Sent false
19482 5935 Acked Read 2 Done Done Sent false
19483 5936 Acked Write 9 Done Done Sent false
19484 5937 Acked Read 8 Done Done Sent false
19485 5938 Acked Read 9 Done Done Sent false
19486 5939 Acked Read 9 Done Done Sent false
19487 5940 Acked Write 6 Done Done Sent false
19488 5941 Acked Flush 0 Done Done Sent false
19489 5942 Acked Read 6 Done Done Sent false
19490 5943 Acked Read 8 Done Done Sent false
19491 5944 Acked Write 9 Done Done Sent false
19492 5945 Acked Write 4 Done Done Sent false
19493 5946 Acked Flush 0 Done Done Sent false
19494 5947 Acked Read 1 Done Done Sent false
19495 5948 Acked Read 2 Done Done Sent false
19496 5949 Acked Read 4 Done Done Sent false
19497 5950 Acked Read 1 Done Done Sent false
19498 5951 Acked Write 5 Done Done Sent false
19499 5952 Acked Read 3 Done Done Sent false
19500 5953 Acked Flush 0 Done Done Sent false
19501 5954 Acked Read 3 Done Done Sent false
19502 5955 Acked Read 8 Done Done Sent false
19503 5956 Acked Read 8 Done Done Sent false
19504 5957 Acked Read 6 Done Done Sent false
19505 5958 Acked Read 2 Done Done Sent false
19506 5959 Acked Write 6 Done Done Sent false
19507 5960 Acked Read 10 Done Done Sent false
19508 5961 Acked Flush 0 Done Done Sent false
19509 5962 Acked Read 7 Done Done Sent false
19510 5963 Acked Read 7 Done Done Sent false
19511 5964 Acked Write 1 Done Done Sent false
19512 5965 Acked Read 7 Done Done Sent false
19513 5966 Acked Write 1 Done Done Sent false
19514 5967 Acked Flush 0 Done Done Sent false
19515 5968 Acked Write 9 Done Done Sent false
19516 5969 Acked Write 6 Done Done Sent false
19517 5970 Acked Write 8 Done Done Sent false
19518 5971 Acked Flush 0 Done Done Sent false
19519 5972 Acked Write 1 Done Done Sent false
19520 5973 Acked Read 4 Done Done Sent false
19521 5974 Acked Read 7 Done Done Sent false
19522 5975 Acked Flush 0 Done Done Sent false
19523 5976 Acked Read 2 Done Done Sent false
19524 5977 Acked Read 10 Done Done Sent false
19525 5978 Acked Read 4 Done Done Sent false
19526 5979 Acked Read 7 Done Done Sent false
19527 5980 Acked Read 3 Done Done Sent false
19528 5981 Acked Write 8 Done Done Sent false
19529 5982 Acked Read 10 Done Done Sent false
19530 5983 Acked Write 9 Done Done Sent false
19531 5984 Acked Read 6 Done Done Sent false
19532 5985 Acked Read 1 Done Done Sent false
19533 5986 Acked Write 1 Done Done Sent false
19534 5987 Acked Write 9 Done Done Sent false
19535 5988 Acked Read 7 Done Done Sent false
19536 5989 Acked Write 10 Done Done Sent false
19537 5990 Acked Write 5 Done Done Sent false
19538 5991 Acked Read 1 Done Done Sent false
19539 5992 Acked Write 10 Done Done Sent false
19540 5993 Acked Read 6 Done Done Sent false
19541 5994 Acked Read 5 Done Done Sent false
19542 5995 Acked Write 10 Done Done Sent false
19543 5996 Acked Read 3 Done Done Sent false
19544 5997 Acked Write 3 Done Done Sent false
19545 5998 Acked Read 1 Done Done Sent false
19546 5999 Acked Read 7 Done Done Sent false
19547 6000 Acked Flush 0 Done Done Sent false
19548 6001 Acked Write 7 Done Done Sent false
19549 6002 Acked Write 10 Done Done Sent false
19550 6003 Acked Write 6 Done Done Sent false
19551 6004 Acked Write 6 Done Done Sent false
19552 6005 Acked Write 3 Done Done Sent false
19553 6006 Acked Read 3 Done Done Sent false
19554 6007 Acked Read 1 Done Done Sent false
19555 6008 Acked Write 4 Done Done Sent false
19556 6009 Acked Read 1 Done Done Sent false
19557 6010 Acked Write 7 Done Done Sent false
19558 6011 Acked Write 1 Done Done Sent false
19559 6012 Acked Read 8 Done Done Sent false
19560 6013 Acked Write 1 Done Done Sent false
19561 6014 Acked Read 3 Done Done Sent false
19562 6015 Acked Read 1 Done Done Sent false
19563 6016 Acked Read 8 Done Done Sent false
19564 6017 Acked Flush 0 Done Done Sent false
19565 6018 Acked Write 7 Done Done Sent false
19566 6019 Acked Read 9 Done Done Sent false
19567 6020 Acked Read 1 Done Done Sent false
19568 6021 Acked Write 9 Done Done Sent false
19569 6022 Acked Read 6 Done Done Sent false
19570 6023 Acked Flush 0 Done Done Sent false
19571 6024 Acked Read 1 Done Done Sent false
19572 6025 Acked Write 9 Done Done Sent false
19573 6026 Acked Write 7 Done Done Sent false
19574 6027 Acked Write 6 Done Done Sent false
19575 6028 Acked Read 9 Done Done Sent false
19576 6029 Acked Read 7 Done Done Sent false
19577 6030 Acked Write 9 Done Done Sent false
19578 6031 Acked Read 3 Done Done Sent false
19579 6032 Acked Read 4 Done Done Sent false
19580 6033 Acked Write 6 Done Done Sent false
19581 6034 Acked Read 1 Done Done Sent false
19582 6035 Acked Write 6 Done Done Sent false
19583 6036 Acked Write 10 Done Done Sent false
19584 6037 Acked Read 5 Done Done Sent false
19585 6038 Acked Read 4 Done Done Sent false
19586 6039 Acked Flush 0 Done Done Sent false
19587 6040 Acked Read 5 Done Done Sent false
19588 6041 Acked Write 4 Done Done Sent false
19589 6042 Acked Flush 0 Done Done Sent false
19590 6043 Acked Write 7 Done Done Sent false
19591 6044 Acked Read 9 Done Done Sent false
19592 6045 Acked Write 10 Done Done Sent false
19593 6046 Acked Write 4 Done Done Sent false
19594 6047 Acked Read 10 Done Done Sent false
19595 6048 Acked Write 2 Done Done Sent false
19596 6049 Acked Flush 0 Done Done Sent false
19597 6050 Acked Write 1 Done Done Sent false
19598 6051 Acked Write 4 Done Done Sent false
19599 6052 Acked Flush 0 Done Done Sent false
19600 6053 Acked Flush 0 Done Done Sent false
19601 6054 Acked Write 6 Done Done Sent false
19602 6055 Acked Read 3 Done Done Sent false
19603 6056 Acked Read 7 Done Done Sent false
19604 6057 Acked Write 6 Done Done Sent false
19605 6058 Acked Read 6 Done Done Sent false
19606 6059 Acked Write 7 Done Done Sent false
19607 6060 Acked Write 7 Done Done Sent false
19608 6061 Acked Write 7 Done Done Sent false
19609 6062 Acked Write 8 Done Done Sent false
19610 6063 Acked Write 1 Done Done Sent false
19611 6064 Acked Read 7 Done Done Sent false
19612 6065 Acked Read 7 Done Done Sent false
19613 6066 Acked Read 1 Done Done Sent false
19614 6067 Acked Read 10 Done Done Sent false
19615 6068 Acked Write 4 Done Done Sent false
19616 6069 Acked Read 8 Done Done Sent false
19617 6070 Acked Read 4 Done Done Sent false
19618 6071 Acked Read 9 Done Done Sent false
19619 6072 Acked Read 8 Done Done Sent false
19620 6073 Acked Read 10 Done Done Sent false
19621 6074 Acked Write 3 Done Done Sent false
19622 6075 Acked Write 8 Done Done Sent false
19623 6076 Acked Read 9 Done Done Sent false
19624 6077 Acked Read 4 Done Done Sent false
19625 6078 Acked Write 6 Done Done Sent false
19626 6079 Acked Read 4 Done Done Sent false
19627 6080 Acked Read 6 Done Done Sent false
19628 6081 Acked Read 7 Done Done Sent false
19629 6082 Acked Write 7 Done Done Sent false
19630 6083 Acked Read 6 Done Done Sent false
19631 6084 Acked Read 6 Done Done Sent false
19632 6085 Acked Write 9 Done Done Sent false
19633 6086 Acked Write 1 Done Done Sent false
19634 6087 Acked Read 4 Done Done Sent false
19635 6088 Acked Read 9 Done Done Sent false
19636 6089 Acked Flush 0 Done Done Sent false
19637 6090 Acked Write 4 Done Done Sent false
19638 6091 Acked Read 6 Done Done Sent false
19639 6092 Acked Read 8 Done Done Sent false
19640 6093 Acked Read 6 Done Done Sent false
19641 6094 Acked Write 2 Done Done Sent false
19642 6095 Acked Write 5 Done Done Sent false
19643 6096 Acked Read 3 Done Done Sent false
19644 6097 Acked Write 5 Done Done Sent false
19645 6098 Acked Write 9 Done Done Sent false
19646 6099 Acked Read 8 Done Done Sent false
19647 6100 Acked Read 1 Done Done Sent false
19648 6101 Acked Read 3 Done Done Sent false
19649 6102 Acked Read 9 Done Done Sent false
19650 6103 Acked Read 8 Done Done Sent false
19651 6104 Acked Write 8 Done Done Sent false
19652 6105 Acked Write 3 Done Done Sent false
19653 6106 Acked Read 4 Done Done Sent false
19654 6107 Acked Read 7 Done Done Sent false
19655 6108 Acked Write 5 Done Done Sent false
19656 6109 Acked Read 10 Done Done Sent false
19657 6110 Acked Read 4 Done Done Sent false
19658 6111 Acked Write 6 Done Done Sent false
19659 6112 Acked Read 10 Done Done Sent false
19660 6113 Acked Write 9 Done Done Sent false
19661 6114 Acked Flush 0 Done Done Sent false
19662 6115 Acked Write 5 Done Done Sent false
19663 6116 Acked Read 10 Done Done Sent false
19664 6117 Acked Read 6 Done Done Sent false
19665 6118 Acked Read 4 Done Done Sent false
19666 6119 Acked Write 7 Done Done Sent false
19667 6120 Acked Write 2 Done Done Sent false
19668 6121 Acked Read 1 Done Done Sent false
19669 6122 Acked Read 6 Done Done Sent false
19670 6123 Acked Read 1 Done Done Sent false
19671 6124 Acked Flush 0 Done Done Sent false
19672 6125 Acked Read 5 Done Done Sent false
19673 6126 Acked Write 2 Done Done Sent false
19674 6127 Acked Read 2 Done Done Sent false
19675 6128 Acked Write 3 Done Done Sent false
19676 6129 Acked Read 7 Done Done Sent false
19677 6130 Acked Write 8 Done Done Sent false
19678 6131 Acked Flush 0 Done Done Sent false
19679 STATES DS:0 DS:1 DS:2 TOTAL
19680 Sent 0 0 301 301
19681 Done 301 301 0 602
19682 Skipped 0 0 0 0
19683 Error 0 0 0 0
19684 Last Flush: Some(JobId(6131)) Some(JobId(6131)) Some(JobId(5830))
19685 Downstairs last five completed: 5830 5829 5828 5827 5826
19686 Upstairs last five completed: 6131 6130 6129 6128 6127
19687 CLIENT: Up:0 ds:301 act:2
196882025-02-25T20:50:53.468ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196892025-02-25T20:50:54.467ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196902025-02-25T20:50:54.719ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196912025-02-25T20:50:54.719ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44564) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196922025-02-25T20:50:54.720ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196932025-02-25T20:50:54.721ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196942025-02-25T20:50:54.721ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196952025-02-25T20:50:54.721ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196962025-02-25T20:50:54.721ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196972025-02-25T20:50:54.721ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
196982025-02-25T20:50:54.721ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
19699 ----------------------------------------------------------------
19700 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19701 Downstairs last five completed: 6131 6130 6129 6128 6127
19702 Upstairs last five completed: 6131 6130 6129 6128 6127
19703 CLIENT: Up:0 ds:0 act:3
19704 Replay: All jobs finished
19705 Main task has work to do, go find it
19706 got dsc Stop(0)
19707 stop 0
19708 [8810] Got stop action so:false kr:false
19709 Feb 25 20:50:56.970 INFO request completed, latency_us: 403, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5b4e91f9-f101-4d31-9500-543c7f360999, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
197102025-02-25T20:50:56.971ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
197112025-02-25T20:50:56.971ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
197122025-02-25T20:50:56.971ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
197132025-02-25T20:50:56.971ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
19714 Feb 25 20:50:56.971 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4dd9dc4a-0929-4085-8e44-2cd216d4467b, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
19715 [8810] Exited with: ExitStatus(unix_wait_status(9))
19716 [8810][0] reports Exit
19717 Feb 25 20:51:00.974 INFO request completed, latency_us: 347, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9270a722-cf05-4779-a5ab-181be42cc096, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
19718 001/300 Flush
19719 002/300 Read block 1865 len 20480
19720 003/300 Read block 2792 len 24576
19721 004/300 Read block 4449 len 20480
19722 005/300 Flush
19723 006/300 Read block 2544 len 28672
19724 007/300 Write block 910 len 16384 data: 2 2 2 2
19725 008/300 Write block 1656 len 20480 data: 4 5 5 5 5
19726 009/300 Read block 659 len 8192
19727 010/300 Flush
19728 011/300 Write block 2688 len 28672 data: 5 4 4 3 3 3 3
19729 012/300 Write block 820 len 28672 data: 5 3 4 3 3 3 3
19730 013/300 Read block 3904 len 40960
19731 014/300 Write block 1558 len 40960 data: 5 6 7 7 8 4 4 3 3 3
19732 015/300 Read block 3113 len 8192
19733 016/300 Write block 3352 len 12288 data: 4 3 2
19734 017/300 Write block 493 len 8192 data: 2 2
19735 018/300 Read block 2890 len 4096
19736 019/300 Read block 3767 len 8192
19737 020/300 Read block 602 len 16384
19738 021/300 Read block 280 len 8192
19739 022/300 Flush
19740 023/300 Read block 4743 len 36864
19741 024/300 Read block 4 len 16384
19742 025/300 Read block 2462 len 4096
19743 026/300 Read block 2233 len 40960
19744 027/300 Read block 1348 len 40960
19745 028/300 Read block 2425 len 24576
19746 029/300 Write block 146 len 32768 data: 2 2 2 3 3 3 3 4
19747 030/300 Write block 3646 len 32768 data: 7 4 4 5 5 3 3 3
19748 031/300 Write block 932 len 20480 data: 4 4 5 3 4
19749 032/300 Read block 2317 len 24576
19750 033/300 Read block 4074 len 20480
19751 034/300 Read block 2184 len 24576
19752 035/300 Read block 3430 len 28672
19753 036/300 Write block 3403 len 4096 data: 3
19754 037/300 Write block 560 len 24576 data: 4 4 3 2 2 2
19755 038/300 Read block 1595 len 28672
19756 039/300 Flush
19757 040/300 Read block 1416 len 36864
19758 041/300 Write block 3174 len 4096 data: 4
19759 042/300 Read block 3508 len 32768
19760 043/300 Flush
19761 044/300 Read block 2758 len 8192
19762 045/300 Write block 1849 len 12288 data: 4 4 5
19763 046/300 Read block 4154 len 32768
19764 047/300 Read block 935 len 36864
19765 048/300 Read block 1585 len 24576
19766 049/300 Read block 4035 len 32768
19767 050/300 Read block 825 len 8192
19768 051/300 Write block 2270 len 24576 data: 2 3 5 5 5 5
19769 052/300 Read block 3727 len 24576
19770 053/300 Read block 1633 len 12288
19771 054/300 Read block 1575 len 28672
19772 055/300 Flush
19773 056/300 Read block 571 len 40960
19774 057/300 Write block 4332 len 28672 data: 3 3 3 2 2 2 2
19775 058/300 Read block 3196 len 12288
19776 059/300 Write block 1235 len 16384 data: 3 3 3 3
19777 060/300 Read block 1669 len 32768
19778 061/300 Read block 899 len 4096
19779 062/300 Read block 2937 len 12288
19780 063/300 Write block 3172 len 36864 data: 5 4 5 5 4 5 4 4 5
19781 064/300 Read block 1382 len 32768
19782 065/300 Read block 3520 len 28672
19783 066/300 Flush
19784 067/300 Write block 3894 len 32768 data: 3 3 3 3 3 4 3 4
19785 068/300 Write block 2606 len 32768 data: 6 6 6 5 5 5 4 3
19786 069/300 Flush
19787 070/300 Read block 4745 len 12288
19788 071/300 Read block 2062 len 20480
19789 072/300 Flush
19790 073/300 Flush
19791 074/300 Write block 3308 len 40960 data: 4 4 6 5 5 5 5 6 5 4
19792 075/300 Read block 298 len 16384
19793 076/300 Read block 412 len 32768
19794 077/300 Read block 2568 len 28672
19795 078/300 Read block 949 len 16384
19796 079/300 Read block 901 len 36864
19797 080/300 Write block 3686 len 8192 data: 4 4
19798 081/300 Write block 3869 len 16384 data: 4 3 6 5
19799 082/300 Read block 4124 len 16384
19800 083/300 Read block 119 len 16384
19801 084/300 Read block 532 len 8192
19802 085/300 Write block 1152 len 12288 data: 3 3 3
19803 086/300 Read block 2944 len 20480
19804 087/300 Write block 2416 len 8192 data: 3 3
19805 088/300 Write block 3604 len 36864 data: 4 4 3 3 3 4 3 5 6
19806 089/300 Read block 2787 len 28672
19807 090/300 Read block 4929 len 28672
19808 091/300 Write block 2451 len 20480 data: 9 7 8 7 7
19809 092/300 Flush
19810 093/300 Read block 1218 len 8192
19811 094/300 Read block 3491 len 20480
19812 095/300 Write block 4697 len 12288 data: 6 6 5
19813 096/300 Flush
19814 097/300 Flush
19815 098/300 Write block 2583 len 8192 data: 5 6
19816 099/300 Flush
19817 100/300 Write block 3189 len 36864 data: 3 3 3 3 3 3 4 4 4
19818 101/300 Write block 4080 len 24576 data: 4 4 5 4 4 3
19819 102/300 Read block 634 len 20480
19820 103/300 Write block 3103 len 32768 data: 6 6 7 6 5 3 4 4
19821 104/300 Read block 3468 len 28672
19822 105/300 Read block 3037 len 36864
19823 106/300 Read block 2784 len 40960
19824 107/300 Write block 630 len 24576 data: 3 3 4 3 3 3
19825 108/300 Read block 1388 len 28672
19826 109/300 Read block 3180 len 16384
19827 110/300 Read block 1897 len 12288
19828 111/300 Read block 1575 len 28672
19829 112/300 Write block 2777 len 36864 data: 2 2 2 2 3 3 4 3 2
19830 113/300 Write block 3711 len 20480 data: 5 5 4 4 4
19831 114/300 Flush
19832 115/300 Write block 532 len 16384 data: 6 5 4 3
19833 116/300 Read block 4246 len 32768
19834 117/300 Flush
19835 118/300 Read block 2218 len 32768
19836 119/300 Write block 3776 len 20480 data: 5 5 5 5 4
19837 120/300 Write block 2333 len 8192 data: 5 4
19838 121/300 Read block 4094 len 4096
19839 122/300 Write block 2 len 4096 data: 2
19840 123/300 Write block 4242 len 8192 data: 3 3
19841 124/300 Read block 4999 len 4096
19842 125/300 Write block 4439 len 16384 data: 4 5 5 4
19843 126/300 Write block 1071 len 20480 data: 3 4 6 5 4
19844 127/300 Write block 587 len 16384 data: 3 4 3 5
19845 128/300 Read block 4976 len 40960
19846 129/300 Read block 1638 len 20480
19847 130/300 Write block 1168 len 36864 data: 5 5 3 3 2 2 3 3 3
19848 131/300 Write block 2349 len 8192 data: 5 6
19849 132/300 Read block 4153 len 8192
19850 133/300 Write block 2887 len 20480 data: 2 2 5 6 3
19851 134/300 Read block 353 len 12288
19852 135/300 Read block 1252 len 32768
19853 136/300 Read block 2550 len 20480
19854 137/300 Flush
19855 138/300 Write block 266 len 24576 data: 4 4 6 7 6 5
19856 139/300 Read block 4540 len 4096
19857 140/300 Read block 2637 len 32768
19858 141/300 Flush
19859 142/300 Write block 3368 len 28672 data: 4 4 6 6 5 4 4
19860 143/300 Write block 1087 len 20480 data: 3 4 4 4 5
19861 144/300 Write block 4821 len 24576 data: 4 5 5 5 5 4
19862 145/300 Write block 1368 len 24576 data: 5 6 6 5 5 5
19863 146/300 Read block 2558 len 28672
19864 147/300 Read block 4437 len 32768
19865 148/300 Flush
19866 149/300 Read block 3875 len 8192
19867 150/300 Read block 1145 len 12288
19868 151/300 Write block 322 len 32768 data: 6 4 4 3 3 2 3 4
19869 152/300 Write block 671 len 24576 data: 4 5 6 8 10 7
19870 153/300 Read block 2580 len 24576
19871 154/300 Write block 589 len 20480 data: 4 6 5 4 5
19872 155/300 Flush
19873 156/300 Write block 2474 len 24576 data: 3 3 3 3 2 2
19874 157/300 Write block 4750 len 16384 data: 4 4 4 5
19875 158/300 Write block 3324 len 20480 data: 4 3 3 2 3
19876 159/300 Write block 369 len 4096 data: 4
19877 160/300 Flush
19878 161/300 Read block 2054 len 12288
19879 162/300 Read block 404 len 40960
19880 163/300 Read block 1429 len 8192
19881 164/300 Read block 3898 len 12288
19882 165/300 Read block 4037 len 40960
19883 166/300 Read block 756 len 40960
19884 167/300 Read block 2227 len 20480
19885 168/300 Write block 875 len 24576 data: 3 3 3 3 3 2
19886 169/300 Read block 313 len 28672
19887 170/300 Write block 2417 len 8192 data: 4 2
19888 171/300 Read block 4352 len 32768
19889 172/300 Write block 1651 len 32768 data: 3 2 3 4 4 5 6 6
19890 173/300 Read block 3735 len 24576
19891 174/300 Flush
19892 175/300 Read block 277 len 4096
19893 176/300 Read block 1306 len 36864
19894 177/300 Read block 4641 len 32768
19895 178/300 Read block 481 len 40960
19896 179/300 Read block 2683 len 40960
19897 180/300 Read block 800 len 4096
19898 181/300 Write block 1385 len 12288 data: 4 4 5
19899 182/300 Write block 2000 len 36864 data: 6 5 4 4 5 5 4 6 7
19900 183/300 Read block 1561 len 32768
19901 184/300 Read block 3636 len 32768
19902 185/300 Read block 4889 len 20480
19903 186/300 Read block 3756 len 8192
19904 187/300 Write block 506 len 4096 data: 3
19905 188/300 Read block 1524 len 4096
19906 189/300 Read block 1903 len 32768
19907 190/300 Read block 1640 len 16384
19908 191/300 Read block 3672 len 16384
19909 192/300 Write block 4903 len 12288 data: 4 4 3
19910 193/300 Read block 2179 len 12288
19911 194/300 Read block 535 len 20480
19912 195/300 Write block 1471 len 28672 data: 3 3 4 4 4 6 6
19913 196/300 Read block 1166 len 20480
19914 197/300 Read block 4712 len 40960
19915 198/300 Read block 365 len 20480
19916 199/300 Write block 3963 len 12288 data: 4 3 2
19917 200/300 Read block 4207 len 28672
19918 201/300 Write block 1129 len 8192 data: 4 3
19919 202/300 Write block 4407 len 16384 data: 4 4 3 3
19920 203/300 Write block 103 len 16384 data: 3 3 3 3
19921 204/300 Flush
19922 205/300 Read block 1494 len 40960
19923 206/300 Read block 770 len 36864
19924 207/300 Read block 1977 len 24576
19925 208/300 Read block 3649 len 16384
19926 209/300 Read block 988 len 20480
19927 210/300 Write block 4462 len 28672 data: 7 5 4 3 3 3 4
19928 211/300 Write block 1429 len 16384 data: 4 4 5 5
19929 212/300 Flush
19930 213/300 Read block 2407 len 8192
19931 214/300 Flush
19932 215/300 Read block 3867 len 4096
19933 216/300 Read block 3063 len 32768
19934 217/300 Read block 4495 len 32768
19935 218/300 Write block 1605 len 4096 data: 5
19936 219/300 Write block 614 len 24576 data: 4 4 5 6 6 6
19937 220/300 Read block 4173 len 4096
19938 221/300 Write block 3388 len 40960 data: 2 4 4 4 4 4 4 4 3 2
19939 222/300 Read block 3118 len 40960
19940 223/300 Write block 4227 len 16384 data: 5 5 6 5
19941 224/300 Write block 3144 len 28672 data: 5 5 3 3 3 4 4
19942 225/300 Write block 956 len 12288 data: 4 5 6
19943 226/300 Read block 4953 len 24576
19944 227/300 Flush
19945 228/300 Read block 1345 len 16384
19946 229/300 Read block 1448 len 28672
19947 230/300 Read block 2449 len 12288
19948 231/300 Write block 3139 len 36864 data: 4 4 3 4 4 6 6 4 4
19949 232/300 Read block 2905 len 16384
19950 233/300 Write block 4082 len 12288 data: 6 5 5
19951 234/300 Write block 1491 len 24576 data: 2 2 3 4 4 3
19952 235/300 Flush
19953 236/300 Read block 1055 len 32768
19954 237/300 Write block 2141 len 12288 data: 2 2 2
19955 238/300 Read block 2350 len 32768
19956 239/300 Flush
19957 240/300 Read block 1396 len 8192
19958 241/300 Write block 2992 len 16384 data: 7 6 4 4
19959 242/300 Read block 3405 len 28672
19960 243/300 Write block 1680 len 16384 data: 4 4 4 4
19961 244/300 Flush
19962 245/300 Write block 2024 len 8192 data: 3 2
19963 246/300 Write block 3469 len 32768 data: 3 3 2 2 2 2 2 3
19964 247/300 Write block 545 len 36864 data: 3 3 2 3 3 4 4 4 5
19965 248/300 Write block 1050 len 36864 data: 5 5 7 6 5 5 6 5 5
19966 249/300 Read block 4909 len 12288
19967 250/300 Flush
19968 251/300 Read block 3259 len 40960
19969 252/300 Write block 4223 len 4096 data: 4
19970 253/300 Write block 276 len 12288 data: 5 5 4
19971 254/300 Flush
19972 255/300 Write block 3349 len 40960 data: 6 6 4 5 4 3 2 2 2 3
19973 256/300 Read block 2255 len 32768
19974 257/300 Read block 1314 len 12288
19975 258/300 Read block 4875 len 12288
19976 259/300 Read block 1230 len 12288
19977 260/300 Read block 2628 len 28672
19978 261/300 Write block 1501 len 28672 data: 2 2 2 3 5 6 5
19979 262/300 Read block 4449 len 28672
19980 263/300 Read block 1303 len 28672
19981 264/300 Write block 4902 len 4096 data: 4
19982 265/300 Flush
19983 266/300 Write block 2311 len 28672 data: 4 4 4 5 4 4 5
19984 267/300 Read block 3858 len 40960
19985 268/300 Write block 2007 len 24576 data: 7 8 9 9 6 5
19986 269/300 Write block 894 len 20480 data: 2 2 2 3 3
19987 270/300 Write block 2606 len 36864 data: 7 7 7 6 6 6 5 4 3
19988 271/300 Read block 449 len 20480
19989 272/300 Read block 4109 len 40960
19990 273/300 Write block 1566 len 12288 data: 4 4 3
19991 274/300 Write block 745 len 24576 data: 3 3 3 2 2 2
19992 275/300 Write block 3785 len 28672 data: 4 5 4 3 4 3 4
19993 276/300 Read block 377 len 32768
19994 277/300 Read block 3125 len 12288
19995 278/300 Read block 4172 len 36864
19996 279/300 Read block 2580 len 8192
19997 280/300 Write block 4815 len 28672 data: 3 2 4 4 4 4 5
19998 281/300 Write block 3835 len 20480 data: 4 5 5 4 4
19999 282/300 Write block 2507 len 8192 data: 4 4
20000 283/300 Flush
20001 284/300 Read block 2872 len 8192
20002 285/300 Read block 3244 len 32768
20003 286/300 Write block 2485 len 28672 data: 3 3 4 5 5 4 4
20004 287/300 Flush
20005 288/300 Write block 1521 len 40960 data: 4 5 5 6 5 4 5 6 7 7
20006 289/300 Read block 4413 len 40960
20007 290/300 Flush
20008 291/300 Read block 886 len 40960
20009 292/300 Read block 2656 len 16384
20010 293/300 Write block 612 len 8192 data: 3 4
20011 294/300 Write block 2633 len 16384 data: 3 2 2 2
20012 295/300 Read block 1109 len 4096
20013 296/300 Read block 165 len 16384
20014 297/300 Read block 3341 len 40960
20015 298/300 Read block 3577 len 36864
20016 299/300 Flush
20017 300/300 Write block 2255 len 12288 data: 7 8 6
20018 301/300 Read block 1769 len 20480
20019 Main task has work to do, go find it
20020 got dsc Start(0)
20021 start 0
20022 Feb 25 20:51:01.227 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8ced95d9-bc22-4131-a330-9f2a297d6e5e, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
20023 Replay: started 0, returned:Ok(())
20024 ----------------------------------------------------------------
20025 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20026 States: Offline Active Active
20027 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20028 6132 Acked Flush 0 Sent Done Done false
20029 6133 Acked Read 5 Sent Done Done false
20030 6134 Acked Read 6 Sent Done Done false
20031 6135 Acked Read 5 Sent Done Done false
20032 6136 Acked Flush 0 Sent Done Done false
20033 6137 Acked Read 7 Sent Done Done false
20034 6138 Acked Write 4 Sent Done Done false
20035 6139 Acked Write 5 Sent Done Done false
20036 6140 Acked Read 2 Sent Done Done false
20037 6141 Acked Flush 0 Sent Done Done false
20038 6142 Acked Write 7 Sent Done Done false
20039 6143 Acked Write 7 Sent Done Done false
20040 6144 Acked Read 10 Sent Done Done false
20041 6145 Acked Write 10 Sent Done Done false
20042 6146 Acked Read 2 Sent Done Done false
20043 6147 Acked Write 3 Sent Done Done false
20044 6148 Acked Write 2 Sent Done Done false
20045 6149 Acked Read 1 Sent Done Done false
20046 6150 Acked Read 2 Sent Done Done false
20047 6151 Acked Read 4 Sent Done Done false
20048 6152 Acked Read 2 Sent Done Done false
20049 6153 Acked Flush 0 Sent Done Done false
20050 6154 Acked Read 9 Sent Done Done false
20051 6155 Acked Read 4 Sent Done Done false
20052 6156 Acked Read 1 Sent Done Done false
20053 6157 Acked Read 10 Sent Done Done false
20054 6158 Acked Read 10 Sent Done Done false
20055 6159 Acked Read 6 Sent Done Done false
20056 6160 Acked Write 8 Sent Done Done false
20057 6161 Acked Write 8 Sent Done Done false
20058 6162 Acked Write 5 Sent Done Done false
20059 6163 Acked Read 6 Sent Done Done false
20060 6164 Acked Read 5 Sent Done Done false
20061 6165 Acked Read 6 Sent Done Done false
20062 6166 Acked Read 7 Sent Done Done false
20063 6167 Acked Write 1 Sent Done Done false
20064 6168 Acked Write 6 Sent Done Done false
20065 6169 Acked Read 7 Sent Done Done false
20066 6170 Acked Flush 0 Sent Done Done false
20067 6171 Acked Read 9 Sent Done Done false
20068 6172 Acked Write 1 Sent Done Done false
20069 6173 Acked Read 8 Sent Done Done false
20070 6174 Acked Flush 0 Sent Done Done false
20071 6175 Acked Read 2 Sent Done Done false
20072 6176 Acked Write 3 Sent Done Done false
20073 6177 Acked Read 8 Sent Done Done false
20074 6178 Acked Read 9 Sent Done Done false
20075 6179 Acked Read 6 Sent Done Done false
20076 6180 Acked Read 8 Sent Done Done false
20077 6181 Acked Read 2 Sent Done Done false
20078 6182 Acked Write 6 Sent Done Done false
20079 6183 Acked Read 6 Sent Done Done false
20080 6184 Acked Read 3 Sent Done Done false
20081 6185 Acked Read 7 Sent Done Done false
20082 6186 Acked Flush 0 Sent Done Done false
20083 6187 Acked Read 10 Sent Done Done false
20084 6188 Acked Write 7 Sent Done Done false
20085 6189 Acked Read 3 Sent Done Done false
20086 6190 Acked Write 4 Sent Done Done false
20087 6191 Acked Read 8 Sent Done Done false
20088 6192 Acked Read 1 Sent Done Done false
20089 6193 Acked Read 3 Sent Done Done false
20090 6194 Acked Write 9 Sent Done Done false
20091 6195 Acked Read 8 Sent Done Done false
20092 6196 Acked Read 7 Sent Done Done false
20093 6197 Acked Flush 0 Sent Done Done false
20094 6198 Acked Write 8 Sent Done Done false
20095 6199 Acked Write 8 Sent Done Done false
20096 6200 Acked Flush 0 Sent Done Done false
20097 6201 Acked Read 3 Sent Done Done false
20098 6202 Acked Read 5 Sent Done Done false
20099 6203 Acked Flush 0 Sent Done Done false
20100 6204 Acked Flush 0 Sent Done Done false
20101 6205 Acked Write 10 Sent Done Done false
20102 6206 Acked Read 4 Sent Done Done false
20103 6207 Acked Read 8 Sent Done Done false
20104 6208 Acked Read 7 Sent Done Done false
20105 6209 Acked Read 4 Sent Done Done false
20106 6210 Acked Read 9 Sent Done Done false
20107 6211 Acked Write 2 Sent Done Done false
20108 6212 Acked Write 4 Sent Done Done false
20109 6213 Acked Read 4 Sent Done Done false
20110 6214 Acked Read 4 Sent Done Done false
20111 6215 Acked Read 2 Sent Done Done false
20112 6216 Acked Write 3 Sent Done Done false
20113 6217 Acked Read 5 Sent Done Done false
20114 6218 Acked Write 2 Sent Done Done false
20115 6219 Acked Write 9 Sent Done Done false
20116 6220 Acked Read 7 Sent Done Done false
20117 6221 Acked Read 7 Sent Done Done false
20118 6222 Acked Write 5 Sent Done Done false
20119 6223 Acked Flush 0 Sent Done Done false
20120 6224 Acked Read 2 Sent Done Done false
20121 6225 Acked Read 5 Sent Done Done false
20122 6226 Acked Write 3 Sent Done Done false
20123 6227 Acked Flush 0 Sent Done Done false
20124 6228 Acked Flush 0 Sent Done Done false
20125 6229 Acked Write 2 Sent Done Done false
20126 6230 Acked Flush 0 Sent Done Done false
20127 6231 Acked Write 9 Sent Done Done false
20128 6232 Acked Write 6 Sent Done Done false
20129 6233 Acked Read 5 Sent Done Done false
20130 6234 Acked Write 8 Sent Done Done false
20131 6235 Acked Read 7 Sent Done Done false
20132 6236 Acked Read 9 Sent Done Done false
20133 6237 Acked Read 10 Sent Done Done false
20134 6238 Acked Write 6 Sent Done Done false
20135 6239 Acked Read 7 Sent Done Done false
20136 6240 Acked Read 4 Sent Done Done false
20137 6241 Acked Read 3 Sent Done Done false
20138 6242 Acked Read 7 Sent Done Done false
20139 6243 Acked Write 9 Sent Done Done false
20140 6244 Acked Write 5 Sent Done Done false
20141 6245 Acked Flush 0 Sent Done Done false
20142 6246 Acked Write 4 Sent Done Done false
20143 6247 Acked Read 8 Sent Done Done false
20144 6248 Acked Flush 0 Sent Done Done false
20145 6249 Acked Read 8 Sent Done Done false
20146 6250 Acked Write 5 Sent Done Done false
20147 6251 Acked Write 2 Sent Done Done false
20148 6252 Acked Read 1 Sent Done Done false
20149 6253 Acked Write 1 Sent Done Done false
20150 6254 Acked Write 2 Sent Done Done false
20151 6255 Acked Read 1 Sent Done Done false
20152 6256 Acked Write 4 Sent Done Done false
20153 6257 Acked Write 5 Sent Done Done false
20154 6258 Acked Write 4 Sent Done Done false
20155 6259 Acked Read 10 Sent Done Done false
20156 6260 Acked Read 5 Sent Done Done false
20157 6261 Acked Write 9 Sent Done Done false
20158 6262 Acked Write 2 Sent Done Done false
20159 6263 Acked Read 2 Sent Done Done false
20160 6264 Acked Write 5 Sent Done Done false
20161 6265 Acked Read 3 Sent Done Done false
20162 6266 Acked Read 8 Sent Done Done false
20163 6267 Acked Read 5 Sent Done Done false
20164 6268 Acked Flush 0 Sent Done Done false
20165 6269 Acked Write 6 Sent Done Done false
20166 6270 Acked Read 1 Sent Done Done false
20167 6271 Acked Read 8 Sent Done Done false
20168 6272 Acked Flush 0 Sent Done Done false
20169 6273 Acked Write 7 Sent Done Done false
20170 6274 Acked Write 5 Sent Done Done false
20171 6275 Acked Write 6 Sent Done Done false
20172 6276 Acked Write 6 Sent Done Done false
20173 6277 Acked Read 7 Sent Done Done false
20174 6278 Acked Read 8 Sent Done Done false
20175 6279 Acked Flush 0 Sent Done Done false
20176 6280 Acked Read 2 Sent Done Done false
20177 6281 Acked Read 3 Sent Done Done false
20178 6282 Acked Write 8 Sent Done Done false
20179 6283 Acked Write 6 Sent Done Done false
20180 6284 Acked Read 6 Sent Done Done false
20181 6285 Acked Write 5 Sent Done Done false
20182 6286 Acked Flush 0 Sent Done Done false
20183 6287 Acked Write 6 Sent Done Done false
20184 6288 Acked Write 4 Sent Done Done false
20185 6289 Acked Write 5 Sent Done Done false
20186 6290 Acked Write 1 Sent Done Done false
20187 6291 Acked Flush 0 Sent Done Done false
20188 6292 Acked Read 3 Sent Done Done false
20189 6293 Acked Read 10 Sent Done Done false
20190 6294 Acked Read 2 Sent Done Done false
20191 6295 Acked Read 3 Sent Done Done false
20192 6296 Acked Read 10 Sent Done Done false
20193 6297 Acked Read 10 Sent Done Done false
20194 6298 Acked Read 5 Sent Done Done false
20195 6299 Acked Write 6 Sent Done Done false
20196 6300 Acked Read 7 Sent Done Done false
20197 6301 Acked Write 2 Sent Done Done false
20198 6302 Acked Read 8 Sent Done Done false
20199 6303 Acked Write 8 Sent Done Done false
20200 6304 Acked Read 6 Sent Done Done false
20201 6305 Acked Flush 0 Sent Done Done false
20202 6306 Acked Read 1 Sent Done Done false
20203 6307 Acked Read 9 Sent Done Done false
20204 6308 Acked Read 8 Sent Done Done false
20205 6309 Acked Read 10 Sent Done Done false
20206 6310 Acked Read 10 Sent Done Done false
20207 6311 Acked Read 1 Sent Done Done false
20208 6312 Acked Write 3 Sent Done Done false
20209 6313 Acked Write 9 Sent Done Done false
20210 6314 Acked Read 8 Sent Done Done false
20211 6315 Acked Read 8 Sent Done Done false
20212 6316 Acked Read 5 Sent Done Done false
20213 6317 Acked Read 2 Sent Done Done false
20214 6318 Acked Write 1 Sent Done Done false
20215 6319 Acked Read 1 Sent Done Done false
20216 6320 Acked Read 8 Sent Done Done false
20217 6321 Acked Read 4 Sent Done Done false
20218 6322 Acked Read 4 Sent Done Done false
20219 6323 Acked Write 3 Sent Done Done false
20220 6324 Acked Read 3 Sent Done Done false
20221 6325 Acked Read 5 Sent Done Done false
20222 6326 Acked Write 7 Sent Done Done false
20223 6327 Acked Read 5 Sent Done Done false
20224 6328 Acked Read 10 Sent Done Done false
20225 6329 Acked Read 5 Sent Done Done false
20226 6330 Acked Write 3 Sent Done Done false
20227 6331 Acked Read 7 Sent Done Done false
20228 6332 Acked Write 2 Sent Done Done false
20229 6333 Acked Write 4 Sent Done Done false
20230 6334 Acked Write 4 Sent Done Done false
20231 6335 Acked Flush 0 Sent Done Done false
20232 6336 Acked Read 10 Sent Done Done false
20233 6337 Acked Read 9 Sent Done Done false
20234 6338 Acked Read 6 Sent Done Done false
20235 6339 Acked Read 4 Sent Done Done false
20236 6340 Acked Read 5 Sent Done Done false
20237 6341 Acked Write 7 Sent Done Done false
20238 6342 Acked Write 4 Sent Done Done false
20239 6343 Acked Flush 0 Sent Done Done false
20240 6344 Acked Read 2 Sent Done Done false
20241 6345 Acked Flush 0 Sent Done Done false
20242 6346 Acked Read 1 Sent Done Done false
20243 6347 Acked Read 8 Sent Done Done false
20244 6348 Acked Read 8 Sent Done Done false
20245 6349 Acked Write 1 Sent Done Done false
20246 6350 Acked Write 6 Sent Done Done false
20247 6351 Acked Read 1 Sent Done Done false
20248 6352 Acked Write 10 Sent Done Done false
20249 6353 Acked Read 10 Sent Done Done false
20250 6354 Acked Write 4 Sent Done Done false
20251 6355 Acked Write 7 Sent Done Done false
20252 6356 Acked Write 3 Sent Done Done false
20253 6357 Acked Read 6 Sent Done Done false
20254 6358 Acked Flush 0 Sent Done Done false
20255 6359 Acked Read 4 Sent Done Done false
20256 6360 Acked Read 7 Sent Done Done false
20257 6361 Acked Read 3 Sent Done Done false
20258 6362 Acked Write 9 Sent Done Done false
20259 6363 Acked Read 4 Sent Done Done false
20260 6364 Acked Write 3 Sent Done Done false
20261 6365 Acked Write 6 Sent Done Done false
20262 6366 Acked Flush 0 Sent Done Done false
20263 6367 Acked Read 8 Sent Done Done false
20264 6368 Acked Write 3 Sent Done Done false
20265 6369 Acked Read 8 Sent Done Done false
20266 6370 Acked Flush 0 Sent Done Done false
20267 6371 Acked Read 2 Sent Done Done false
20268 6372 Acked Write 4 Sent Done Done false
20269 6373 Acked Read 7 Sent Done Done false
20270 6374 Acked Write 4 Sent Done Done false
20271 6375 Acked Flush 0 Sent Done Done false
20272 6376 Acked Write 2 Sent Done Done false
20273 6377 Acked Write 8 Sent Done Done false
20274 6378 Acked Write 9 Sent Done Done false
20275 6379 Acked Write 9 Sent Done Done false
20276 6380 Acked Read 3 Sent Done Done false
20277 6381 Acked Flush 0 Sent Done Done false
20278 6382 Acked Read 10 Sent Done Done false
20279 6383 Acked Write 1 Sent Done Done false
20280 6384 Acked Write 3 Sent Done Done false
20281 6385 Acked Flush 0 Sent Done Done false
20282 6386 Acked Write 10 Sent Done Done false
20283 6387 Acked Read 8 Sent Done Done false
20284 6388 Acked Read 3 Sent Done Done false
20285 6389 Acked Read 3 Sent Done Done false
20286 6390 Acked Read 3 Sent Done Done false
20287 6391 Acked Read 7 Sent Done Done false
20288 6392 Acked Write 7 Sent Done Done false
20289 6393 Acked Read 7 Sent Done Done false
20290 6394 Acked Read 7 Sent Done Done false
20291 6395 Acked Write 1 Sent Done Done false
20292 6396 Acked Flush 0 Sent Done Done false
20293 6397 Acked Write 7 Sent Done Done false
20294 6398 Acked Read 10 Sent Done Done false
20295 6399 Acked Write 6 Sent Done Done false
20296 6400 Acked Write 5 Sent Done Done false
20297 6401 Acked Write 9 Sent Done Done false
20298 6402 Acked Read 5 Sent Done Done false
20299 6403 Acked Read 10 Sent Done Done false
20300 6404 Acked Write 3 Sent Done Done false
20301 6405 Acked Write 6 Sent Done Done false
20302 6406 Acked Write 7 Sent Done Done false
20303 6407 Acked Read 8 Sent Done Done false
20304 6408 Acked Read 3 Sent Done Done false
20305 6409 Acked Read 9 Sent Done Done false
20306 6410 Acked Read 2 Sent Done Done false
20307 6411 Acked Write 7 Sent Done Done false
20308 6412 Acked Write 5 Sent Done Done false
20309 6413 Acked Write 2 Sent Done Done false
20310 6414 Acked Flush 0 Sent Done Done false
20311 6415 Acked Read 2 Sent Done Done false
20312 6416 Acked Read 8 Sent Done Done false
20313 6417 Acked Write 7 Sent Done Done false
20314 6418 Acked Flush 0 Sent Done Done false
20315 6419 Acked Write 10 Sent Done Done false
20316 6420 Acked Read 10 Sent Done Done false
20317 6421 Acked Flush 0 Sent Done Done false
20318 6422 Acked Read 10 Sent Done Done false
20319 6423 Acked Read 4 Sent Done Done false
20320 6424 Acked Write 2 Sent Done Done false
20321 6425 Acked Write 4 Sent Done Done false
20322 6426 Acked Read 1 Sent Done Done false
20323 6427 Acked Read 4 Sent Done Done false
20324 6428 Acked Read 10 Sent Done Done false
20325 6429 Acked Read 9 Sent Done Done false
20326 6430 Acked Flush 0 Sent Done Done false
20327 6431 Acked Write 3 Sent Done Done false
20328 6432 Acked Read 5 Sent Done Done false
20329 STATES DS:0 DS:1 DS:2 TOTAL
20330 Sent 301 0 0 301
20331 Done 0 301 301 602
20332 Skipped 0 0 0 0
20333 Error 0 0 0 0
20334 Last Flush: Some(JobId(6131)) Some(JobId(6430)) Some(JobId(6430))
20335 Downstairs last five completed: 6131 6130 6129 6128 6127
20336 Upstairs last five completed: 6432 6431 6430 6429 6428
20337 CLIENT: Up:0 ds:301 act:2
203382025-02-25T20:51:01.729ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
203392025-02-25T20:51:02.481ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
203402025-02-25T20:51:03.482ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
203412025-02-25T20:51:04.483ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
20342 [8810] got start action
20343 [8810] I am going to restart
20344 Starting downstairs at port 8810
20345 Make output file at "/tmp/dsc/downstairs-8810.txt"
20346 [8810][0] reports Starting
20347 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
20348 [8810][0] reports Running
20349 ----------------------------------------------------------------
20350 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20351 States: Offline Active Active
20352 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20353 6132 Acked Flush 0 Sent Done Done false
20354 6133 Acked Read 5 Sent Done Done false
20355 6134 Acked Read 6 Sent Done Done false
20356 6135 Acked Read 5 Sent Done Done false
20357 6136 Acked Flush 0 Sent Done Done false
20358 6137 Acked Read 7 Sent Done Done false
20359 6138 Acked Write 4 Sent Done Done false
20360 6139 Acked Write 5 Sent Done Done false
20361 6140 Acked Read 2 Sent Done Done false
20362 6141 Acked Flush 0 Sent Done Done false
20363 6142 Acked Write 7 Sent Done Done false
20364 6143 Acked Write 7 Sent Done Done false
20365 6144 Acked Read 10 Sent Done Done false
20366 6145 Acked Write 10 Sent Done Done false
20367 6146 Acked Read 2 Sent Done Done false
20368 6147 Acked Write 3 Sent Done Done false
20369 6148 Acked Write 2 Sent Done Done false
20370 6149 Acked Read 1 Sent Done Done false
20371 6150 Acked Read 2 Sent Done Done false
20372 6151 Acked Read 4 Sent Done Done false
20373 6152 Acked Read 2 Sent Done Done false
20374 6153 Acked Flush 0 Sent Done Done false
20375 6154 Acked Read 9 Sent Done Done false
20376 6155 Acked Read 4 Sent Done Done false
20377 6156 Acked Read 1 Sent Done Done false
20378 6157 Acked Read 10 Sent Done Done false
20379 6158 Acked Read 10 Sent Done Done false
20380 6159 Acked Read 6 Sent Done Done false
20381 6160 Acked Write 8 Sent Done Done false
20382 6161 Acked Write 8 Sent Done Done false
20383 6162 Acked Write 5 Sent Done Done false
20384 6163 Acked Read 6 Sent Done Done false
20385 6164 Acked Read 5 Sent Done Done false
20386 6165 Acked Read 6 Sent Done Done false
20387 6166 Acked Read 7 Sent Done Done false
20388 6167 Acked Write 1 Sent Done Done false
20389 6168 Acked Write 6 Sent Done Done false
20390 6169 Acked Read 7 Sent Done Done false
20391 6170 Acked Flush 0 Sent Done Done false
20392 6171 Acked Read 9 Sent Done Done false
20393 6172 Acked Write 1 Sent Done Done false
20394 6173 Acked Read 8 Sent Done Done false
20395 6174 Acked Flush 0 Sent Done Done false
20396 6175 Acked Read 2 Sent Done Done false
20397 6176 Acked Write 3 Sent Done Done false
20398 6177 Acked Read 8 Sent Done Done false
20399 6178 Acked Read 9 Sent Done Done false
20400 6179 Acked Read 6 Sent Done Done false
20401 6180 Acked Read 8 Sent Done Done false
20402 6181 Acked Read 2 Sent Done Done false
20403 6182 Acked Write 6 Sent Done Done false
20404 6183 Acked Read 6 Sent Done Done false
20405 6184 Acked Read 3 Sent Done Done false
20406 6185 Acked Read 7 Sent Done Done false
20407 6186 Acked Flush 0 Sent Done Done false
20408 6187 Acked Read 10 Sent Done Done false
20409 6188 Acked Write 7 Sent Done Done false
20410 6189 Acked Read 3 Sent Done Done false
20411 6190 Acked Write 4 Sent Done Done false
20412 6191 Acked Read 8 Sent Done Done false
20413 6192 Acked Read 1 Sent Done Done false
20414 6193 Acked Read 3 Sent Done Done false
20415 6194 Acked Write 9 Sent Done Done false
20416 6195 Acked Read 8 Sent Done Done false
20417 6196 Acked Read 7 Sent Done Done false
20418 6197 Acked Flush 0 Sent Done Done false
20419 6198 Acked Write 8 Sent Done Done false
20420 6199 Acked Write 8 Sent Done Done false
20421 6200 Acked Flush 0 Sent Done Done false
20422 6201 Acked Read 3 Sent Done Done false
20423 6202 Acked Read 5 Sent Done Done false
20424 6203 Acked Flush 0 Sent Done Done false
20425 6204 Acked Flush 0 Sent Done Done false
20426 6205 Acked Write 10 Sent Done Done false
20427 6206 Acked Read 4 Sent Done Done false
20428 6207 Acked Read 8 Sent Done Done false
20429 6208 Acked Read 7 Sent Done Done false
20430 6209 Acked Read 4 Sent Done Done false
20431 6210 Acked Read 9 Sent Done Done false
20432 6211 Acked Write 2 Sent Done Done false
20433 6212 Acked Write 4 Sent Done Done false
20434 6213 Acked Read 4 Sent Done Done false
20435 6214 Acked Read 4 Sent Done Done false
20436 6215 Acked Read 2 Sent Done Done false
20437 6216 Acked Write 3 Sent Done Done false
20438 6217 Acked Read 5 Sent Done Done false
20439 6218 Acked Write 2 Sent Done Done false
20440 6219 Acked Write 9 Sent Done Done false
20441 6220 Acked Read 7 Sent Done Done false
20442 6221 Acked Read 7 Sent Done Done false
20443 6222 Acked Write 5 Sent Done Done false
20444 6223 Acked Flush 0 Sent Done Done false
20445 6224 Acked Read 2 Sent Done Done false
20446 6225 Acked Read 5 Sent Done Done false
20447 6226 Acked Write 3 Sent Done Done false
20448 6227 Acked Flush 0 Sent Done Done false
20449 6228 Acked Flush 0 Sent Done Done false
20450 6229 Acked Write 2 Sent Done Done false
20451 6230 Acked Flush 0 Sent Done Done false
20452 6231 Acked Write 9 Sent Done Done false
20453 6232 Acked Write 6 Sent Done Done false
20454 6233 Acked Read 5 Sent Done Done false
20455 6234 Acked Write 8 Sent Done Done false
20456 6235 Acked Read 7 Sent Done Done false
20457 6236 Acked Read 9 Sent Done Done false
20458 6237 Acked Read 10 Sent Done Done false
20459 6238 Acked Write 6 Sent Done Done false
20460 6239 Acked Read 7 Sent Done Done false
20461 6240 Acked Read 4 Sent Done Done false
20462 6241 Acked Read 3 Sent Done Done false
20463 6242 Acked Read 7 Sent Done Done false
20464 6243 Acked Write 9 Sent Done Done false
20465 6244 Acked Write 5 Sent Done Done false
20466 6245 Acked Flush 0 Sent Done Done false
20467 6246 Acked Write 4 Sent Done Done false
20468 6247 Acked Read 8 Sent Done Done false
20469 6248 Acked Flush 0 Sent Done Done false
20470 6249 Acked Read 8 Sent Done Done false
20471 6250 Acked Write 5 Sent Done Done false
20472 6251 Acked Write 2 Sent Done Done false
20473 6252 Acked Read 1 Sent Done Done false
20474 6253 Acked Write 1 Sent Done Done false
20475 6254 Acked Write 2 Sent Done Done false
20476 6255 Acked Read 1 Sent Done Done false
20477 6256 Acked Write 4 Sent Done Done false
20478 6257 Acked Write 5 Sent Done Done false
20479 6258 Acked Write 4 Sent Done Done false
20480 6259 Acked Read 10 Sent Done Done false
20481 6260 Acked Read 5 Sent Done Done false
20482 6261 Acked Write 9 Sent Done Done false
20483 6262 Acked Write 2 Sent Done Done false
20484 6263 Acked Read 2 Sent Done Done false
20485 6264 Acked Write 5 Sent Done Done false
20486 6265 Acked Read 3 Sent Done Done false
20487 6266 Acked Read 8 Sent Done Done false
20488 6267 Acked Read 5 Sent Done Done false
20489 6268 Acked Flush 0 Sent Done Done false
20490 6269 Acked Write 6 Sent Done Done false
20491 6270 Acked Read 1 Sent Done Done false
20492 6271 Acked Read 8 Sent Done Done false
20493 6272 Acked Flush 0 Sent Done Done false
20494 6273 Acked Write 7 Sent Done Done false
20495 6274 Acked Write 5 Sent Done Done false
20496 6275 Acked Write 6 Sent Done Done false
20497 6276 Acked Write 6 Sent Done Done false
20498 6277 Acked Read 7 Sent Done Done false
20499 6278 Acked Read 8 Sent Done Done false
20500 6279 Acked Flush 0 Sent Done Done false
20501 6280 Acked Read 2 Sent Done Done false
20502 6281 Acked Read 3 Sent Done Done false
20503 6282 Acked Write 8 Sent Done Done false
20504 6283 Acked Write 6 Sent Done Done false
20505 6284 Acked Read 6 Sent Done Done false
20506 6285 Acked Write 5 Sent Done Done false
20507 6286 Acked Flush 0 Sent Done Done false
20508 6287 Acked Write 6 Sent Done Done false
20509 6288 Acked Write 4 Sent Done Done false
20510 6289 Acked Write 5 Sent Done Done false
20511 6290 Acked Write 1 Sent Done Done false
20512 6291 Acked Flush 0 Sent Done Done false
20513 6292 Acked Read 3 Sent Done Done false
20514 6293 Acked Read 10 Sent Done Done false
20515 6294 Acked Read 2 Sent Done Done false
20516 6295 Acked Read 3 Sent Done Done false
20517 6296 Acked Read 10 Sent Done Done false
20518 6297 Acked Read 10 Sent Done Done false
20519 6298 Acked Read 5 Sent Done Done false
20520 6299 Acked Write 6 Sent Done Done false
20521 6300 Acked Read 7 Sent Done Done false
20522 6301 Acked Write 2 Sent Done Done false
20523 6302 Acked Read 8 Sent Done Done false
20524 6303 Acked Write 8 Sent Done Done false
20525 6304 Acked Read 6 Sent Done Done false
20526 6305 Acked Flush 0 Sent Done Done false
20527 6306 Acked Read 1 Sent Done Done false
20528 6307 Acked Read 9 Sent Done Done false
20529 6308 Acked Read 8 Sent Done Done false
20530 6309 Acked Read 10 Sent Done Done false
20531 6310 Acked Read 10 Sent Done Done false
20532 6311 Acked Read 1 Sent Done Done false
20533 6312 Acked Write 3 Sent Done Done false
20534 6313 Acked Write 9 Sent Done Done false
20535 6314 Acked Read 8 Sent Done Done false
20536 6315 Acked Read 8 Sent Done Done false
20537 6316 Acked Read 5 Sent Done Done false
20538 6317 Acked Read 2 Sent Done Done false
20539 6318 Acked Write 1 Sent Done Done false
20540 6319 Acked Read 1 Sent Done Done false
20541 6320 Acked Read 8 Sent Done Done false
20542 6321 Acked Read 4 Sent Done Done false
20543 6322 Acked Read 4 Sent Done Done false
20544 6323 Acked Write 3 Sent Done Done false
20545 6324 Acked Read 3 Sent Done Done false
20546 6325 Acked Read 5 Sent Done Done false
20547 6326 Acked Write 7 Sent Done Done false
20548 6327 Acked Read 5 Sent Done Done false
20549 6328 Acked Read 10 Sent Done Done false
20550 6329 Acked Read 5 Sent Done Done false
20551 6330 Acked Write 3 Sent Done Done false
20552 6331 Acked Read 7 Sent Done Done false
20553 6332 Acked Write 2 Sent Done Done false
20554 6333 Acked Write 4 Sent Done Done false
20555 6334 Acked Write 4 Sent Done Done false
20556 6335 Acked Flush 0 Sent Done Done false
20557 6336 Acked Read 10 Sent Done Done false
20558 6337 Acked Read 9 Sent Done Done false
20559 6338 Acked Read 6 Sent Done Done false
20560 6339 Acked Read 4 Sent Done Done false
20561 6340 Acked Read 5 Sent Done Done false
20562 6341 Acked Write 7 Sent Done Done false
20563 6342 Acked Write 4 Sent Done Done false
20564 6343 Acked Flush 0 Sent Done Done false
20565 6344 Acked Read 2 Sent Done Done false
20566 6345 Acked Flush 0 Sent Done Done false
20567 6346 Acked Read 1 Sent Done Done false
20568 6347 Acked Read 8 Sent Done Done false
20569 6348 Acked Read 8 Sent Done Done false
20570 6349 Acked Write 1 Sent Done Done false
20571 6350 Acked Write 6 Sent Done Done false
20572 6351 Acked Read 1 Sent Done Done false
20573 6352 Acked Write 10 Sent Done Done false
20574 6353 Acked Read 10 Sent Done Done false
20575 6354 Acked Write 4 Sent Done Done false
20576 6355 Acked Write 7 Sent Done Done false
20577 6356 Acked Write 3 Sent Done Done false
20578 6357 Acked Read 6 Sent Done Done false
20579 6358 Acked Flush 0 Sent Done Done false
20580 6359 Acked Read 4 Sent Done Done false
20581 6360 Acked Read 7 Sent Done Done false
20582 6361 Acked Read 3 Sent Done Done false
20583 6362 Acked Write 9 Sent Done Done false
20584 6363 Acked Read 4 Sent Done Done false
20585 6364 Acked Write 3 Sent Done Done false
20586 6365 Acked Write 6 Sent Done Done false
20587 6366 Acked Flush 0 Sent Done Done false
20588 6367 Acked Read 8 Sent Done Done false
20589 6368 Acked Write 3 Sent Done Done false
20590 6369 Acked Read 8 Sent Done Done false
20591 6370 Acked Flush 0 Sent Done Done false
20592 6371 Acked Read 2 Sent Done Done false
20593 6372 Acked Write 4 Sent Done Done false
20594 6373 Acked Read 7 Sent Done Done false
20595 6374 Acked Write 4 Sent Done Done false
20596 6375 Acked Flush 0 Sent Done Done false
20597 6376 Acked Write 2 Sent Done Done false
20598 6377 Acked Write 8 Sent Done Done false
20599 6378 Acked Write 9 Sent Done Done false
20600 6379 Acked Write 9 Sent Done Done false
20601 6380 Acked Read 3 Sent Done Done false
20602 6381 Acked Flush 0 Sent Done Done false
20603 6382 Acked Read 10 Sent Done Done false
20604 6383 Acked Write 1 Sent Done Done false
20605 6384 Acked Write 3 Sent Done Done false
20606 6385 Acked Flush 0 Sent Done Done false
20607 6386 Acked Write 10 Sent Done Done false
20608 6387 Acked Read 8 Sent Done Done false
20609 6388 Acked Read 3 Sent Done Done false
20610 6389 Acked Read 3 Sent Done Done false
20611 6390 Acked Read 3 Sent Done Done false
20612 6391 Acked Read 7 Sent Done Done false
20613 6392 Acked Write 7 Sent Done Done false
20614 6393 Acked Read 7 Sent Done Done false
20615 6394 Acked Read 7 Sent Done Done false
20616 6395 Acked Write 1 Sent Done Done false
20617 6396 Acked Flush 0 Sent Done Done false
20618 6397 Acked Write 7 Sent Done Done false
20619 6398 Acked Read 10 Sent Done Done false
20620 6399 Acked Write 6 Sent Done Done false
20621 6400 Acked Write 5 Sent Done Done false
20622 6401 Acked Write 9 Sent Done Done false
20623 6402 Acked Read 5 Sent Done Done false
20624 6403 Acked Read 10 Sent Done Done false
20625 6404 Acked Write 3 Sent Done Done false
20626 6405 Acked Write 6 Sent Done Done false
20627 6406 Acked Write 7 Sent Done Done false
20628 6407 Acked Read 8 Sent Done Done false
20629 6408 Acked Read 3 Sent Done Done false
20630 6409 Acked Read 9 Sent Done Done false
20631 6410 Acked Read 2 Sent Done Done false
20632 6411 Acked Write 7 Sent Done Done false
20633 6412 Acked Write 5 Sent Done Done false
20634 6413 Acked Write 2 Sent Done Done false
20635 6414 Acked Flush 0 Sent Done Done false
20636 6415 Acked Read 2 Sent Done Done false
20637 6416 Acked Read 8 Sent Done Done false
20638 6417 Acked Write 7 Sent Done Done false
20639 6418 Acked Flush 0 Sent Done Done false
20640 6419 Acked Write 10 Sent Done Done false
20641 6420 Acked Read 10 Sent Done Done false
20642 6421 Acked Flush 0 Sent Done Done false
20643 6422 Acked Read 10 Sent Done Done false
20644 6423 Acked Read 4 Sent Done Done false
20645 6424 Acked Write 2 Sent Done Done false
20646 6425 Acked Write 4 Sent Done Done false
20647 6426 Acked Read 1 Sent Done Done false
20648 6427 Acked Read 4 Sent Done Done false
20649 6428 Acked Read 10 Sent Done Done false
20650 6429 Acked Read 9 Sent Done Done false
20651 6430 Acked Flush 0 Sent Done Done false
20652 6431 Acked Write 3 Sent Done Done false
20653 6432 Acked Read 5 Sent Done Done false
20654 STATES DS:0 DS:1 DS:2 TOTAL
20655 Sent 301 0 0 301
20656 Done 0 301 301 602
20657 Skipped 0 0 0 0
20658 Error 0 0 0 0
20659 Last Flush: Some(JobId(6131)) Some(JobId(6430)) Some(JobId(6430))
20660 Downstairs last five completed: 6131 6130 6129 6128 6127
20661 Upstairs last five completed: 6432 6431 6430 6429 6428
20662 CLIENT: Up:0 ds:301 act:2
206632025-02-25T20:51:06.485ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206642025-02-25T20:51:06.973ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206652025-02-25T20:51:06.973ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39966) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206662025-02-25T20:51:06.974ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206672025-02-25T20:51:06.975ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206682025-02-25T20:51:06.975ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206692025-02-25T20:51:06.975ZINFOcrucible: send last flush ID to this DS: Some(JobId(6131)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206702025-02-25T20:51:06.975ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206712025-02-25T20:51:06.975ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6131)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206722025-02-25T20:51:06.975ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6131)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
20673 ----------------------------------------------------------------
20674 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20675 Downstairs last five completed: 6433 6432 6431 6430 6429
20676 Upstairs last five completed: 6433 6432 6431 6430 6429
20677 CLIENT: Up:0 ds:0 act:3
20678 Replay: All jobs finished
20679 Main task has work to do, go find it
20680 got dsc Stop(0)
20681 stop 0
20682 Feb 25 20:51:09.232 INFO request completed, latency_us: 380, response_code: [8810] Got stop action so:false kr:false
20683 204, uri: /stop/cid/0, method: POST, req_id: 2ce6ce30-3f40-449f-9084-c5fabeb2efa6, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
206842025-02-25T20:51:09.233ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206852025-02-25T20:51:09.233ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206862025-02-25T20:51:09.233ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
206872025-02-25T20:51:09.233ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
20688 Feb 25 20:51:09.233 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9a1fa80a-ecfa-408c-9e0b-a093a0d47caf, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
20689 [8810] Exited with: ExitStatus(unix_wait_status(9))
20690 [8810][0] reports Exit
20691 Feb 25 20:51:13.236 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/0, method: GET, req_id: a713f56b-13c2-462f-af4f-ae62c287dd27, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
20692 001/300 Write block 828 len 8192 data: 3 3
20693 002/300 Write block 1205 len 8192 data: 4 4
20694 003/300 Read block 2836 len 4096
20695 004/300 Write block 1320 len 28672 data: 8 8 7 8 7 8 7
20696 005/300 Read block 20 len 20480
20697 006/300 Read block 3886 len 16384
20698 007/300 Read block 3053 len 4096
20699 008/300 Flush
20700 009/300 Write block 4542 len 32768 data: 3 3 3 3 4 4 3 3
20701 010/300 Write block 3284 len 20480 data: 5 4 4 4 3
20702 011/300 Write block 140 len 40960 data: 2 2 2 2 2 2 3 3 3 4
20703 012/300 Write block 1130 len 32768 data: 4 3 2 3 3 3 2 2
20704 013/300 Write block 2594 len 16384 data: 4 4 5 4
20705 014/300 Read block 552 len 40960
20706 015/300 Write block 3924 len 4096 data: 3
20707 016/300 Write block 2905 len 40960 data: 5 5 5 5 3 3 3 3 3 4
20708 017/300 Read block 1816 len 40960
20709 018/300 Read block 2603 len 24576
20710 019/300 Read block 3599 len 4096
20711 020/300 Read block 2328 len 40960
20712 021/300 Read block 2193 len 16384
20713 022/300 Read block 551 len 16384
20714 023/300 Write block 4661 len 20480 data: 4 4 4 4 5
20715 024/300 Write block 2470 len 36864 data: 4 5 4 4 4 4 4 4 3
20716 025/300 Read block 3199 len 32768
20717 026/300 Write block 1714 len 8192 data: 4 5
20718 027/300 Write block 960 len 36864 data: 7 6 5 4 3 2 4 4 4
20719 028/300 Write block 1699 len 20480 data: 3 3 3 3 3
20720 029/300 Flush
20721 030/300 Read block 1931 len 24576
20722 031/300 Read block 4889 len 16384
20723 032/300 Write block 2164 len 20480 data: 5 5 6 6 5
20724 033/300 Write block 2942 len 12288 data: 3 2 2
20725 034/300 Read block 4666 len 8192
20726 035/300 Read block 3139 len 16384
20727 036/300 Read block 4739 len 8192
20728 037/300 Read block 4101 len 12288
20729 038/300 Read block 2509 len 4096
20730 039/300 Read block 327 len 4096
20731 040/300 Write block 3280 len 28672 data: 5 6 4 5 6 5 5
20732 041/300 Write block 3534 len 32768 data: 3 2 2 2 2 2 2 3
20733 042/300 Write block 566 len 8192 data: 2 2
20734 043/300 Write block 1040 len 20480 data: 3 3 4 5 5
20735 044/300 Write block 994 len 36864 data: 6 6 6 3 3 4 4 4 4
20736 045/300 Read block 4329 len 4096
20737 046/300 Read block 1034 len 24576
20738 047/300 Read block 3018 len 40960
20739 048/300 Read block 696 len 40960
20740 049/300 Write block 1397 len 8192 data: 3 3
20741 050/300 Write block 850 len 4096 data: 4
20742 051/300 Write block 2802 len 36864 data: 5 6 6 5 3 3 4 4 3
20743 052/300 Read block 3955 len 20480
20744 053/300 Write block 1741 len 8192 data: 2 2
20745 054/300 Read block 1553 len 24576
20746 055/300 Flush
20747 056/300 Write block 1109 len 32768 data: 4 5 4 5 3 4 4 6
20748 057/300 Read block 3010 len 4096
20749 058/300 Write block 4679 len 8192 data: 3 3
20750 059/300 Read block 745 len 20480
20751 060/300 Read block 138 len 20480
20752 061/300 Read block 1300 len 12288
20753 062/300 Read block 3964 len 12288
20754 063/300 Read block 2227 len 16384
20755 064/300 Write block 4859 len 32768 data: 2 3 3 3 2 2 2 3
20756 065/300 Read block 1607 len 20480
20757 066/300 Write block 1735 len 28672 data: 4 4 3 3 2 2 3
20758 067/300 Read block 2733 len 20480
20759 068/300 Flush
20760 069/300 Write block 948 len 28672 data: 5 3 2 3 3 3 4
20761 070/300 Read block 2889 len 20480
20762 071/300 Write block 1718 len 28672 data: 4 4 5 5 5 5 4
20763 072/300 Read block 777 len 16384
20764 073/300 Read block 1734 len 36864
20765 074/300 Read block 901 len 32768
20766 075/300 Read block 4335 len 24576
20767 076/300 Read block 2675 len 4096
20768 077/300 Write block 554 len 36864 data: 5 6 6 4 4 4 5 5 4
20769 078/300 Read block 1661 len 40960
20770 079/300 Flush
20771 080/300 Read block 3669 len 16384
20772 081/300 Read block 733 len 32768
20773 082/300 Write block 3922 len 24576 data: 4 4 4 3 3 3
20774 083/300 Write block 4331 len 24576 data: 3 4 4 4 3 3
20775 084/300 Read block 3190 len 12288
20776 085/300 Write block 1021 len 40960 data: 6 5 5 7 8 6 7 5 4 4
20777 086/300 Flush
20778 087/300 Write block 3710 len 40960 data: 5 6 6 5 5 5 4 3 3 4
20779 088/300 Read block 1193 len 20480
20780 089/300 Flush
20781 090/300 Write block 441 len 36864 data: 6 6 6 4 4 4 4 4 3
20782 091/300 Write block 1806 len 32768 data: 6 6 4 4 4 4 6 6
20783 092/300 Read block 2684 len 32768
20784 093/300 Read block 3849 len 28672
20785 094/300 Read block 2983 len 32768
20786 095/300 Read block 2121 len 16384
20787 096/300 Write block 4301 len 16384 data: 4 4 3 3
20788 097/300 Write block 4461 len 8192 data: 7 8
20789 098/300 Read block 352 len 8192
20790 099/300 Flush
20791 100/300 Write block 4458 len 32768 data: 4 6 5 8 9 6 5 4
20792 101/300 Read block 4423 len 36864
20793 102/300 Write block 1318 len 32768 data: 5 6 9 9 8 9 8 9
20794 103/300 Flush
20795 104/300 Read block 4476 len 4096
20796 105/300 Write block 3095 len 32768 data: 3 3 3 3 4 4 4 4
20797 106/300 Read block 4824 len 4096
20798 107/300 Read block 1940 len 8192
20799 108/300 Read block 1789 len 28672
20800 109/300 Write block 4942 len 40960 data: 6 6 5 3 3 3 3 5 5 5
20801 110/300 Write block 2509 len 20480 data: 3 3 4 4 5
20802 111/300 Read block 511 len 8192
20803 112/300 Write block 804 len 12288 data: 4 4 4
20804 113/300 Read block 3956 len 40960
20805 114/300 Write block 1485 len 28672 data: 3 3 3 2 3 2 3
20806 115/300 Flush
20807 116/300 Read block 4830 len 32768
20808 117/300 Flush
20809 118/300 Read block 3474 len 24576
20810 119/300 Write block 4378 len 28672 data: 4 3 3 3 4 4 4
20811 120/300 Write block 1164 len 24576 data: 3 4 5 5 6 6
20812 121/300 Write block 4454 len 28672 data: 6 5 5 5 5 7 6
20813 122/300 Write block 547 len 40960 data: 3 4 4 5 5 5 6 6 7 7
20814 123/300 Read block 523 len 40960
20815 124/300 Write block 2045 len 12288 data: 3 2 2
20816 125/300 Read block 4970 len 32768
20817 126/300 Write block 3746 len 8192 data: 4 5
20818 127/300 Write block 3976 len 28672 data: 6 7 6 7 5 5 5
20819 128/300 Write block 1412 len 4096 data: 4
20820 129/300 Read block 349 len 12288
20821 130/300 Read block 4092 len 36864
20822 131/300 Read block 3993 len 32768
20823 132/300 Flush
20824 133/300 Read block 4135 len 16384
20825 134/300 Write block 2804 len 32768 data: 7 6 4 4 5 5 4 3
20826 135/300 Read block 4833 len 8192
20827 136/300 Flush
20828 137/300 Write block 2148 len 24576 data: 3 3 3 3 4 3
20829 138/300 Read block 2857 len 20480
20830 139/300 Flush
20831 140/300 Write block 2701 len 8192 data: 4 4
20832 141/300 Read block 1807 len 4096
20833 142/300 Read block 3998 len 36864
20834 143/300 Read block 4402 len 28672
20835 144/300 Write block 844 len 20480 data: 7 6 5 5 5
20836 145/300 Read block 362 len 12288
20837 146/300 Read block 956 len 20480
20838 147/300 Read block 4229 len 8192
20839 148/300 Read block 1508 len 16384
20840 149/300 Read block 1689 len 28672
20841 150/300 Write block 3637 len 12288 data: 7 8 7
20842 151/300 Write block 2781 len 12288 data: 4 4 5
20843 152/300 Write block 2735 len 40960 data: 4 3 5 5 5 5 5 6 7 4
20844 153/300 Read block 4281 len 4096
20845 154/300 Write block 1786 len 32768 data: 6 5 5 6 7 6 6 6
20846 155/300 Write block 3063 len 24576 data: 4 4 3 4 4 4
20847 156/300 Write block 656 len 36864 data: 5 5 5 5 6 6 6 6 6
20848 157/300 Write block 2425 len 36864 data: 5 7 7 7 6 6 6 5 3
20849 158/300 Write block 3067 len 20480 data: 5 5 5 5 4
20850 159/300 Read block 579 len 28672
20851 160/300 Write block 371 len 20480 data: 4 3 3 4 4
20852 161/300 Write block 2316 len 36864 data: 5 6 5 5 6 5 6 5 5
20853 162/300 Write block 503 len 40960 data: 5 4 3 4 4 2 2 2 3 5
20854 163/300 Flush
20855 164/300 Read block 1004 len 8192
20856 165/300 Write block 2826 len 40960 data: 5 4 5 5 5 4 5 5 5 5
20857 166/300 Write block 3609 len 8192 data: 5 4
20858 167/300 Write block 4094 len 40960 data: 4 4 4 3 3 6 7 6 6 6
20859 168/300 Read block 3273 len 16384
20860 169/300 Read block 286 len 12288
20861 170/300 Read block 3299 len 12288
20862 171/300 Read block 3242 len 32768
20863 172/300 Flush
20864 173/300 Write block 2151 len 12288 data: 4 5 4
20865 174/300 Write block 429 len 40960 data: 5 7 6 4 3 4 4 4 7 6
20866 175/300 Read block 405 len 36864
20867 176/300 Write block 1074 len 8192 data: 6 5
20868 177/300 Read block 579 len 20480
20869 178/300 Write block 3348 len 12288 data: 6 7 7
20870 179/300 Read block 1335 len 24576
20871 180/300 Read block 3523 len 36864
20872 181/300 Write block 549 len 8192 data: 5 6
20873 182/300 Write block 4703 len 36864 data: 2 3 3 2 3 3 4 4 5
20874 183/300 Flush
20875 184/300 Write block 2512 len 32768 data: 5 6 6 5 5 6 7 7
20876 185/300 Read block 1429 len 16384
20877 186/300 Read block 3657 len 24576
20878 187/300 Read block 1419 len 36864
20879 188/300 Read block 506 len 16384
20880 189/300 Read block 3898 len 8192
20881 190/300 Write block 3913 len 40960 data: 4 4 5 5 5 4 5 5 5 5
20882 191/300 Read block 4883 len 40960
20883 192/300 Write block 3304 len 36864 data: 9 7 5 4 5 5 7 6 6
20884 193/300 Flush
20885 194/300 Read block 1679 len 4096
20886 195/300 Flush
20887 196/300 Read block 4782 len 4096
20888 197/300 Read block 1787 len 12288
20889 198/300 Flush
20890 199/300 Read block 3682 len 8192
20891 200/300 Read block 4165 len 28672
20892 201/300 Read block 1578 len 8192
20893 202/300 Flush
20894 203/300 Read block 2057 len 4096
20895 204/300 Flush
20896 205/300 Write block 3403 len 8192 data: 4 4
20897 206/300 Read block 4097 len 8192
20898 207/300 Write block 2826 len 8192 data: 6 5
20899 208/300 Read block 2915 len 8192
20900 209/300 Write block 2155 len 28672 data: 5 5 6 6 8 8 6
20901 210/300 Read block 4603 len 4096
20902 211/300 Read block 2347 len 4096
20903 212/300 Read block 860 len 40960
20904 213/300 Write block 3627 len 20480 data: 6 6 8 8 7
20905 214/300 Flush
20906 215/300 Read block 2137 len 12288
20907 216/300 Read block 885 len 20480
20908 217/300 Read block 2755 len 12288
20909 218/300 Write block 1231 len 36864 data: 3 3 3 2 4 4 4 4 2
20910 219/300 Read block 2782 len 24576
20911 220/300 Flush
20912 221/300 Read block 2805 len 4096
20913 222/300 Read block 1650 len 20480
20914 223/300 Read block 436 len 16384
20915 224/300 Read block 4107 len 20480
20916 225/300 Read block 3125 len 20480
20917 226/300 Write block 655 len 40960 data: 4 6 6 6 6 7 7 7 7 7
20918 227/300 Write block 3851 len 28672 data: 5 6 6 7 6 5 4
20919 228/300 Read block 291 len 16384
20920 229/300 Read block 4775 len 20480
20921 230/300 Read block 1203 len 32768
20922 231/300 Read block 1365 len 12288
20923 232/300 Read block 4985 len 12288
20924 233/300 Read block 1191 len 32768
20925 234/300 Flush
20926 235/300 Read block 4226 len 28672
20927 236/300 Write block 640 len 20480 data: 2 2 2 2 2
20928 237/300 Read block 2195 len 4096
20929 238/300 Read block 1464 len 32768
20930 239/300 Flush
20931 240/300 Write block 3192 len 40960 data: 4 4 4 5 5 5 6 5 5 5
20932 241/300 Write block 468 len 36864 data: 2 2 2 3 3 3 2 2 2
20933 242/300 Read block 2200 len 32768
20934 243/300 Write block 3680 len 4096 data: 5
20935 244/300 Write block 3167 len 8192 data: 3 4
20936 245/300 Read block 835 len 40960
20937 246/300 Read block 1700 len 16384
20938 247/300 Read block 1575 len 40960
20939 248/300 Read block 3993 len 20480
20940 249/300 Write block 2495 len 32768 data: 5 4 5 5 5 5 5 5
20941 250/300 Read block 2639 len 12288
20942 251/300 Read block 4460 len 12288
20943 252/300 Read block 3470 len 8192
20944 253/300 Flush
20945 254/300 Read block 2702 len 4096
20946 255/300 Read block 4994 len 24576
20947 256/300 Read block 4631 len 20480
20948 257/300 Read block 3723 len 20480
20949 258/300 Read block 2918 len 32768
20950 259/300 Read block 200 len 20480
20951 260/300 Write block 3680 len 8192 data: 6 5
20952 261/300 Read block 647 len 16384
20953 262/300 Read block 835 len 36864
20954 263/300 Write block 3874 len 20480 data: 6 6 6 7 6
20955 264/300 Read block 2804 len 8192
20956 265/300 Write block 3319 len 40960 data: 3 3 3 3 4 5 4 4 3 4
20957 266/300 Write block 215 len 36864 data: 6 6 4 5 6 7 6 6 5
20958 267/300 Write block 4903 len 16384 data: 5 5 4 3
20959 268/300 Write block 1605 len 16384 data: 6 6 6 5
20960 269/300 Read block 2456 len 8192
20961 270/300 Write block 3316 len 32768 data: 6 5 3 4 4 4 4 5
20962 271/300 Read block 2257 len 36864
20963 272/300 Write block 2074 len 28672 data: 3 4 5 3 3 3 3
20964 273/300 Write block 4940 len 28672 data: 6 6 7 7 6 4 4
20965 274/300 Read block 4190 len 16384
20966 275/300 Read block 1362 len 36864
20967 276/300 Flush
20968 277/300 Read block 4669 len 4096
20969 278/300 Read block 3062 len 4096
20970 279/300 Write block 3181 len 24576 data: 5 5 5 4 4 4
20971 280/300 Read block 3795 len 8192
20972 281/300 Read block 532 len 36864
20973 282/300 Write block 4367 len 8192 data: 4 7
20974 283/300 Read block 51 len 20480
20975 284/300 Flush
20976 285/300 Read block 3447 len 24576
20977 286/300 Write block 343 len 24576 data: 4 4 5 5 5 5
20978 287/300 Write block 666 len 16384 data: 6 6 7 4
20979 288/300 Read block 668 len 40960
20980 289/300 Write block 1167 len 16384 data: 6 7 7 4
20981 290/300 Read block 1586 len 40960
20982 291/300 Read block 3722 len 4096
20983 292/300 Write block 4432 len 28672 data: 3 4 4 4 5 5 4
20984 293/300 Write block 1094 len 36864 data: 6 8 8 6 4 4 5 5 6
20985 294/300 Read block 4692 len 12288
20986 295/300 Read block 1514 len 8192
20987 296/300 Write block 4654 len 16384 data: 4 4 4 5
20988 297/300 Read block 2683 len 16384
20989 298/300 Read block 660 len 40960
20990 299/300 Read block 3896 len 28672
20991 300/300 Write block 4025 len 20480 data: 4 4 3 4 5
20992 301/300 Read block 4554 len 24576
20993 Main task has work to do, go find it
20994 got dsc Start(0)
20995 start 0
20996 Feb 25 20:51:13.483 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1d4015c7-c89c-420a-9f61-8877c31d03cc, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
20997 Replay: started 0, returned:Ok(())
20998 ----------------------------------------------------------------
20999 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21000 States: Offline Active Active
21001 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21002 6434 Acked Write 2 Sent Done Done false
21003 6435 Acked Write 2 Sent Done Done false
21004 6436 Acked Read 1 Sent Done Done false
21005 6437 Acked Write 7 Sent Done Done false
21006 6438 Acked Read 5 Sent Done Done false
21007 6439 Acked Read 4 Sent Done Done false
21008 6440 Acked Read 1 Sent Done Done false
21009 6441 Acked Flush 0 Sent Done Done false
21010 6442 Acked Write 8 Sent Done Done false
21011 6443 Acked Write 5 Sent Done Done false
21012 6444 Acked Write 10 Sent Done Done false
21013 6445 Acked Write 8 Sent Done Done false
21014 6446 Acked Write 4 Sent Done Done false
21015 6447 Acked Read 10 Sent Done Done false
21016 6448 Acked Write 1 Sent Done Done false
21017 6449 Acked Write 10 Sent Done Done false
21018 6450 Acked Read 10 Sent Done Done false
21019 6451 Acked Read 6 Sent Done Done false
21020 6452 Acked Read 1 Sent Done Done false
21021 6453 Acked Read 10 Sent Done Done false
21022 6454 Acked Read 4 Sent Done Done false
21023 6455 Acked Read 4 Sent Done Done false
21024 6456 Acked Write 5 Sent Done Done false
21025 6457 Acked Write 9 Sent Done Done false
21026 6458 Acked Read 8 Sent Done Done false
21027 6459 Acked Write 2 Sent Done Done false
21028 6460 Acked Write 9 Sent Done Done false
21029 6461 Acked Write 5 Sent Done Done false
21030 6462 Acked Flush 0 Sent Done Done false
21031 6463 Acked Read 6 Sent Done Done false
21032 6464 Acked Read 4 Sent Done Done false
21033 6465 Acked Write 5 Sent Done Done false
21034 6466 Acked Write 3 Sent Done Done false
21035 6467 Acked Read 2 Sent Done Done false
21036 6468 Acked Read 4 Sent Done Done false
21037 6469 Acked Read 2 Sent Done Done false
21038 6470 Acked Read 3 Sent Done Done false
21039 6471 Acked Read 1 Sent Done Done false
21040 6472 Acked Read 1 Sent Done Done false
21041 6473 Acked Write 7 Sent Done Done false
21042 6474 Acked Write 8 Sent Done Done false
21043 6475 Acked Write 2 Sent Done Done false
21044 6476 Acked Write 5 Sent Done Done false
21045 6477 Acked Write 9 Sent Done Done false
21046 6478 Acked Read 1 Sent Done Done false
21047 6479 Acked Read 6 Sent Done Done false
21048 6480 Acked Read 10 Sent Done Done false
21049 6481 Acked Read 10 Sent Done Done false
21050 6482 Acked Write 2 Sent Done Done false
21051 6483 Acked Write 1 Sent Done Done false
21052 6484 Acked Write 9 Sent Done Done false
21053 6485 Acked Read 5 Sent Done Done false
21054 6486 Acked Write 2 Sent Done Done false
21055 6487 Acked Read 6 Sent Done Done false
21056 6488 Acked Flush 0 Sent Done Done false
21057 6489 Acked Write 8 Sent Done Done false
21058 6490 Acked Read 1 Sent Done Done false
21059 6491 Acked Write 2 Sent Done Done false
21060 6492 Acked Read 5 Sent Done Done false
21061 6493 Acked Read 5 Sent Done Done false
21062 6494 Acked Read 3 Sent Done Done false
21063 6495 Acked Read 3 Sent Done Done false
21064 6496 Acked Read 4 Sent Done Done false
21065 6497 Acked Write 8 Sent Done Done false
21066 6498 Acked Read 5 Sent Done Done false
21067 6499 Acked Write 7 Sent Done Done false
21068 6500 Acked Read 5 Sent Done Done false
21069 6501 Acked Flush 0 Sent Done Done false
21070 6502 Acked Write 7 Sent Done Done false
21071 6503 Acked Read 5 Sent Done Done false
21072 6504 Acked Write 7 Sent Done Done false
21073 6505 Acked Read 4 Sent Done Done false
21074 6506 Acked Read 9 Sent Done Done false
21075 6507 Acked Read 8 Sent Done Done false
21076 6508 Acked Read 6 Sent Done Done false
21077 6509 Acked Read 1 Sent Done Done false
21078 6510 Acked Write 9 Sent Done Done false
21079 6511 Acked Read 10 Sent Done Done false
21080 6512 Acked Flush 0 Sent Done Done false
21081 6513 Acked Read 4 Sent Done Done false
21082 6514 Acked Read 8 Sent Done Done false
21083 6515 Acked Write 6 Sent Done Done false
21084 6516 Acked Write 6 Sent Done Done false
21085 6517 Acked Read 3 Sent Done Done false
21086 6518 Acked Write 10 Sent Done Done false
21087 6519 Acked Flush 0 Sent Done Done false
21088 6520 Acked Write 10 Sent Done Done false
21089 6521 Acked Read 5 Sent Done Done false
21090 6522 Acked Flush 0 Sent Done Done false
21091 6523 Acked Write 9 Sent Done Done false
21092 6524 Acked Write 8 Sent Done Done false
21093 6525 Acked Read 8 Sent Done Done false
21094 6526 Acked Read 7 Sent Done Done false
21095 6527 Acked Read 8 Sent Done Done false
21096 6528 Acked Read 4 Sent Done Done false
21097 6529 Acked Write 4 Sent Done Done false
21098 6530 Acked Write 2 Sent Done Done false
21099 6531 Acked Read 2 Sent Done Done false
21100 6532 Acked Flush 0 Sent Done Done false
21101 6533 Acked Write 8 Sent Done Done false
21102 6534 Acked Read 9 Sent Done Done false
21103 6535 Acked Write 8 Sent Done Done false
21104 6536 Acked Flush 0 Sent Done Done false
21105 6537 Acked Read 1 Sent Done Done false
21106 6538 Acked Write 8 Sent Done Done false
21107 6539 Acked Read 1 Sent Done Done false
21108 6540 Acked Read 2 Sent Done Done false
21109 6541 Acked Read 7 Sent Done Done false
21110 6542 Acked Write 10 Sent Done Done false
21111 6543 Acked Write 5 Sent Done Done false
21112 6544 Acked Read 2 Sent Done Done false
21113 6545 Acked Write 3 Sent Done Done false
21114 6546 Acked Read 10 Sent Done Done false
21115 6547 Acked Write 7 Sent Done Done false
21116 6548 Acked Flush 0 Sent Done Done false
21117 6549 Acked Read 8 Sent Done Done false
21118 6550 Acked Flush 0 Sent Done Done false
21119 6551 Acked Read 6 Sent Done Done false
21120 6552 Acked Write 7 Sent Done Done false
21121 6553 Acked Write 6 Sent Done Done false
21122 6554 Acked Write 7 Sent Done Done false
21123 6555 Acked Write 10 Sent Done Done false
21124 6556 Acked Read 10 Sent Done Done false
21125 6557 Acked Write 3 Sent Done Done false
21126 6558 Acked Read 8 Sent Done Done false
21127 6559 Acked Write 2 Sent Done Done false
21128 6560 Acked Write 7 Sent Done Done false
21129 6561 Acked Write 1 Sent Done Done false
21130 6562 Acked Read 3 Sent Done Done false
21131 6563 Acked Read 9 Sent Done Done false
21132 6564 Acked Read 8 Sent Done Done false
21133 6565 Acked Flush 0 Sent Done Done false
21134 6566 Acked Read 4 Sent Done Done false
21135 6567 Acked Write 8 Sent Done Done false
21136 6568 Acked Read 2 Sent Done Done false
21137 6569 Acked Flush 0 Sent Done Done false
21138 6570 Acked Write 6 Sent Done Done false
21139 6571 Acked Read 5 Sent Done Done false
21140 6572 Acked Flush 0 Sent Done Done false
21141 6573 Acked Write 2 Sent Done Done false
21142 6574 Acked Read 1 Sent Done Done false
21143 6575 Acked Read 9 Sent Done Done false
21144 6576 Acked Read 7 Sent Done Done false
21145 6577 Acked Write 5 Sent Done Done false
21146 6578 Acked Read 3 Sent Done Done false
21147 6579 Acked Read 5 Sent Done Done false
21148 6580 Acked Read 2 Sent Done Done false
21149 6581 Acked Read 4 Sent Done Done false
21150 6582 Acked Read 7 Sent Done Done false
21151 6583 Acked Write 3 Sent Done Done false
21152 6584 Acked Write 3 Sent Done Done false
21153 6585 Acked Write 10 Sent Done Done false
21154 6586 Acked Read 1 Sent Done Done false
21155 6587 Acked Write 8 Sent Done Done false
21156 6588 Acked Write 6 Sent Done Done false
21157 6589 Acked Write 9 Sent Done Done false
21158 6590 Acked Write 9 Sent Done Done false
21159 6591 Acked Write 5 Sent Done Done false
21160 6592 Acked Read 7 Sent Done Done false
21161 6593 Acked Write 5 Sent Done Done false
21162 6594 Acked Write 9 Sent Done Done false
21163 6595 Acked Write 10 Sent Done Done false
21164 6596 Acked Flush 0 Sent Done Done false
21165 6597 Acked Read 2 Sent Done Done false
21166 6598 Acked Write 10 Sent Done Done false
21167 6599 Acked Write 2 Sent Done Done false
21168 6600 Acked Write 10 Sent Done Done false
21169 6601 Acked Read 4 Sent Done Done false
21170 6602 Acked Read 3 Sent Done Done false
21171 6603 Acked Read 3 Sent Done Done false
21172 6604 Acked Read 8 Sent Done Done false
21173 6605 Acked Flush 0 Sent Done Done false
21174 6606 Acked Write 3 Sent Done Done false
21175 6607 Acked Write 10 Sent Done Done false
21176 6608 Acked Read 9 Sent Done Done false
21177 6609 Acked Write 2 Sent Done Done false
21178 6610 Acked Read 5 Sent Done Done false
21179 6611 Acked Write 3 Sent Done Done false
21180 6612 Acked Read 6 Sent Done Done false
21181 6613 Acked Read 9 Sent Done Done false
21182 6614 Acked Write 2 Sent Done Done false
21183 6615 Acked Write 9 Sent Done Done false
21184 6616 Acked Flush 0 Sent Done Done false
21185 6617 Acked Write 8 Sent Done Done false
21186 6618 Acked Read 4 Sent Done Done false
21187 6619 Acked Read 6 Sent Done Done false
21188 6620 Acked Read 9 Sent Done Done false
21189 6621 Acked Read 4 Sent Done Done false
21190 6622 Acked Read 2 Sent Done Done false
21191 6623 Acked Write 10 Sent Done Done false
21192 6624 Acked Read 10 Sent Done Done false
21193 6625 Acked Write 9 Sent Done Done false
21194 6626 Acked Flush 0 Sent Done Done false
21195 6627 Acked Read 1 Sent Done Done false
21196 6628 Acked Flush 0 Sent Done Done false
21197 6629 Acked Read 1 Sent Done Done false
21198 6630 Acked Read 3 Sent Done Done false
21199 6631 Acked Flush 0 Sent Done Done false
21200 6632 Acked Read 2 Sent Done Done false
21201 6633 Acked Read 7 Sent Done Done false
21202 6634 Acked Read 2 Sent Done Done false
21203 6635 Acked Flush 0 Sent Done Done false
21204 6636 Acked Read 1 Sent Done Done false
21205 6637 Acked Flush 0 Sent Done Done false
21206 6638 Acked Write 2 Sent Done Done false
21207 6639 Acked Read 2 Sent Done Done false
21208 6640 Acked Write 2 Sent Done Done false
21209 6641 Acked Read 2 Sent Done Done false
21210 6642 Acked Write 7 Sent Done Done false
21211 6643 Acked Read 1 Sent Done Done false
21212 6644 Acked Read 1 Sent Done Done false
21213 6645 Acked Read 10 Sent Done Done false
21214 6646 Acked Write 5 Sent Done Done false
21215 6647 Acked Flush 0 Sent Done Done false
21216 6648 Acked Read 3 Sent Done Done false
21217 6649 Acked Read 5 Sent Done Done false
21218 6650 Acked Read 3 Sent Done Done false
21219 6651 Acked Write 9 Sent Done Done false
21220 6652 Acked Read 6 Sent Done Done false
21221 6653 Acked Flush 0 Sent Done Done false
21222 6654 Acked Read 1 Sent Done Done false
21223 6655 Acked Read 5 Sent Done Done false
21224 6656 Acked Read 4 Sent Done Done false
21225 6657 Acked Read 5 Sent Done Done false
21226 6658 Acked Read 5 Sent Done Done false
21227 6659 Acked Write 10 Sent Done Done false
21228 6660 Acked Write 7 Sent Done Done false
21229 6661 Acked Read 4 Sent Done Done false
21230 6662 Acked Read 5 Sent Done Done false
21231 6663 Acked Read 8 Sent Done Done false
21232 6664 Acked Read 3 Sent Done Done false
21233 6665 Acked Read 3 Sent Done Done false
21234 6666 Acked Read 8 Sent Done Done false
21235 6667 Acked Flush 0 Sent Done Done false
21236 6668 Acked Read 7 Sent Done Done false
21237 6669 Acked Write 5 Sent Done Done false
21238 6670 Acked Read 1 Sent Done Done false
21239 6671 Acked Read 8 Sent Done Done false
21240 6672 Acked Flush 0 Sent Done Done false
21241 6673 Acked Write 10 Sent Done Done false
21242 6674 Acked Write 9 Sent Done Done false
21243 6675 Acked Read 8 Sent Done Done false
21244 6676 Acked Write 1 Sent Done Done false
21245 6677 Acked Write 2 Sent Done Done false
21246 6678 Acked Read 10 Sent Done Done false
21247 6679 Acked Read 4 Sent Done Done false
21248 6680 Acked Read 10 Sent Done Done false
21249 6681 Acked Read 5 Sent Done Done false
21250 6682 Acked Write 8 Sent Done Done false
21251 6683 Acked Read 3 Sent Done Done false
21252 6684 Acked Read 3 Sent Done Done false
21253 6685 Acked Read 2 Sent Done Done false
21254 6686 Acked Flush 0 Sent Done Done false
21255 6687 Acked Read 1 Sent Done Done false
21256 6688 Acked Read 6 Sent Done Done false
21257 6689 Acked Read 5 Sent Done Done false
21258 6690 Acked Read 5 Sent Done Done false
21259 6691 Acked Read 8 Sent Done Done false
21260 6692 Acked Read 5 Sent Done Done false
21261 6693 Acked Write 2 Sent Done Done false
21262 6694 Acked Read 4 Sent Done Done false
21263 6695 Acked Read 9 Sent Done Done false
21264 6696 Acked Write 5 Sent Done Done false
21265 6697 Acked Read 2 Sent Done Done false
21266 6698 Acked Write 10 Sent Done Done false
21267 6699 Acked Write 9 Sent Done Done false
21268 6700 Acked Write 4 Sent Done Done false
21269 6701 Acked Write 4 Sent Done Done false
21270 6702 Acked Read 2 Sent Done Done false
21271 6703 Acked Write 8 Sent Done Done false
21272 6704 Acked Read 9 Sent Done Done false
21273 6705 Acked Write 7 Sent Done Done false
21274 6706 Acked Write 7 Sent Done Done false
21275 6707 Acked Read 4 Sent Done Done false
21276 6708 Acked Read 9 Sent Done Done false
21277 6709 Acked Flush 0 Sent Done Done false
21278 6710 Acked Read 1 Sent Done Done false
21279 6711 Acked Read 1 Sent Done Done false
21280 6712 Acked Write 6 Sent Done Done false
21281 6713 Acked Read 2 Sent Done Done false
21282 6714 Acked Read 9 Sent Done Done false
21283 6715 Acked Write 2 Sent Done Done false
21284 6716 Acked Read 5 Sent Done Done false
21285 6717 Acked Flush 0 Sent Done Done false
21286 6718 Acked Read 6 Sent Done Done false
21287 6719 Acked Write 6 Sent Done Done false
21288 6720 Acked Write 4 Sent Done Done false
21289 6721 Acked Read 10 Sent Done Done false
21290 6722 Acked Write 4 Sent Done Done false
21291 6723 Acked Read 10 Sent Done Done false
21292 6724 Acked Read 1 Sent Done Done false
21293 6725 Acked Write 7 Sent Done Done false
21294 6726 Acked Write 9 Sent Done Done false
21295 6727 Acked Read 3 Sent Done Done false
21296 6728 Acked Read 2 Sent Done Done false
21297 6729 Acked Write 4 Sent Done Done false
21298 6730 Acked Read 4 Sent Done Done false
21299 6731 Acked Read 10 Sent Done Done false
21300 6732 Acked Read 7 Sent Done Done false
21301 6733 Acked Write 5 Sent Done Done false
21302 6734 Acked Read 6 Sent Done Done false
21303 STATES DS:0 DS:1 DS:2 TOTAL
21304 Sent 301 0 0 301
21305 Done 0 301 301 602
21306 Skipped 0 0 0 0
21307 Error 0 0 0 0
21308 Last Flush: Some(JobId(6433)) Some(JobId(6717)) Some(JobId(6717))
21309 Downstairs last five completed: 6433 6432 6431 6430 6429
21310 Upstairs last five completed: 6734 6733 6732 6731 6730
21311 CLIENT: Up:0 ds:301 act:2
213122025-02-25T20:51:13.986ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
213132025-02-25T20:51:14.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
213142025-02-25T20:51:15.499ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
213152025-02-25T20:51:16.501ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21316 ----------------------------------------------------------------
21317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21318 States: Offline Active Active
21319 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21320 6434 Acked Write 2 Sent Done Done false
21321 6435 Acked Write 2 Sent Done Done false
21322 6436 Acked Read 1 Sent Done Done false
21323 6437 Acked Write 7 Sent Done Done false
21324 6438 Acked Read 5 Sent Done Done false
21325 6439 Acked Read 4 Sent Done Done false
21326 6440 Acked Read 1 Sent Done Done false
21327 6441 Acked Flush 0 Sent Done Done false
21328 6442 Acked Write 8 Sent Done Done false
21329 6443 Acked Write 5 Sent Done Done false
21330 6444 Acked Write 10 Sent Done Done false
21331 6445 Acked Write 8 Sent Done Done false
21332 6446 Acked Write 4 Sent Done Done false
21333 6447 Acked Read 10 Sent Done Done false
21334 6448 Acked Write 1 Sent Done Done false
21335 6449 Acked Write 10 Sent Done Done false
21336 6450 Acked Read 10 Sent Done Done false
21337 6451 Acked Read 6 Sent Done Done false
21338 6452 Acked Read 1 Sent Done Done false
21339 6453 Acked Read 10 Sent Done Done false
21340 6454 Acked Read 4 Sent Done Done false
21341 6455 Acked Read 4 Sent Done Done false
21342 6456 Acked Write 5 Sent Done Done false
21343 6457 Acked Write 9 Sent Done Done false
21344 6458 Acked Read 8 Sent Done Done false
21345 6459 Acked Write 2 Sent Done Done false
21346 6460 Acked Write 9 Sent Done Done false
21347 6461 Acked Write 5 Sent Done Done false
21348 6462 Acked Flush 0 Sent Done Done false
21349 6463 Acked Read 6 Sent Done Done false
21350 6464 Acked Read 4 Sent Done Done false
21351 6465 Acked Write 5 Sent Done Done false
21352 6466 Acked Write 3 Sent Done Done false
21353 6467 Acked Read 2 Sent Done Done false
21354 6468 Acked Read 4 Sent Done Done false
21355 6469 Acked Read 2 Sent Done Done false
21356 6470 Acked Read 3 Sent Done Done false
21357 6471 Acked Read 1 Sent Done Done false
21358 6472 Acked Read 1 Sent Done Done false
21359 6473 Acked Write 7 Sent Done Done false
21360 6474 Acked Write 8 Sent Done Done false
21361 6475 Acked Write 2 Sent Done Done false
21362 6476 Acked Write 5 Sent Done Done false
21363 6477 Acked Write 9 Sent Done Done false
21364 6478 Acked Read 1 Sent Done Done false
21365 6479 Acked Read 6 Sent Done Done false
21366 6480 Acked Read 10 Sent Done Done false
21367 6481 Acked Read 10 Sent Done Done false
21368 6482 Acked Write 2 Sent Done Done false
21369 6483 Acked Write 1 Sent Done Done false
21370 6484 Acked Write 9 Sent Done Done false
21371 6485 Acked Read 5 Sent Done Done false
21372 6486 Acked Write 2 Sent Done Done false
21373 6487 Acked Read 6 Sent Done Done false
21374 6488 Acked Flush 0 Sent Done Done false
21375 6489 Acked Write 8 Sent Done Done false
21376 6490 Acked Read 1 Sent Done Done false
21377 6491 Acked Write 2 Sent Done Done false
21378 6492 Acked Read 5 Sent Done Done false
21379 6493 Acked Read 5 Sent Done Done false
21380 6494 Acked Read 3 Sent Done Done false
21381 6495 Acked Read 3 Sent Done Done false
21382 6496 Acked Read 4 Sent Done Done false
21383 6497 Acked Write 8 Sent Done Done false
21384 6498 Acked Read 5 Sent Done Done false
21385 6499 Acked Write 7 Sent Done Done false
21386 6500 Acked Read 5 Sent Done Done false
21387 6501 Acked Flush 0 Sent Done Done false
21388 6502 Acked Write 7 Sent Done Done false
21389 6503 Acked Read 5 Sent Done Done false
21390 6504 Acked Write 7 Sent Done Done false
21391 6505 Acked Read 4 Sent Done Done false
21392 6506 Acked Read 9 Sent Done Done false
21393 6507 Acked Read 8 Sent Done Done false
21394 6508 Acked Read 6 Sent Done Done false
21395 6509 Acked Read 1 Sent Done Done false
21396 6510 Acked Write 9 Sent Done Done false
21397 6511 Acked Read 10 Sent Done Done false
21398 6512 Acked Flush 0 Sent Done Done false
21399 6513 Acked Read 4 Sent Done Done false
21400 6514 Acked Read 8 Sent Done Done false
21401 6515 Acked Write 6 Sent Done Done false
21402 6516 Acked Write 6 Sent Done Done false
21403 6517 Acked Read 3 Sent Done Done false
21404 6518 Acked Write 10 Sent Done Done false
21405 6519 Acked Flush 0 Sent Done Done false
21406 6520 Acked Write 10 Sent Done Done false
21407 6521 Acked Read 5 Sent Done Done false
21408 6522 Acked Flush 0 Sent Done Done false
21409 6523 Acked Write 9 Sent Done Done false
21410 6524 Acked Write 8 Sent Done Done false
21411 6525 Acked Read 8 Sent Done Done false
21412 6526 Acked Read 7 Sent Done Done false
21413 6527 Acked Read 8 Sent Done Done false
21414 6528 Acked Read 4 Sent Done Done false
21415 6529 Acked Write 4 Sent Done Done false
21416 6530 Acked Write 2 Sent Done Done false
21417 6531 Acked Read 2 Sent Done Done false
21418 6532 Acked Flush 0 Sent Done Done false
21419 6533 Acked Write 8 Sent Done Done false
21420 6534 Acked Read 9 Sent Done Done false
21421 6535 Acked Write 8 Sent Done Done false
21422 6536 Acked Flush 0 Sent Done Done false
21423 6537 Acked Read 1 Sent Done Done false
21424 6538 Acked Write 8 Sent Done Done false
21425 6539 Acked Read 1 Sent Done Done false
21426 6540 Acked Read 2 Sent Done Done false
21427 6541 Acked Read 7 Sent Done Done false
21428 6542 Acked Write 10 Sent Done Done false
21429 6543 Acked Write 5 Sent Done Done false
21430 6544 Acked Read 2 Sent Done Done false
21431 6545 Acked Write 3 Sent Done Done false
21432 6546 Acked Read 10 Sent Done Done false
21433 6547 Acked Write 7 Sent Done Done false
21434 6548 Acked Flush 0 Sent Done Done false
21435 6549 Acked Read 8 Sent Done Done false
21436 6550 Acked Flush 0 Sent Done Done false
21437 6551 Acked Read 6 Sent Done Done false
21438 6552 Acked Write 7 Sent Done Done false
21439 6553 Acked Write 6 Sent Done Done false
21440 6554 Acked Write 7 Sent Done Done false
21441 6555 Acked Write 10 Sent Done Done false
21442 6556 Acked Read 10 Sent Done Done false
21443 6557 Acked Write 3 Sent Done Done false
21444 6558 Acked Read 8 Sent Done Done false
21445 6559 Acked Write 2 Sent Done Done false
21446 6560 Acked Write 7 Sent Done Done false
21447 6561 Acked Write 1 Sent Done Done false
21448 6562 Acked Read 3 Sent Done Done false
21449 6563 Acked Read 9 Sent Done Done false
21450 6564 Acked Read 8 Sent Done Done false
21451 6565 Acked Flush 0 Sent Done Done false
21452 6566 Acked Read 4 Sent Done Done false
21453 6567 Acked Write 8 Sent Done Done false
21454 6568 Acked Read 2 Sent Done Done false
21455 6569 Acked Flush 0 Sent Done Done false
21456 6570 Acked Write 6 Sent Done Done false
21457 6571 Acked Read 5 Sent Done Done false
21458 6572 Acked Flush 0 Sent Done Done false
21459 6573 Acked Write 2 Sent Done Done false
21460 6574 Acked Read 1 Sent Done Done false
21461 6575 Acked Read 9 Sent Done Done false
21462 6576 Acked Read 7 Sent Done Done false
21463 6577 Acked Write 5 Sent Done Done false
21464 6578 Acked Read 3 Sent Done Done false
21465 6579 Acked Read 5 Sent Done Done false
21466 6580 Acked Read 2 Sent Done Done false
21467 6581 Acked Read 4 Sent Done Done false
21468 6582 Acked Read 7 Sent Done Done false
21469 6583 Acked Write 3 Sent Done Done false
21470 6584 Acked Write 3 Sent Done Done false
21471 6585 Acked Write 10 Sent Done Done false
21472 6586 Acked Read 1 Sent Done Done false
21473 6587 Acked Write 8 Sent Done Done false
21474 6588 Acked Write 6 Sent Done Done false
21475 6589 Acked Write 9 Sent Done Done false
21476 6590 Acked Write 9 Sent Done Done false
21477 6591 Acked Write 5 Sent Done Done false
21478 6592 Acked Read 7 Sent Done Done false
21479 6593 Acked Write 5 Sent Done Done false
21480 6594 Acked Write 9 Sent Done Done false
21481 6595 Acked Write 10 Sent Done Done false
21482 6596 Acked Flush 0 Sent Done Done false
21483 6597 Acked Read 2 Sent Done Done false
21484 6598 Acked Write 10 Sent Done Done false
21485 6599 Acked Write 2 Sent Done Done false
21486 6600 Acked Write 10 Sent Done Done false
21487 6601 Acked Read 4 Sent Done Done false
21488 6602 Acked Read 3 Sent Done Done false
21489 6603 Acked Read 3 Sent Done Done false
21490 6604 Acked Read 8 Sent Done Done false
21491 6605 Acked Flush 0 Sent Done Done false
21492 6606 Acked Write 3 Sent Done Done false
21493 6607 Acked Write 10 Sent Done Done false
21494 6608 Acked Read 9 Sent Done Done false
21495 6609 Acked Write 2 Sent Done Done false
21496 6610 Acked Read 5 Sent Done Done false
21497 6611 Acked Write 3 Sent Done Done false
21498 6612 Acked Read 6 Sent Done Done false
21499 6613 Acked Read 9 Sent Done Done false
21500 6614 Acked Write 2 Sent Done Done false
21501 6615 Acked Write 9 Sent Done Done false
21502 6616 Acked Flush 0 Sent Done Done false
21503 6617 Acked Write 8 Sent Done Done false
21504 6618 Acked Read 4 Sent Done Done false
21505 6619 Acked Read 6 Sent Done Done false
21506 6620 Acked Read 9 Sent Done Done false
21507 6621 Acked Read 4 Sent Done Done false
21508 6622 Acked Read 2 Sent Done Done false
21509 6623 Acked Write 10 Sent Done Done false
21510 6624 Acked Read 10 Sent Done Done false
21511 6625 Acked Write 9 Sent Done Done false
21512 6626 Acked Flush 0 Sent Done Done false
21513 6627 Acked Read 1 Sent Done Done false
21514 6628 Acked Flush 0 Sent Done Done false
21515 6629 Acked Read 1 Sent Done Done false
21516 6630 Acked Read 3 Sent Done Done false
21517 6631 Acked Flush 0 Sent Done Done false
21518 6632 Acked Read 2 Sent Done Done false
21519 6633 Acked Read 7 Sent Done Done false
21520 6634 Acked Read 2 Sent Done Done false
21521 6635 Acked Flush 0 Sent Done Done false
21522 6636 Acked Read 1 Sent Done Done false
21523 6637 Acked Flush 0 Sent Done Done false
21524 6638 Acked Write 2 Sent Done Done false
21525 6639 Acked Read 2 Sent Done Done false
21526 6640 Acked Write 2 Sent Done Done false
21527 6641 Acked Read 2 Sent Done Done false
21528 6642 Acked Write 7 Sent Done Done false
21529 6643 Acked Read 1 Sent Done Done false
21530 6644 Acked Read 1 Sent Done Done false
21531 6645 Acked Read 10 Sent Done Done false
21532 6646 Acked Write 5 Sent Done Done false
21533 6647 Acked Flush 0 Sent Done Done false
21534 6648 Acked Read 3 Sent Done Done false
21535 6649 Acked Read 5 Sent Done Done false
21536 6650 Acked Read 3 Sent Done Done false
21537 6651 Acked Write 9 Sent Done Done false
21538 6652 Acked Read 6 Sent Done Done false
21539 6653 Acked Flush 0 Sent Done Done false
21540 6654 Acked Read 1 Sent Done Done false
21541 6655 Acked Read 5 Sent Done Done false
21542 6656 Acked Read 4 Sent Done Done false
21543 6657 Acked Read 5 Sent Done Done false
21544 6658 Acked Read 5 Sent Done Done false
21545 6659 Acked Write 10 Sent Done Done false
21546 6660 Acked Write 7 Sent Done Done false
21547 6661 Acked Read 4 Sent Done Done false
21548 6662 Acked Read 5 Sent Done Done false
21549 6663 Acked Read 8 Sent Done Done false
21550 6664 Acked Read 3 Sent Done Done false
21551 6665 Acked Read 3 Sent Done Done false
21552 6666 Acked Read 8 Sent Done Done false
21553 6667 Acked Flush 0 Sent Done Done false
21554 6668 Acked Read 7 Sent Done Done false
21555 6669 Acked Write 5 Sent Done Done false
21556 6670 Acked Read 1 Sent Done Done false
21557 6671 Acked Read 8 Sent Done Done false
21558 6672 Acked Flush 0 Sent Done Done false
21559 6673 Acked Write 10 Sent Done Done false
21560 6674 Acked Write 9 Sent Done Done false
21561 6675 Acked Read 8 Sent Done Done false
21562 6676 Acked Write 1 Sent Done Done false
21563 6677 Acked Write 2 Sent Done Done false
21564 6678 Acked Read 10 Sent Done Done false
21565 6679 Acked Read 4 Sent Done Done false
21566 6680 Acked Read 10 Sent Done Done false
21567 6681 Acked Read 5 Sent Done Done false
21568 6682 Acked Write 8 Sent Done Done false
21569 6683 Acked Read 3 Sent Done Done false
21570 6684 Acked Read 3 Sent Done Done false
21571 6685 Acked Read 2 Sent Done Done false
21572 6686 Acked Flush 0 Sent Done Done false
21573 6687 Acked Read 1 Sent Done Done false
21574 6688 Acked Read 6 Sent Done Done false
21575 6689 Acked Read 5 Sent Done Done false
21576 6690 Acked Read 5 Sent Done Done false
21577 6691 Acked Read 8 Sent Done Done false
21578 6692 Acked Read 5 Sent Done Done false
21579 6693 Acked Write 2 Sent Done Done false
21580 6694 Acked Read 4 Sent Done Done false
21581 6695 Acked Read 9 Sent Done Done false
21582 6696 Acked Write 5 Sent Done Done false
21583 6697 Acked Read 2 Sent Done Done false
21584 6698 Acked Write 10 Sent Done Done false
21585 6699 Acked Write 9 Sent Done Done false
21586 6700 Acked Write 4 Sent Done Done false
21587 6701 Acked Write 4 Sent Done Done false
21588 6702 Acked Read 2 Sent Done Done false
21589 6703 Acked Write 8 Sent Done Done false
21590 6704 Acked Read 9 Sent Done Done false
21591 6705 Acked Write 7 Sent Done Done false
21592 6706 Acked Write 7 Sent Done Done false
21593 6707 Acked Read 4 Sent Done Done false
21594 6708 Acked Read 9 Sent Done Done false
21595 6709 Acked Flush 0 Sent Done Done false
21596 6710 Acked Read 1 Sent Done Done false
21597 6711 Acked Read 1 Sent Done Done false
21598 6712 Acked Write 6 Sent Done Done false
21599 6713 Acked Read 2 Sent Done Done false
21600 6714 Acked Read 9 Sent Done Done false
21601 6715 Acked Write 2 Sent Done Done false
21602 6716 Acked Read 5 Sent Done Done false
21603 6717 Acked Flush 0 Sent Done Done false
21604 6718 Acked Read 6 Sent Done Done false
21605 6719 Acked Write 6 Sent Done Done false
21606 6720 Acked Write 4 Sent Done Done false
21607 6721 Acked Read 10 Sent Done Done false
21608 6722 Acked Write 4 Sent Done Done false
21609 6723 Acked Read 10 Sent Done Done false
21610 6724 Acked Read 1 Sent Done Done false
21611 6725 Acked Write 7 Sent Done Done false
21612 6726 Acked Write 9 Sent Done Done false
21613 6727 Acked Read 3 Sent Done Done false
21614 6728 Acked Read 2 Sent Done Done false
21615 6729 Acked Write 4 Sent Done Done false
21616 6730 Acked Read 4 Sent Done Done false
21617 6731 Acked Read 10 Sent Done Done false
21618 6732 Acked Read 7 Sent Done Done false
21619 6733 Acked Write 5 Sent Done Done false
21620 6734 Acked Read 6 Sent Done Done false
21621 STATES DS:0 DS:1 DS:2 TOTAL
21622 Sent 301 0 0 301
21623 Done 0 301 301 602
21624 Skipped 0 0 0 0
21625 Error 0 0 0 0
21626 Last Flush: Some(JobId(6433)) Some(JobId(6717)) Some(JobId(6717))
21627 Downstairs last five completed: 6433 6432 6431 6430 6429
21628 Upstairs last five completed: 6734 6733 6732 6731 6730
21629 CLIENT: Up:0 ds:301 act:2
216302025-02-25T20:51:17.987ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216312025-02-25T20:51:18.503ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216322025-02-25T20:51:19.235ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216332025-02-25T20:51:19.235ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216342025-02-25T20:51:19.235ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216352025-02-25T20:51:19.235ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
216362025-02-25T20:51:19.235ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21637 [8810] got start action
21638 [8810] I am going to restart
21639 Starting downstairs at port 8810
21640 Make output file at "/tmp/dsc/downstairs-8810.txt"
21641 [8810][0] reports Starting
21642 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225)
21643 [8810][0] reports Running
216442025-02-25T20:51:20.507ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21645 ----------------------------------------------------------------
21646 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21647 States: Offline Active Active
21648 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21649 6434 Acked Write 2 Sent Done Done false
21650 6435 Acked Write 2 Sent Done Done false
21651 6436 Acked Read 1 Sent Done Done false
21652 6437 Acked Write 7 Sent Done Done false
21653 6438 Acked Read 5 Sent Done Done false
21654 6439 Acked Read 4 Sent Done Done false
21655 6440 Acked Read 1 Sent Done Done false
21656 6441 Acked Flush 0 Sent Done Done false
21657 6442 Acked Write 8 Sent Done Done false
21658 6443 Acked Write 5 Sent Done Done false
21659 6444 Acked Write 10 Sent Done Done false
21660 6445 Acked Write 8 Sent Done Done false
21661 6446 Acked Write 4 Sent Done Done false
21662 6447 Acked Read 10 Sent Done Done false
21663 6448 Acked Write 1 Sent Done Done false
21664 6449 Acked Write 10 Sent Done Done false
21665 6450 Acked Read 10 Sent Done Done false
21666 6451 Acked Read 6 Sent Done Done false
21667 6452 Acked Read 1 Sent Done Done false
21668 6453 Acked Read 10 Sent Done Done false
21669 6454 Acked Read 4 Sent Done Done false
21670 6455 Acked Read 4 Sent Done Done false
21671 6456 Acked Write 5 Sent Done Done false
21672 6457 Acked Write 9 Sent Done Done false
21673 6458 Acked Read 8 Sent Done Done false
21674 6459 Acked Write 2 Sent Done Done false
21675 6460 Acked Write 9 Sent Done Done false
21676 6461 Acked Write 5 Sent Done Done false
21677 6462 Acked Flush 0 Sent Done Done false
21678 6463 Acked Read 6 Sent Done Done false
21679 6464 Acked Read 4 Sent Done Done false
21680 6465 Acked Write 5 Sent Done Done false
21681 6466 Acked Write 3 Sent Done Done false
21682 6467 Acked Read 2 Sent Done Done false
21683 6468 Acked Read 4 Sent Done Done false
21684 6469 Acked Read 2 Sent Done Done false
21685 6470 Acked Read 3 Sent Done Done false
21686 6471 Acked Read 1 Sent Done Done false
21687 6472 Acked Read 1 Sent Done Done false
21688 6473 Acked Write 7 Sent Done Done false
21689 6474 Acked Write 8 Sent Done Done false
21690 6475 Acked Write 2 Sent Done Done false
21691 6476 Acked Write 5 Sent Done Done false
21692 6477 Acked Write 9 Sent Done Done false
21693 6478 Acked Read 1 Sent Done Done false
21694 6479 Acked Read 6 Sent Done Done false
21695 6480 Acked Read 10 Sent Done Done false
21696 6481 Acked Read 10 Sent Done Done false
21697 6482 Acked Write 2 Sent Done Done false
21698 6483 Acked Write 1 Sent Done Done false
21699 6484 Acked Write 9 Sent Done Done false
21700 6485 Acked Read 5 Sent Done Done false
21701 6486 Acked Write 2 Sent Done Done false
21702 6487 Acked Read 6 Sent Done Done false
21703 6488 Acked Flush 0 Sent Done Done false
21704 6489 Acked Write 8 Sent Done Done false
21705 6490 Acked Read 1 Sent Done Done false
21706 6491 Acked Write 2 Sent Done Done false
21707 6492 Acked Read 5 Sent Done Done false
21708 6493 Acked Read 5 Sent Done Done false
21709 6494 Acked Read 3 Sent Done Done false
21710 6495 Acked Read 3 Sent Done Done false
21711 6496 Acked Read 4 Sent Done Done false
21712 6497 Acked Write 8 Sent Done Done false
21713 6498 Acked Read 5 Sent Done Done false
21714 6499 Acked Write 7 Sent Done Done false
21715 6500 Acked Read 5 Sent Done Done false
21716 6501 Acked Flush 0 Sent Done Done false
21717 6502 Acked Write 7 Sent Done Done false
21718 6503 Acked Read 5 Sent Done Done false
21719 6504 Acked Write 7 Sent Done Done false
21720 6505 Acked Read 4 Sent Done Done false
21721 6506 Acked Read 9 Sent Done Done false
21722 6507 Acked Read 8 Sent Done Done false
21723 6508 Acked Read 6 Sent Done Done false
21724 6509 Acked Read 1 Sent Done Done false
21725 6510 Acked Write 9 Sent Done Done false
21726 6511 Acked Read 10 Sent Done Done false
21727 6512 Acked Flush 0 Sent Done Done false
21728 6513 Acked Read 4 Sent Done Done false
21729 6514 Acked Read 8 Sent Done Done false
21730 6515 Acked Write 6 Sent Done Done false
21731 6516 Acked Write 6 Sent Done Done false
21732 6517 Acked Read 3 Sent Done Done false
21733 6518 Acked Write 10 Sent Done Done false
21734 6519 Acked Flush 0 Sent Done Done false
21735 6520 Acked Write 10 Sent Done Done false
21736 6521 Acked Read 5 Sent Done Done false
21737 6522 Acked Flush 0 Sent Done Done false
21738 6523 Acked Write 9 Sent Done Done false
21739 6524 Acked Write 8 Sent Done Done false
21740 6525 Acked Read 8 Sent Done Done false
21741 6526 Acked Read 7 Sent Done Done false
21742 6527 Acked Read 8 Sent Done Done false
21743 6528 Acked Read 4 Sent Done Done false
21744 6529 Acked Write 4 Sent Done Done false
21745 6530 Acked Write 2 Sent Done Done false
21746 6531 Acked Read 2 Sent Done Done false
21747 6532 Acked Flush 0 Sent Done Done false
21748 6533 Acked Write 8 Sent Done Done false
21749 6534 Acked Read 9 Sent Done Done false
21750 6535 Acked Write 8 Sent Done Done false
21751 6536 Acked Flush 0 Sent Done Done false
21752 6537 Acked Read 1 Sent Done Done false
21753 6538 Acked Write 8 Sent Done Done false
21754 6539 Acked Read 1 Sent Done Done false
21755 6540 Acked Read 2 Sent Done Done false
21756 6541 Acked Read 7 Sent Done Done false
21757 6542 Acked Write 10 Sent Done Done false
21758 6543 Acked Write 5 Sent Done Done false
21759 6544 Acked Read 2 Sent Done Done false
21760 6545 Acked Write 3 Sent Done Done false
21761 6546 Acked Read 10 Sent Done Done false
21762 6547 Acked Write 7 Sent Done Done false
21763 6548 Acked Flush 0 Sent Done Done false
21764 6549 Acked Read 8 Sent Done Done false
21765 6550 Acked Flush 0 Sent Done Done false
21766 6551 Acked Read 6 Sent Done Done false
21767 6552 Acked Write 7 Sent Done Done false
21768 6553 Acked Write 6 Sent Done Done false
21769 6554 Acked Write 7 Sent Done Done false
21770 6555 Acked Write 10 Sent Done Done false
21771 6556 Acked Read 10 Sent Done Done false
21772 6557 Acked Write 3 Sent Done Done false
21773 6558 Acked Read 8 Sent Done Done false
21774 6559 Acked Write 2 Sent Done Done false
21775 6560 Acked Write 7 Sent Done Done false
21776 6561 Acked Write 1 Sent Done Done false
21777 6562 Acked Read 3 Sent Done Done false
21778 6563 Acked Read 9 Sent Done Done false
21779 6564 Acked Read 8 Sent Done Done false
21780 6565 Acked Flush 0 Sent Done Done false
21781 6566 Acked Read 4 Sent Done Done false
21782 6567 Acked Write 8 Sent Done Done false
21783 6568 Acked Read 2 Sent Done Done false
21784 6569 Acked Flush 0 Sent Done Done false
21785 6570 Acked Write 6 Sent Done Done false
21786 6571 Acked Read 5 Sent Done Done false
21787 6572 Acked Flush 0 Sent Done Done false
21788 6573 Acked Write 2 Sent Done Done false
21789 6574 Acked Read 1 Sent Done Done false
21790 6575 Acked Read 9 Sent Done Done false
21791 6576 Acked Read 7 Sent Done Done false
21792 6577 Acked Write 5 Sent Done Done false
21793 6578 Acked Read 3 Sent Done Done false
21794 6579 Acked Read 5 Sent Done Done false
21795 6580 Acked Read 2 Sent Done Done false
21796 6581 Acked Read 4 Sent Done Done false
21797 6582 Acked Read 7 Sent Done Done false
21798 6583 Acked Write 3 Sent Done Done false
21799 6584 Acked Write 3 Sent Done Done false
21800 6585 Acked Write 10 Sent Done Done false
21801 6586 Acked Read 1 Sent Done Done false
21802 6587 Acked Write 8 Sent Done Done false
21803 6588 Acked Write 6 Sent Done Done false
21804 6589 Acked Write 9 Sent Done Done false
21805 6590 Acked Write 9 Sent Done Done false
21806 6591 Acked Write 5 Sent Done Done false
21807 6592 Acked Read 7 Sent Done Done false
21808 6593 Acked Write 5 Sent Done Done false
21809 6594 Acked Write 9 Sent Done Done false
21810 6595 Acked Write 10 Sent Done Done false
21811 6596 Acked Flush 0 Sent Done Done false
21812 6597 Acked Read 2 Sent Done Done false
21813 6598 Acked Write 10 Sent Done Done false
21814 6599 Acked Write 2 Sent Done Done false
21815 6600 Acked Write 10 Sent Done Done false
21816 6601 Acked Read 4 Sent Done Done false
21817 6602 Acked Read 3 Sent Done Done false
21818 6603 Acked Read 3 Sent Done Done false
21819 6604 Acked Read 8 Sent Done Done false
21820 6605 Acked Flush 0 Sent Done Done false
21821 6606 Acked Write 3 Sent Done Done false
21822 6607 Acked Write 10 Sent Done Done false
21823 6608 Acked Read 9 Sent Done Done false
21824 6609 Acked Write 2 Sent Done Done false
21825 6610 Acked Read 5 Sent Done Done false
21826 6611 Acked Write 3 Sent Done Done false
21827 6612 Acked Read 6 Sent Done Done false
21828 6613 Acked Read 9 Sent Done Done false
21829 6614 Acked Write 2 Sent Done Done false
21830 6615 Acked Write 9 Sent Done Done false
21831 6616 Acked Flush 0 Sent Done Done false
21832 6617 Acked Write 8 Sent Done Done false
21833 6618 Acked Read 4 Sent Done Done false
21834 6619 Acked Read 6 Sent Done Done false
21835 6620 Acked Read 9 Sent Done Done false
21836 6621 Acked Read 4 Sent Done Done false
21837 6622 Acked Read 2 Sent Done Done false
21838 6623 Acked Write 10 Sent Done Done false
21839 6624 Acked Read 10 Sent Done Done false
21840 6625 Acked Write 9 Sent Done Done false
21841 6626 Acked Flush 0 Sent Done Done false
21842 6627 Acked Read 1 Sent Done Done false
21843 6628 Acked Flush 0 Sent Done Done false
21844 6629 Acked Read 1 Sent Done Done false
21845 6630 Acked Read 3 Sent Done Done false
21846 6631 Acked Flush 0 Sent Done Done false
21847 6632 Acked Read 2 Sent Done Done false
21848 6633 Acked Read 7 Sent Done Done false
21849 6634 Acked Read 2 Sent Done Done false
21850 6635 Acked Flush 0 Sent Done Done false
21851 6636 Acked Read 1 Sent Done Done false
21852 6637 Acked Flush 0 Sent Done Done false
21853 6638 Acked Write 2 Sent Done Done false
21854 6639 Acked Read 2 Sent Done Done false
21855 6640 Acked Write 2 Sent Done Done false
21856 6641 Acked Read 2 Sent Done Done false
21857 6642 Acked Write 7 Sent Done Done false
21858 6643 Acked Read 1 Sent Done Done false
21859 6644 Acked Read 1 Sent Done Done false
21860 6645 Acked Read 10 Sent Done Done false
21861 6646 Acked Write 5 Sent Done Done false
21862 6647 Acked Flush 0 Sent Done Done false
21863 6648 Acked Read 3 Sent Done Done false
21864 6649 Acked Read 5 Sent Done Done false
21865 6650 Acked Read 3 Sent Done Done false
21866 6651 Acked Write 9 Sent Done Done false
21867 6652 Acked Read 6 Sent Done Done false
21868 6653 Acked Flush 0 Sent Done Done false
21869 6654 Acked Read 1 Sent Done Done false
21870 6655 Acked Read 5 Sent Done Done false
21871 6656 Acked Read 4 Sent Done Done false
21872 6657 Acked Read 5 Sent Done Done false
21873 6658 Acked Read 5 Sent Done Done false
21874 6659 Acked Write 10 Sent Done Done false
21875 6660 Acked Write 7 Sent Done Done false
21876 6661 Acked Read 4 Sent Done Done false
21877 6662 Acked Read 5 Sent Done Done false
21878 6663 Acked Read 8 Sent Done Done false
21879 6664 Acked Read 3 Sent Done Done false
21880 6665 Acked Read 3 Sent Done Done false
21881 6666 Acked Read 8 Sent Done Done false
21882 6667 Acked Flush 0 Sent Done Done false
21883 6668 Acked Read 7 Sent Done Done false
21884 6669 Acked Write 5 Sent Done Done false
21885 6670 Acked Read 1 Sent Done Done false
21886 6671 Acked Read 8 Sent Done Done false
21887 6672 Acked Flush 0 Sent Done Done false
21888 6673 Acked Write 10 Sent Done Done false
21889 6674 Acked Write 9 Sent Done Done false
21890 6675 Acked Read 8 Sent Done Done false
21891 6676 Acked Write 1 Sent Done Done false
21892 6677 Acked Write 2 Sent Done Done false
21893 6678 Acked Read 10 Sent Done Done false
21894 6679 Acked Read 4 Sent Done Done false
21895 6680 Acked Read 10 Sent Done Done false
21896 6681 Acked Read 5 Sent Done Done false
21897 6682 Acked Write 8 Sent Done Done false
21898 6683 Acked Read 3 Sent Done Done false
21899 6684 Acked Read 3 Sent Done Done false
21900 6685 Acked Read 2 Sent Done Done false
21901 6686 Acked Flush 0 Sent Done Done false
21902 6687 Acked Read 1 Sent Done Done false
21903 6688 Acked Read 6 Sent Done Done false
21904 6689 Acked Read 5 Sent Done Done false
21905 6690 Acked Read 5 Sent Done Done false
21906 6691 Acked Read 8 Sent Done Done false
21907 6692 Acked Read 5 Sent Done Done false
21908 6693 Acked Write 2 Sent Done Done false
21909 6694 Acked Read 4 Sent Done Done false
21910 6695 Acked Read 9 Sent Done Done false
21911 6696 Acked Write 5 Sent Done Done false
21912 6697 Acked Read 2 Sent Done Done false
21913 6698 Acked Write 10 Sent Done Done false
21914 6699 Acked Write 9 Sent Done Done false
21915 6700 Acked Write 4 Sent Done Done false
21916 6701 Acked Write 4 Sent Done Done false
21917 6702 Acked Read 2 Sent Done Done false
21918 6703 Acked Write 8 Sent Done Done false
21919 6704 Acked Read 9 Sent Done Done false
21920 6705 Acked Write 7 Sent Done Done false
21921 6706 Acked Write 7 Sent Done Done false
21922 6707 Acked Read 4 Sent Done Done false
21923 6708 Acked Read 9 Sent Done Done false
21924 6709 Acked Flush 0 Sent Done Done false
21925 6710 Acked Read 1 Sent Done Done false
21926 6711 Acked Read 1 Sent Done Done false
21927 6712 Acked Write 6 Sent Done Done false
21928 6713 Acked Read 2 Sent Done Done false
21929 6714 Acked Read 9 Sent Done Done false
21930 6715 Acked Write 2 Sent Done Done false
21931 6716 Acked Read 5 Sent Done Done false
21932 6717 Acked Flush 0 Sent Done Done false
21933 6718 Acked Read 6 Sent Done Done false
21934 6719 Acked Write 6 Sent Done Done false
21935 6720 Acked Write 4 Sent Done Done false
21936 6721 Acked Read 10 Sent Done Done false
21937 6722 Acked Write 4 Sent Done Done false
21938 6723 Acked Read 10 Sent Done Done false
21939 6724 Acked Read 1 Sent Done Done false
21940 6725 Acked Write 7 Sent Done Done false
21941 6726 Acked Write 9 Sent Done Done false
21942 6727 Acked Read 3 Sent Done Done false
21943 6728 Acked Read 2 Sent Done Done false
21944 6729 Acked Write 4 Sent Done Done false
21945 6730 Acked Read 4 Sent Done Done false
21946 6731 Acked Read 10 Sent Done Done false
21947 6732 Acked Read 7 Sent Done Done false
21948 6733 Acked Write 5 Sent Done Done false
21949 6734 Acked Read 6 Sent Done Done false
21950 STATES DS:0 DS:1 DS:2 TOTAL
21951 Sent 301 0 0 301
21952 Done 0 301 301 602
21953 Skipped 0 0 0 0
21954 Error 0 0 0 0
21955 Last Flush: Some(JobId(6433)) Some(JobId(6717)) Some(JobId(6717))
21956 Downstairs last five completed: 6433 6432 6431 6430 6429
21957 Upstairs last five completed: 6734 6733 6732 6731 6730
21958 CLIENT: Up:0 ds:301 act:2
219592025-02-25T20:51:21.989ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
219602025-02-25T20:51:22.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
219612025-02-25T20:51:23.511ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
219622025-02-25T20:51:24.513ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
21963 ----------------------------------------------------------------
21964 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21965 States: Offline Active Active
21966 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21967 6434 Acked Write 2 Sent Done Done false
21968 6435 Acked Write 2 Sent Done Done false
21969 6436 Acked Read 1 Sent Done Done false
21970 6437 Acked Write 7 Sent Done Done false
21971 6438 Acked Read 5 Sent Done Done false
21972 6439 Acked Read 4 Sent Done Done false
21973 6440 Acked Read 1 Sent Done Done false
21974 6441 Acked Flush 0 Sent Done Done false
21975 6442 Acked Write 8 Sent Done Done false
21976 6443 Acked Write 5 Sent Done Done false
21977 6444 Acked Write 10 Sent Done Done false
21978 6445 Acked Write 8 Sent Done Done false
21979 6446 Acked Write 4 Sent Done Done false
21980 6447 Acked Read 10 Sent Done Done false
21981 6448 Acked Write 1 Sent Done Done false
21982 6449 Acked Write 10 Sent Done Done false
21983 6450 Acked Read 10 Sent Done Done false
21984 6451 Acked Read 6 Sent Done Done false
21985 6452 Acked Read 1 Sent Done Done false
21986 6453 Acked Read 10 Sent Done Done false
21987 6454 Acked Read 4 Sent Done Done false
21988 6455 Acked Read 4 Sent Done Done false
21989 6456 Acked Write 5 Sent Done Done false
21990 6457 Acked Write 9 Sent Done Done false
21991 6458 Acked Read 8 Sent Done Done false
21992 6459 Acked Write 2 Sent Done Done false
21993 6460 Acked Write 9 Sent Done Done false
21994 6461 Acked Write 5 Sent Done Done false
21995 6462 Acked Flush 0 Sent Done Done false
21996 6463 Acked Read 6 Sent Done Done false
21997 6464 Acked Read 4 Sent Done Done false
21998 6465 Acked Write 5 Sent Done Done false
21999 6466 Acked Write 3 Sent Done Done false
22000 6467 Acked Read 2 Sent Done Done false
22001 6468 Acked Read 4 Sent Done Done false
22002 6469 Acked Read 2 Sent Done Done false
22003 6470 Acked Read 3 Sent Done Done false
22004 6471 Acked Read 1 Sent Done Done false
22005 6472 Acked Read 1 Sent Done Done false
22006 6473 Acked Write 7 Sent Done Done false
22007 6474 Acked Write 8 Sent Done Done false
22008 6475 Acked Write 2 Sent Done Done false
22009 6476 Acked Write 5 Sent Done Done false
22010 6477 Acked Write 9 Sent Done Done false
22011 6478 Acked Read 1 Sent Done Done false
22012 6479 Acked Read 6 Sent Done Done false
22013 6480 Acked Read 10 Sent Done Done false
22014 6481 Acked Read 10 Sent Done Done false
22015 6482 Acked Write 2 Sent Done Done false
22016 6483 Acked Write 1 Sent Done Done false
22017 6484 Acked Write 9 Sent Done Done false
22018 6485 Acked Read 5 Sent Done Done false
22019 6486 Acked Write 2 Sent Done Done false
22020 6487 Acked Read 6 Sent Done Done false
22021 6488 Acked Flush 0 Sent Done Done false
22022 6489 Acked Write 8 Sent Done Done false
22023 6490 Acked Read 1 Sent Done Done false
22024 6491 Acked Write 2 Sent Done Done false
22025 6492 Acked Read 5 Sent Done Done false
22026 6493 Acked Read 5 Sent Done Done false
22027 6494 Acked Read 3 Sent Done Done false
22028 6495 Acked Read 3 Sent Done Done false
22029 6496 Acked Read 4 Sent Done Done false
22030 6497 Acked Write 8 Sent Done Done false
22031 6498 Acked Read 5 Sent Done Done false
22032 6499 Acked Write 7 Sent Done Done false
22033 6500 Acked Read 5 Sent Done Done false
22034 6501 Acked Flush 0 Sent Done Done false
22035 6502 Acked Write 7 Sent Done Done false
22036 6503 Acked Read 5 Sent Done Done false
22037 6504 Acked Write 7 Sent Done Done false
22038 6505 Acked Read 4 Sent Done Done false
22039 6506 Acked Read 9 Sent Done Done false
22040 6507 Acked Read 8 Sent Done Done false
22041 6508 Acked Read 6 Sent Done Done false
22042 6509 Acked Read 1 Sent Done Done false
22043 6510 Acked Write 9 Sent Done Done false
22044 6511 Acked Read 10 Sent Done Done false
22045 6512 Acked Flush 0 Sent Done Done false
22046 6513 Acked Read 4 Sent Done Done false
22047 6514 Acked Read 8 Sent Done Done false
22048 6515 Acked Write 6 Sent Done Done false
22049 6516 Acked Write 6 Sent Done Done false
22050 6517 Acked Read 3 Sent Done Done false
22051 6518 Acked Write 10 Sent Done Done false
22052 6519 Acked Flush 0 Sent Done Done false
22053 6520 Acked Write 10 Sent Done Done false
22054 6521 Acked Read 5 Sent Done Done false
22055 6522 Acked Flush 0 Sent Done Done false
22056 6523 Acked Write 9 Sent Done Done false
22057 6524 Acked Write 8 Sent Done Done false
22058 6525 Acked Read 8 Sent Done Done false
22059 6526 Acked Read 7 Sent Done Done false
22060 6527 Acked Read 8 Sent Done Done false
22061 6528 Acked Read 4 Sent Done Done false
22062 6529 Acked Write 4 Sent Done Done false
22063 6530 Acked Write 2 Sent Done Done false
22064 6531 Acked Read 2 Sent Done Done false
22065 6532 Acked Flush 0 Sent Done Done false
22066 6533 Acked Write 8 Sent Done Done false
22067 6534 Acked Read 9 Sent Done Done false
22068 6535 Acked Write 8 Sent Done Done false
22069 6536 Acked Flush 0 Sent Done Done false
22070 6537 Acked Read 1 Sent Done Done false
22071 6538 Acked Write 8 Sent Done Done false
22072 6539 Acked Read 1 Sent Done Done false
22073 6540 Acked Read 2 Sent Done Done false
22074 6541 Acked Read 7 Sent Done Done false
22075 6542 Acked Write 10 Sent Done Done false
22076 6543 Acked Write 5 Sent Done Done false
22077 6544 Acked Read 2 Sent Done Done false
22078 6545 Acked Write 3 Sent Done Done false
22079 6546 Acked Read 10 Sent Done Done false
22080 6547 Acked Write 7 Sent Done Done false
22081 6548 Acked Flush 0 Sent Done Done false
22082 6549 Acked Read 8 Sent Done Done false
22083 6550 Acked Flush 0 Sent Done Done false
22084 6551 Acked Read 6 Sent Done Done false
22085 6552 Acked Write 7 Sent Done Done false
22086 6553 Acked Write 6 Sent Done Done false
22087 6554 Acked Write 7 Sent Done Done false
22088 6555 Acked Write 10 Sent Done Done false
22089 6556 Acked Read 10 Sent Done Done false
22090 6557 Acked Write 3 Sent Done Done false
22091 6558 Acked Read 8 Sent Done Done false
22092 6559 Acked Write 2 Sent Done Done false
22093 6560 Acked Write 7 Sent Done Done false
22094 6561 Acked Write 1 Sent Done Done false
22095 6562 Acked Read 3 Sent Done Done false
22096 6563 Acked Read 9 Sent Done Done false
22097 6564 Acked Read 8 Sent Done Done false
22098 6565 Acked Flush 0 Sent Done Done false
22099 6566 Acked Read 4 Sent Done Done false
22100 6567 Acked Write 8 Sent Done Done false
22101 6568 Acked Read 2 Sent Done Done false
22102 6569 Acked Flush 0 Sent Done Done false
22103 6570 Acked Write 6 Sent Done Done false
22104 6571 Acked Read 5 Sent Done Done false
22105 6572 Acked Flush 0 Sent Done Done false
22106 6573 Acked Write 2 Sent Done Done false
22107 6574 Acked Read 1 Sent Done Done false
22108 6575 Acked Read 9 Sent Done Done false
22109 6576 Acked Read 7 Sent Done Done false
22110 6577 Acked Write 5 Sent Done Done false
22111 6578 Acked Read 3 Sent Done Done false
22112 6579 Acked Read 5 Sent Done Done false
22113 6580 Acked Read 2 Sent Done Done false
22114 6581 Acked Read 4 Sent Done Done false
22115 6582 Acked Read 7 Sent Done Done false
22116 6583 Acked Write 3 Sent Done Done false
22117 6584 Acked Write 3 Sent Done Done false
22118 6585 Acked Write 10 Sent Done Done false
22119 6586 Acked Read 1 Sent Done Done false
22120 6587 Acked Write 8 Sent Done Done false
22121 6588 Acked Write 6 Sent Done Done false
22122 6589 Acked Write 9 Sent Done Done false
22123 6590 Acked Write 9 Sent Done Done false
22124 6591 Acked Write 5 Sent Done Done false
22125 6592 Acked Read 7 Sent Done Done false
22126 6593 Acked Write 5 Sent Done Done false
22127 6594 Acked Write 9 Sent Done Done false
22128 6595 Acked Write 10 Sent Done Done false
22129 6596 Acked Flush 0 Sent Done Done false
22130 6597 Acked Read 2 Sent Done Done false
22131 6598 Acked Write 10 Sent Done Done false
22132 6599 Acked Write 2 Sent Done Done false
22133 6600 Acked Write 10 Sent Done Done false
22134 6601 Acked Read 4 Sent Done Done false
22135 6602 Acked Read 3 Sent Done Done false
22136 6603 Acked Read 3 Sent Done Done false
22137 6604 Acked Read 8 Sent Done Done false
22138 6605 Acked Flush 0 Sent Done Done false
22139 6606 Acked Write 3 Sent Done Done false
22140 6607 Acked Write 10 Sent Done Done false
22141 6608 Acked Read 9 Sent Done Done false
22142 6609 Acked Write 2 Sent Done Done false
22143 6610 Acked Read 5 Sent Done Done false
22144 6611 Acked Write 3 Sent Done Done false
22145 6612 Acked Read 6 Sent Done Done false
22146 6613 Acked Read 9 Sent Done Done false
22147 6614 Acked Write 2 Sent Done Done false
22148 6615 Acked Write 9 Sent Done Done false
22149 6616 Acked Flush 0 Sent Done Done false
22150 6617 Acked Write 8 Sent Done Done false
22151 6618 Acked Read 4 Sent Done Done false
22152 6619 Acked Read 6 Sent Done Done false
22153 6620 Acked Read 9 Sent Done Done false
22154 6621 Acked Read 4 Sent Done Done false
22155 6622 Acked Read 2 Sent Done Done false
22156 6623 Acked Write 10 Sent Done Done false
22157 6624 Acked Read 10 Sent Done Done false
22158 6625 Acked Write 9 Sent Done Done false
22159 6626 Acked Flush 0 Sent Done Done false
22160 6627 Acked Read 1 Sent Done Done false
22161 6628 Acked Flush 0 Sent Done Done false
22162 6629 Acked Read 1 Sent Done Done false
22163 6630 Acked Read 3 Sent Done Done false
22164 6631 Acked Flush 0 Sent Done Done false
22165 6632 Acked Read 2 Sent Done Done false
22166 6633 Acked Read 7 Sent Done Done false
22167 6634 Acked Read 2 Sent Done Done false
22168 6635 Acked Flush 0 Sent Done Done false
22169 6636 Acked Read 1 Sent Done Done false
22170 6637 Acked Flush 0 Sent Done Done false
22171 6638 Acked Write 2 Sent Done Done false
22172 6639 Acked Read 2 Sent Done Done false
22173 6640 Acked Write 2 Sent Done Done false
22174 6641 Acked Read 2 Sent Done Done false
22175 6642 Acked Write 7 Sent Done Done false
22176 6643 Acked Read 1 Sent Done Done false
22177 6644 Acked Read 1 Sent Done Done false
22178 6645 Acked Read 10 Sent Done Done false
22179 6646 Acked Write 5 Sent Done Done false
22180 6647 Acked Flush 0 Sent Done Done false
22181 6648 Acked Read 3 Sent Done Done false
22182 6649 Acked Read 5 Sent Done Done false
22183 6650 Acked Read 3 Sent Done Done false
22184 6651 Acked Write 9 Sent Done Done false
22185 6652 Acked Read 6 Sent Done Done false
22186 6653 Acked Flush 0 Sent Done Done false
22187 6654 Acked Read 1 Sent Done Done false
22188 6655 Acked Read 5 Sent Done Done false
22189 6656 Acked Read 4 Sent Done Done false
22190 6657 Acked Read 5 Sent Done Done false
22191 6658 Acked Read 5 Sent Done Done false
22192 6659 Acked Write 10 Sent Done Done false
22193 6660 Acked Write 7 Sent Done Done false
22194 6661 Acked Read 4 Sent Done Done false
22195 6662 Acked Read 5 Sent Done Done false
22196 6663 Acked Read 8 Sent Done Done false
22197 6664 Acked Read 3 Sent Done Done false
22198 6665 Acked Read 3 Sent Done Done false
22199 6666 Acked Read 8 Sent Done Done false
22200 6667 Acked Flush 0 Sent Done Done false
22201 6668 Acked Read 7 Sent Done Done false
22202 6669 Acked Write 5 Sent Done Done false
22203 6670 Acked Read 1 Sent Done Done false
22204 6671 Acked Read 8 Sent Done Done false
22205 6672 Acked Flush 0 Sent Done Done false
22206 6673 Acked Write 10 Sent Done Done false
22207 6674 Acked Write 9 Sent Done Done false
22208 6675 Acked Read 8 Sent Done Done false
22209 6676 Acked Write 1 Sent Done Done false
22210 6677 Acked Write 2 Sent Done Done false
22211 6678 Acked Read 10 Sent Done Done false
22212 6679 Acked Read 4 Sent Done Done false
22213 6680 Acked Read 10 Sent Done Done false
22214 6681 Acked Read 5 Sent Done Done false
22215 6682 Acked Write 8 Sent Done Done false
22216 6683 Acked Read 3 Sent Done Done false
22217 6684 Acked Read 3 Sent Done Done false
22218 6685 Acked Read 2 Sent Done Done false
22219 6686 Acked Flush 0 Sent Done Done false
22220 6687 Acked Read 1 Sent Done Done false
22221 6688 Acked Read 6 Sent Done Done false
22222 6689 Acked Read 5 Sent Done Done false
22223 6690 Acked Read 5 Sent Done Done false
22224 6691 Acked Read 8 Sent Done Done false
22225 6692 Acked Read 5 Sent Done Done false
22226 6693 Acked Write 2 Sent Done Done false
22227 6694 Acked Read 4 Sent Done Done false
22228 6695 Acked Read 9 Sent Done Done false
22229 6696 Acked Write 5 Sent Done Done false
22230 6697 Acked Read 2 Sent Done Done false
22231 6698 Acked Write 10 Sent Done Done false
22232 6699 Acked Write 9 Sent Done Done false
22233 6700 Acked Write 4 Sent Done Done false
22234 6701 Acked Write 4 Sent Done Done false
22235 6702 Acked Read 2 Sent Done Done false
22236 6703 Acked Write 8 Sent Done Done false
22237 6704 Acked Read 9 Sent Done Done false
22238 6705 Acked Write 7 Sent Done Done false
22239 6706 Acked Write 7 Sent Done Done false
22240 6707 Acked Read 4 Sent Done Done false
22241 6708 Acked Read 9 Sent Done Done false
22242 6709 Acked Flush 0 Sent Done Done false
22243 6710 Acked Read 1 Sent Done Done false
22244 6711 Acked Read 1 Sent Done Done false
22245 6712 Acked Write 6 Sent Done Done false
22246 6713 Acked Read 2 Sent Done Done false
22247 6714 Acked Read 9 Sent Done Done false
22248 6715 Acked Write 2 Sent Done Done false
22249 6716 Acked Read 5 Sent Done Done false
22250 6717 Acked Flush 0 Sent Done Done false
22251 6718 Acked Read 6 Sent Done Done false
22252 6719 Acked Write 6 Sent Done Done false
22253 6720 Acked Write 4 Sent Done Done false
22254 6721 Acked Read 10 Sent Done Done false
22255 6722 Acked Write 4 Sent Done Done false
22256 6723 Acked Read 10 Sent Done Done false
22257 6724 Acked Read 1 Sent Done Done false
22258 6725 Acked Write 7 Sent Done Done false
22259 6726 Acked Write 9 Sent Done Done false
22260 6727 Acked Read 3 Sent Done Done false
22261 6728 Acked Read 2 Sent Done Done false
22262 6729 Acked Write 4 Sent Done Done false
22263 6730 Acked Read 4 Sent Done Done false
22264 6731 Acked Read 10 Sent Done Done false
22265 6732 Acked Read 7 Sent Done Done false
22266 6733 Acked Write 5 Sent Done Done false
22267 6734 Acked Read 6 Sent Done Done false
22268 STATES DS:0 DS:1 DS:2 TOTAL
22269 Sent 301 0 0 301
22270 Done 0 301 301 602
22271 Skipped 0 0 0 0
22272 Error 0 0 0 0
22273 Last Flush: Some(JobId(6433)) Some(JobId(6717)) Some(JobId(6717))
22274 Downstairs last five completed: 6433 6432 6431 6430 6429
22275 Upstairs last five completed: 6734 6733 6732 6731 6730
22276 CLIENT: Up:0 ds:301 act:2
222772025-02-25T20:51:26.517ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222782025-02-25T20:51:27.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222792025-02-25T20:51:28.520ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222802025-02-25T20:51:29.237ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222812025-02-25T20:51:29.237ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61951) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222822025-02-25T20:51:29.238ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222832025-02-25T20:51:29.238ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222842025-02-25T20:51:29.238ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222852025-02-25T20:51:29.238ZINFOcrucible: send last flush ID to this DS: Some(JobId(6433)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222862025-02-25T20:51:29.238ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222872025-02-25T20:51:29.238ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6433)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
222882025-02-25T20:51:29.239ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6433)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
22289 ----------------------------------------------------------------
22290 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:17
22291 States: Active Active Active
22292 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22293 6718 Acked Read 6 Done Done Done true
22294 6719 Acked Write 6 Done Done Done true
22295 6720 Acked Write 4 Done Done Done true
22296 6721 Acked Read 10 Done Done Done true
22297 6722 Acked Write 4 Done Done Done true
22298 6723 Acked Read 10 Done Done Done true
22299 6724 Acked Read 1 Done Done Done true
22300 6725 Acked Write 7 Done Done Done true
22301 6726 Acked Write 9 Done Done Done true
22302 6727 Acked Read 3 Done Done Done true
22303 6728 Acked Read 2 Done Done Done true
22304 6729 Acked Write 4 Done Done Done true
22305 6730 Acked Read 4 Done Done Done true
22306 6731 Acked Read 10 Done Done Done true
22307 6732 Acked Read 7 Done Done Done true
22308 6733 Acked Write 5 Done Done Done true
22309 6734 Acked Read 6 Done Done Done true
22310 STATES DS:0 DS:1 DS:2 TOTAL
22311 Sent 0 0 0 0
22312 Done 17 17 17 51
22313 Skipped 0 0 0 0
22314 Error 0 0 0 0
22315 Last Flush: Some(JobId(6717)) Some(JobId(6717)) Some(JobId(6717))
22316 Downstairs last five completed: 6717 6716 6715 6714 6713
22317 Upstairs last five completed: 6734 6733 6732 6731 6730
22318 CLIENT: Up:0 ds:17 act:3
22319 ----------------------------------------------------------------
22320 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22321 Downstairs last five completed: 6735 6734 6733 6732 6731
22322 Upstairs last five completed: 6735 6734 6733 6732 6731
22323 CLIENT: Up:0 ds:0 act:3
22324 Replay: All jobs finished
22325 Main task has work to do, go find it
22326 got dsc Stop(0)
22327 stop 0
22328 [8810] Got stop action so:false kr:false
22329 Feb 25 20:51:33.494 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c1a88f9a-ae23-4d1e-96c8-f5e365d9ef8d, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
223302025-02-25T20:51:33.495ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
223312025-02-25T20:51:33.495ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
223322025-02-25T20:51:33.495ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
22333 Feb 25 20:51:33.495 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:51:33.495281369Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
22334 176, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5e439c0f-0e9d-43af-af17-ac2323b48677, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
22335 [8810] Exited with: ExitStatus(unix_wait_status(9))
22336 [8810][0] reports Exit
22337 Feb 25 20:51:37.497 INFO request completed, latency_us: 396, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7561b2b7-8950-4fd1-9fb8-11627a733452, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
22338 001/300 Read block 4968 len 4096
22339 002/300 Read block 551 len 28672
22340 003/300 Write block 1439 len 4096 data: 2
22341 004/300 Read block 3517 len 28672
22342 005/300 Write block 981 len 40960 data: 5 6 7 8 9 8 7 5 5 4
22343 006/300 Write block 3853 len 24576 data: 7 8 7 6 5 3
22344 007/300 Write block 2638 len 4096 data: 2
22345 008/300 Read block 2349 len 36864
22346 009/300 Read block 4061 len 40960
22347 010/300 Flush
22348 011/300 Read block 4070 len 28672
22349 012/300 Read block 4579 len 36864
22350 013/300 Read block 1502 len 4096
22351 014/300 Read block 4921 len 8192
22352 015/300 Write block 4768 len 32768 data: 4 4 5 6 4 5 4 4
22353 016/300 Write block 702 len 20480 data: 4 4 6 7 8
22354 017/300 Read block 891 len 28672
22355 018/300 Read block 2732 len 32768
22356 019/300 Write block 1855 len 4096 data: 4
22357 020/300 Read block 680 len 16384
22358 021/300 Read block 1781 len 12288
22359 022/300 Read block 2654 len 16384
22360 023/300 Read block 2108 len 20480
22361 024/300 Write block 4471 len 36864 data: 3 4 5 5 3 3 3 3 3
22362 025/300 Write block 3874 len 36864 data: 7 7 7 8 7 6 6 6 7
22363 026/300 Read block 440 len 28672
22364 027/300 Write block 165 len 24576 data: 5 4 2 2 3 3
22365 028/300 Read block 936 len 40960
22366 029/300 Read block 3628 len 8192
22367 030/300 Write block 3444 len 20480 data: 5 3 3 3 3
22368 031/300 Read block 1768 len 20480
22369 032/300 Write block 957 len 8192 data: 6 7
22370 033/300 Read block 2015 len 20480
22371 034/300 Write block 4412 len 32768 data: 4 4 4 5 5 5 7 6
22372 035/300 Read block 4169 len 4096
22373 036/300 Read block 2115 len 32768
22374 037/300 Read block 3682 len 24576
22375 038/300 Read block 3130 len 28672
22376 039/300 Read block 1938 len 12288
22377 040/300 Write block 133 len 40960 data: 6 4 4 3 3 3 3 3 3 3
22378 041/300 Flush
22379 042/300 Flush
22380 043/300 Read block 2084 len 8192
22381 044/300 Write block 3808 len 20480 data: 2 2 3 3 3
22382 045/300 Read block 4997 len 8192
22383 046/300 Read block 1677 len 36864
22384 047/300 Read block 2907 len 8192
22385 048/300 Read block 1653 len 28672
22386 049/300 Read block 2060 len 16384
22387 050/300 Write block 3715 len 32768 data: 6 5 4 4 5 6 6 8
22388 051/300 Write block 1522 len 20480 data: 6 6 7 6 5
22389 052/300 Read block 2862 len 36864
22390 053/300 Read block 1313 len 28672
22391 054/300 Read block 2405 len 16384
22392 055/300 Read block 3155 len 16384
22393 056/300 Write block 2468 len 36864 data: 2 3 5 6 5 5 5 5 5
22394 057/300 Read block 4259 len 8192
22395 058/300 Write block 4098 len 8192 data: 4 7
22396 059/300 Write block 2085 len 32768 data: 6 6 6 6 5 5 4 4
22397 060/300 Read block 4735 len 28672
22398 061/300 Read block 988 len 20480
22399 062/300 Read block 2227 len 28672
22400 063/300 Write block 3789 len 8192 data: 5 4
22401 064/300 Write block 1810 len 20480 data: 5 5 7 7 5
22402 065/300 Read block 4615 len 16384
22403 066/300 Read block 1874 len 4096
22404 067/300 Read block 132 len 4096
22405 068/300 Read block 287 len 8192
22406 069/300 Read block 4702 len 28672
22407 070/300 Write block 488 len 36864 data: 3 4 2 2 2 3 3 3 3
22408 071/300 Write block 2730 len 28672 data: 2 2 3 4 4 5 4
22409 072/300 Write block 3569 len 24576 data: 3 5 6 6 6 7
22410 073/300 Read block 2234 len 8192
22411 074/300 Write block 4846 len 28672 data: 5 6 6 5 4 5 4
22412 075/300 Flush
22413 076/300 Read block 1371 len 16384
22414 077/300 Read block 4859 len 16384
22415 078/300 Write block 2300 len 40960 data: 2 2 3 3 2 2 3 3 3 4
22416 079/300 Write block 4701 len 16384 data: 4 3 3 4
22417 080/300 Read block 1366 len 40960
22418 081/300 Read block 2544 len 20480
22419 082/300 Flush
22420 083/300 Read block 3247 len 36864
22421 084/300 Write block 1522 len 16384 data: 7 7 8 7
22422 085/300 Flush
22423 086/300 Read block 1345 len 4096
22424 087/300 Write block 456 len 8192 data: 2 2
22425 088/300 Write block 2379 len 32768 data: 7 4 3 4 5 4 4 4
22426 089/300 Write block 658 len 16384 data: 7 7 8 8
22427 090/300 Write block 1102 len 20480 data: 7 6 7 5 5
22428 091/300 Flush
22429 092/300 Read block 3747 len 32768
22430 093/300 Read block 3123 len 12288
22431 094/300 Write block 3942 len 4096 data: 3
22432 095/300 Read block 4173 len 4096
22433 096/300 Read block 3408 len 28672
22434 097/300 Write block 4253 len 12288 data: 2 3 2
22435 098/300 Read block 4655 len 8192
22436 099/300 Read block 3997 len 8192
22437 100/300 Write block 2748 len 8192 data: 6 6
22438 101/300 Read block 4401 len 32768
22439 102/300 Write block 2752 len 24576 data: 5 5 6 6 7 7
22440 103/300 Read block 699 len 32768
22441 104/300 Flush
22442 105/300 Write block 3 len 36864 data: 2 3 3 4 4 4 5 5 5
22443 106/300 Read block 3122 len 20480
22444 107/300 Read block 2605 len 12288
22445 108/300 Write block 3079 len 28672 data: 3 3 3 3 3 3 4
22446 109/300 Write block 2957 len 12288 data: 6 8 7
22447 110/300 Read block 3866 len 16384
22448 111/300 Write block 836 len 12288 data: 4 4 4
22449 112/300 Read block 1316 len 24576
22450 113/300 Write block 4758 len 36864 data: 4 4 4 4 4 4 3 3 3
22451 114/300 Write block 3443 len 24576 data: 5 6 4 4 4 4
22452 115/300 Read block 361 len 36864
22453 116/300 Write block 3714 len 4096 data: 6
22454 117/300 Write block 4252 len 32768 data: 2 3 4 3 2 2 3 4
22455 118/300 Read block 961 len 20480
22456 119/300 Write block 660 len 4096 data: 9
22457 120/300 Write block 2668 len 40960 data: 4 2 2 2 2 3 4 4 3 3
22458 121/300 Read block 894 len 16384
22459 122/300 Read block 735 len 32768
22460 123/300 Read block 764 len 28672
22461 124/300 Read block 4052 len 40960
22462 125/300 Read block 4916 len 40960
22463 126/300 Write block 4524 len 16384 data: 4 5 7 7
22464 127/300 Write block 1992 len 24576 data: 6 6 6 3 4 4
22465 128/300 Write block 3975 len 12288 data: 7 7 8
22466 129/300 Read block 820 len 16384
22467 130/300 Write block 2126 len 8192 data: 5 5
22468 131/300 Read block 3883 len 16384
22469 132/300 Read block 2852 len 20480
22470 133/300 Read block 3967 len 20480
22471 134/300 Write block 2991 len 28672 data: 7 8 7 5 5 5 5
22472 135/300 Read block 166 len 12288
22473 136/300 Write block 1466 len 28672 data: 3 3 3 3 3 4 4
22474 137/300 Read block 1678 len 24576
22475 138/300 Write block 1728 len 24576 data: 5 5 6 5 4 3
22476 139/300 Read block 4380 len 32768
22477 140/300 Read block 2678 len 28672
22478 141/300 Read block 4937 len 36864
22479 142/300 Read block 3305 len 40960
22480 143/300 Write block 3650 len 28672 data: 6 4 4 4 4 4 6
22481 144/300 Write block 1604 len 4096 data: 5
22482 145/300 Write block 3651 len 24576 data: 5 5 5 5 5 7
22483 146/300 Read block 3400 len 36864
22484 147/300 Write block 4352 len 12288 data: 3 3 3
22485 148/300 Read block 686 len 8192
22486 149/300 Write block 3499 len 4096 data: 3
22487 150/300 Write block 1093 len 28672 data: 6 7 9 9 7 5 5
22488 151/300 Read block 4668 len 32768
22489 152/300 Read block 2615 len 40960
22490 153/300 Write block 29 len 4096 data: 3
22491 154/300 Read block 4731 len 4096
22492 155/300 Flush
22493 156/300 Read block 1299 len 16384
22494 157/300 Read block 325 len 4096
22495 158/300 Write block 486 len 28672 data: 5 3 4 5 3 3 3
22496 159/300 Write block 195 len 40960 data: 5 6 5 6 5 5 5 4 4 4
22497 160/300 Read block 1391 len 20480
22498 161/300 Write block 975 len 12288 data: 5 4 4
22499 162/300 Flush
22500 163/300 Read block 4137 len 32768
22501 164/300 Read block 1750 len 20480
22502 165/300 Write block 208 len 4096 data: 3
22503 166/300 Read block 505 len 4096
22504 167/300 Read block 4874 len 36864
22505 168/300 Write block 4932 len 4096 data: 2
22506 169/300 Read block 4171 len 8192
22507 170/300 Read block 835 len 32768
22508 171/300 Write block 4136 len 40960 data: 4 4 4 5 5 5 6 4 4 5
22509 172/300 Read block 2572 len 4096
22510 173/300 Write block 3234 len 40960 data: 3 3 4 6 6 5 5 5 8 7
22511 174/300 Write block 3083 len 20480 data: 4 4 5 4 3
22512 175/300 Read block 1523 len 8192
22513 176/300 Write block 4045 len 36864 data: 2 3 3 3 3 3 3 3 2
22514 177/300 Write block 4452 len 4096 data: 3
22515 178/300 Write block 2754 len 20480 data: 7 7 8 8 7
22516 179/300 Write block 2787 len 12288 data: 2 2 2
22517 180/300 Write block 2513 len 12288 data: 7 7 6
22518 181/300 Write block 3060 len 24576 data: 3 2 4 5 5 4
22519 182/300 Read block 4181 len 16384
22520 183/300 Write block 2376 len 8192 data: 5 7
22521 184/300 Read block 3405 len 36864
22522 185/300 Read block 201 len 24576
22523 186/300 Read block 2925 len 24576
22524 187/300 Write block 2156 len 40960 data: 6 7 7 9 9 7 6 5 6 6
22525 188/300 Read block 504 len 8192
22526 189/300 Read block 3813 len 36864
22527 190/300 Read block 4509 len 20480
22528 191/300 Read block 4419 len 12288
22529 192/300 Read block 1505 len 12288
22530 193/300 Write block 421 len 4096 data: 6
22531 194/300 Read block 1866 len 40960
22532 195/300 Write block 4820 len 20480 data: 5 6 6 6 6
22533 196/300 Read block 992 len 16384
22534 197/300 Write block 4586 len 4096 data: 4
22535 198/300 Read block 3835 len 8192
22536 199/300 Write block 2680 len 20480 data: 7 8 7 7 7
22537 200/300 Write block 4082 len 12288 data: 7 6 6
22538 201/300 Read block 4844 len 24576
22539 202/300 Read block 4079 len 8192
22540 203/300 Read block 446 len 28672
22541 204/300 Write block 101 len 40960 data: 3 2 4 4 4 4 3 4 4 4
22542 205/300 Read block 819 len 12288
22543 206/300 Write block 1257 len 4096 data: 6
22544 207/300 Read block 2030 len 36864
22545 208/300 Write block 2481 len 16384 data: 3 3 4 4
22546 209/300 Read block 4159 len 32768
22547 210/300 Write block 506 len 28672 data: 5 5 3 3 3 4 6
22548 211/300 Flush
22549 212/300 Flush
22550 213/300 Flush
22551 214/300 Write block 2709 len 8192 data: 6 6
22552 215/300 Write block 3696 len 28672 data: 3 3 4 4 4 6 6
22553 216/300 Flush
22554 217/300 Read block 4856 len 28672
22555 218/300 Write block 3209 len 16384 data: 3 3 3 3
22556 219/300 Read block 1233 len 32768
22557 220/300 Write block 4008 len 32768 data: 2 2 2 2 2 3 3 4
22558 221/300 Read block 516 len 4096
22559 222/300 Write block 3162 len 20480 data: 4 4 5 6 6
22560 223/300 Read block 2111 len 36864
22561 224/300 Write block 4125 len 4096 data: 6
22562 225/300 Write block 4243 len 16384 data: 4 3 4 5
22563 226/300 Read block 3747 len 28672
22564 227/300 Write block 2464 len 24576 data: 5 3 2 2 3 4
22565 228/300 Read block 4884 len 40960
22566 229/300 Read block 1560 len 24576
22567 230/300 Read block 2009 len 28672
22568 231/300 Read block 1896 len 40960
22569 232/300 Read block 496 len 4096
22570 233/300 Write block 1636 len 8192 data: 5 4
22571 234/300 Read block 3236 len 20480
22572 235/300 Read block 4864 len 20480
22573 236/300 Write block 1282 len 4096 data: 2
22574 237/300 Read block 1183 len 32768
22575 238/300 Flush
22576 239/300 Read block 4216 len 28672
22577 240/300 Read block 841 len 20480
22578 241/300 Write block 1233 len 40960 data: 4 3 5 5 5 5 3 2 2 2
22579 242/300 Read block 790 len 20480
22580 243/300 Read block 1761 len 12288
22581 244/300 Write block 4383 len 28672 data: 5 5 3 4 5 4 4
22582 245/300 Write block 2251 len 36864 data: 5 6 5 6 8 9 7 7 8
22583 246/300 Write block 2390 len 4096 data: 2
22584 247/300 Read block 2512 len 4096
22585 248/300 Write block 2627 len 16384 data: 7 6 4 4
22586 249/300 Write block 3453 len 28672 data: 3 3 3 3 3 2 2
22587 250/300 Read block 4795 len 32768
22588 251/300 Write block 484 len 12288 data: 5 6 6
22589 252/300 Read block 4674 len 8192
22590 253/300 Read block 1397 len 28672
22591 254/300 Write block 3748 len 36864 data: 4 4 4 4 3 3 3 5 5
22592 255/300 Read block 4252 len 28672
22593 256/300 Write block 3429 len 36864 data: 4 3 4 5 6 8 7 7 6
22594 257/300 Write block 3118 len 32768 data: 5 5 5 6 5 5 6 6
22595 258/300 Read block 3924 len 36864
22596 259/300 Write block 728 len 40960 data: 9 9 7 6 3 3 3 2 2 2
22597 260/300 Read block 4933 len 8192
22598 261/300 Write block 4652 len 20480 data: 4 4 5 5 5
22599 262/300 Read block 4619 len 32768
22600 263/300 Read block 1086 len 20480
22601 264/300 Read block 3844 len 40960
22602 265/300 Write block 4227 len 8192 data: 6 6
22603 266/300 Read block 3562 len 36864
22604 267/300 Read block 2149 len 28672
22605 268/300 Read block 2044 len 4096
22606 269/300 Read block 1692 len 12288
22607 270/300 Read block 2471 len 4096
22608 271/300 Flush
22609 272/300 Write block 2329 len 4096 data: 4
22610 273/300 Write block 4190 len 24576 data: 2 3 3 3 3 3
22611 274/300 Write block 1255 len 32768 data: 4 5 7 7 6 8 8 8
22612 275/300 Read block 4905 len 32768
22613 276/300 Read block 3424 len 32768
22614 277/300 Write block 3186 len 36864 data: 5 3 2 4 4 4 5 5 5
22615 278/300 Write block 4963 len 8192 data: 4 5
22616 279/300 Write block 4747 len 40960 data: 4 4 5 5 5 5 6 5 6 6
22617 280/300 Read block 629 len 12288
22618 281/300 Read block 2256 len 20480
22619 282/300 Read block 1450 len 12288
22620 283/300 Write block 918 len 32768 data: 5 6 6 6 5 5 5 4
22621 284/300 Write block 2262 len 16384 data: 3 3 2 2
22622 285/300 Read block 3772 len 16384
22623 286/300 Read block 4370 len 40960
22624 287/300 Write block 371 len 24576 data: 5 4 4 5 5 3
22625 288/300 Read block 824 len 24576
22626 289/300 Read block 2025 len 28672
22627 290/300 Read block 2214 len 8192
22628 291/300 Read block 4417 len 36864
22629 292/300 Read block 715 len 40960
22630 293/300 Write block 4932 len 36864 data: 3 2 2 3 5 5 5 6 7
22631 294/300 Read block 2685 len 36864
22632 295/300 Flush
22633 296/300 Write block 4511 len 36864 data: 4 4 4 5 5 4 4 5 5
22634 297/300 Flush
22635 298/300 Read block 4457 len 16384
22636 299/300 Flush
22637 300/300 Write block 4865 len 32768 data: 3 4 3 3 3 4 5 6
22638 301/300 Write block 509 len 16384 data: 4 4 5 7
22639 Main task has work to do, go find it
22640 got dsc Start(0)
22641 start 0
22642 Feb 25 20:51:37.752 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3157d2ad-2655-4ebd-a1fa-0873e24ff0fe, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
22643 Replay: started 0, returned:Ok(())
22644 ----------------------------------------------------------------
22645 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22646 States: Offline Active Active
22647 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22648 6736 Acked Read 1 Sent Done Done false
22649 6737 Acked Read 7 Sent Done Done false
22650 6738 Acked Write 1 Sent Done Done false
22651 6739 Acked Read 7 Sent Done Done false
22652 6740 Acked Write 10 Sent Done Done false
22653 6741 Acked Write 6 Sent Done Done false
22654 6742 Acked Write 1 Sent Done Done false
22655 6743 Acked Read 9 Sent Done Done false
22656 6744 Acked Read 10 Sent Done Done false
22657 6745 Acked Flush 0 Sent Done Done false
22658 6746 Acked Read 7 Sent Done Done false
22659 6747 Acked Read 9 Sent Done Done false
22660 6748 Acked Read 1 Sent Done Done false
22661 6749 Acked Read 2 Sent Done Done false
22662 6750 Acked Write 8 Sent Done Done false
22663 6751 Acked Write 5 Sent Done Done false
22664 6752 Acked Read 7 Sent Done Done false
22665 6753 Acked Read 8 Sent Done Done false
22666 6754 Acked Write 1 Sent Done Done false
22667 6755 Acked Read 4 Sent Done Done false
22668 6756 Acked Read 3 Sent Done Done false
22669 6757 Acked Read 4 Sent Done Done false
22670 6758 Acked Read 5 Sent Done Done false
22671 6759 Acked Write 9 Sent Done Done false
22672 6760 Acked Write 9 Sent Done Done false
22673 6761 Acked Read 7 Sent Done Done false
22674 6762 Acked Write 6 Sent Done Done false
22675 6763 Acked Read 10 Sent Done Done false
22676 6764 Acked Read 2 Sent Done Done false
22677 6765 Acked Write 5 Sent Done Done false
22678 6766 Acked Read 5 Sent Done Done false
22679 6767 Acked Write 2 Sent Done Done false
22680 6768 Acked Read 5 Sent Done Done false
22681 6769 Acked Write 8 Sent Done Done false
22682 6770 Acked Read 1 Sent Done Done false
22683 6771 Acked Read 8 Sent Done Done false
22684 6772 Acked Read 6 Sent Done Done false
22685 6773 Acked Read 7 Sent Done Done false
22686 6774 Acked Read 3 Sent Done Done false
22687 6775 Acked Write 10 Sent Done Done false
22688 6776 Acked Flush 0 Sent Done Done false
22689 6777 Acked Flush 0 Sent Done Done false
22690 6778 Acked Read 2 Sent Done Done false
22691 6779 Acked Write 5 Sent Done Done false
22692 6780 Acked Read 2 Sent Done Done false
22693 6781 Acked Read 9 Sent Done Done false
22694 6782 Acked Read 2 Sent Done Done false
22695 6783 Acked Read 7 Sent Done Done false
22696 6784 Acked Read 4 Sent Done Done false
22697 6785 Acked Write 8 Sent Done Done false
22698 6786 Acked Write 5 Sent Done Done false
22699 6787 Acked Read 9 Sent Done Done false
22700 6788 Acked Read 7 Sent Done Done false
22701 6789 Acked Read 4 Sent Done Done false
22702 6790 Acked Read 4 Sent Done Done false
22703 6791 Acked Write 9 Sent Done Done false
22704 6792 Acked Read 2 Sent Done Done false
22705 6793 Acked Write 2 Sent Done Done false
22706 6794 Acked Write 8 Sent Done Done false
22707 6795 Acked Read 7 Sent Done Done false
22708 6796 Acked Read 5 Sent Done Done false
22709 6797 Acked Read 7 Sent Done Done false
22710 6798 Acked Write 2 Sent Done Done false
22711 6799 Acked Write 5 Sent Done Done false
22712 6800 Acked Read 4 Sent Done Done false
22713 6801 Acked Read 1 Sent Done Done false
22714 6802 Acked Read 1 Sent Done Done false
22715 6803 Acked Read 2 Sent Done Done false
22716 6804 Acked Read 7 Sent Done Done false
22717 6805 Acked Write 9 Sent Done Done false
22718 6806 Acked Write 7 Sent Done Done false
22719 6807 Acked Write 6 Sent Done Done false
22720 6808 Acked Read 2 Sent Done Done false
22721 6809 Acked Write 7 Sent Done Done false
22722 6810 Acked Flush 0 Sent Done Done false
22723 6811 Acked Read 4 Sent Done Done false
22724 6812 Acked Read 4 Sent Done Done false
22725 6813 Acked Write 10 Sent Done Done false
22726 6814 Acked Write 4 Sent Done Done false
22727 6815 Acked Read 10 Sent Done Done false
22728 6816 Acked Read 5 Sent Done Done false
22729 6817 Acked Flush 0 Sent Done Done false
22730 6818 Acked Read 9 Sent Done Done false
22731 6819 Acked Write 4 Sent Done Done false
22732 6820 Acked Flush 0 Sent Done Done false
22733 6821 Acked Read 1 Sent Done Done false
22734 6822 Acked Write 2 Sent Done Done false
22735 6823 Acked Write 8 Sent Done Done false
22736 6824 Acked Write 4 Sent Done Done false
22737 6825 Acked Write 5 Sent Done Done false
22738 6826 Acked Flush 0 Sent Done Done false
22739 6827 Acked Read 8 Sent Done Done false
22740 6828 Acked Read 3 Sent Done Done false
22741 6829 Acked Write 1 Sent Done Done false
22742 6830 Acked Read 1 Sent Done Done false
22743 6831 Acked Read 7 Sent Done Done false
22744 6832 Acked Write 3 Sent Done Done false
22745 6833 Acked Read 2 Sent Done Done false
22746 6834 Acked Read 2 Sent Done Done false
22747 6835 Acked Write 2 Sent Done Done false
22748 6836 Acked Read 8 Sent Done Done false
22749 6837 Acked Write 6 Sent Done Done false
22750 6838 Acked Read 8 Sent Done Done false
22751 6839 Acked Flush 0 Sent Done Done false
22752 6840 Acked Write 9 Sent Done Done false
22753 6841 Acked Read 5 Sent Done Done false
22754 6842 Acked Read 3 Sent Done Done false
22755 6843 Acked Write 7 Sent Done Done false
22756 6844 Acked Write 3 Sent Done Done false
22757 6845 Acked Read 4 Sent Done Done false
22758 6846 Acked Write 3 Sent Done Done false
22759 6847 Acked Read 6 Sent Done Done false
22760 6848 Acked Write 9 Sent Done Done false
22761 6849 Acked Write 6 Sent Done Done false
22762 6850 Acked Read 9 Sent Done Done false
22763 6851 Acked Write 1 Sent Done Done false
22764 6852 Acked Write 8 Sent Done Done false
22765 6853 Acked Read 5 Sent Done Done false
22766 6854 Acked Write 1 Sent Done Done false
22767 6855 Acked Write 10 Sent Done Done false
22768 6856 Acked Read 4 Sent Done Done false
22769 6857 Acked Read 8 Sent Done Done false
22770 6858 Acked Read 7 Sent Done Done false
22771 6859 Acked Read 10 Sent Done Done false
22772 6860 Acked Read 10 Sent Done Done false
22773 6861 Acked Write 4 Sent Done Done false
22774 6862 Acked Write 6 Sent Done Done false
22775 6863 Acked Write 3 Sent Done Done false
22776 6864 Acked Read 4 Sent Done Done false
22777 6865 Acked Write 2 Sent Done Done false
22778 6866 Acked Read 4 Sent Done Done false
22779 6867 Acked Read 5 Sent Done Done false
22780 6868 Acked Read 5 Sent Done Done false
22781 6869 Acked Write 7 Sent Done Done false
22782 6870 Acked Read 3 Sent Done Done false
22783 6871 Acked Write 7 Sent Done Done false
22784 6872 Acked Read 6 Sent Done Done false
22785 6873 Acked Write 6 Sent Done Done false
22786 6874 Acked Read 8 Sent Done Done false
22787 6875 Acked Read 7 Sent Done Done false
22788 6876 Acked Read 9 Sent Done Done false
22789 6877 Acked Read 10 Sent Done Done false
22790 6878 Acked Write 7 Sent Done Done false
22791 6879 Acked Write 1 Sent Done Done false
22792 6880 Acked Write 6 Sent Done Done false
22793 6881 Acked Read 9 Sent Done Done false
22794 6882 Acked Write 3 Sent Done Done false
22795 6883 Acked Read 2 Sent Done Done false
22796 6884 Acked Write 1 Sent Done Done false
22797 6885 Acked Write 7 Sent Done Done false
22798 6886 Acked Read 8 Sent Done Done false
22799 6887 Acked Read 10 Sent Done Done false
22800 6888 Acked Write 1 Sent Done Done false
22801 6889 Acked Read 1 Sent Done Done false
22802 6890 Acked Flush 0 Sent Done Done false
22803 6891 Acked Read 4 Sent Done Done false
22804 6892 Acked Read 1 Sent Done Done false
22805 6893 Acked Write 7 Sent Done Done false
22806 6894 Acked Write 10 Sent Done Done false
22807 6895 Acked Read 5 Sent Done Done false
22808 6896 Acked Write 3 Sent Done Done false
22809 6897 Acked Flush 0 Sent Done Done false
22810 6898 Acked Read 8 Sent Done Done false
22811 6899 Acked Read 5 Sent Done Done false
22812 6900 Acked Write 1 Sent Done Done false
22813 6901 Acked Read 1 Sent Done Done false
22814 6902 Acked Read 9 Sent Done Done false
22815 6903 Acked Write 1 Sent Done Done false
22816 6904 Acked Read 2 Sent Done Done false
22817 6905 Acked Read 8 Sent Done Done false
22818 6906 Acked Write 10 Sent Done Done false
22819 6907 Acked Read 1 Sent Done Done false
22820 6908 Acked Write 10 Sent Done Done false
22821 6909 Acked Write 5 Sent Done Done false
22822 6910 Acked Read 2 Sent Done Done false
22823 6911 Acked Write 9 Sent Done Done false
22824 6912 Acked Write 1 Sent Done Done false
22825 6913 Acked Write 5 Sent Done Done false
22826 6914 Acked Write 3 Sent Done Done false
22827 6915 Acked Write 3 Sent Done Done false
22828 6916 Acked Write 6 Sent Done Done false
22829 6917 Acked Read 4 Sent Done Done false
22830 6918 Acked Write 2 Sent Done Done false
22831 6919 Acked Read 9 Sent Done Done false
22832 6920 Acked Read 6 Sent Done Done false
22833 6921 Acked Read 6 Sent Done Done false
22834 6922 Acked Write 10 Sent Done Done false
22835 6923 Acked Read 2 Sent Done Done false
22836 6924 Acked Read 9 Sent Done Done false
22837 6925 Acked Read 5 Sent Done Done false
22838 6926 Acked Read 3 Sent Done Done false
22839 6927 Acked Read 3 Sent Done Done false
22840 6928 Acked Write 1 Sent Done Done false
22841 6929 Acked Read 10 Sent Done Done false
22842 6930 Acked Write 5 Sent Done Done false
22843 6931 Acked Read 4 Sent Done Done false
22844 6932 Acked Write 1 Sent Done Done false
22845 6933 Acked Read 2 Sent Done Done false
22846 6934 Acked Write 5 Sent Done Done false
22847 6935 Acked Write 3 Sent Done Done false
22848 6936 Acked Read 6 Sent Done Done false
22849 6937 Acked Read 2 Sent Done Done false
22850 6938 Acked Read 7 Sent Done Done false
22851 6939 Acked Write 10 Sent Done Done false
22852 6940 Acked Read 3 Sent Done Done false
22853 6941 Acked Write 1 Sent Done Done false
22854 6942 Acked Read 9 Sent Done Done false
22855 6943 Acked Write 4 Sent Done Done false
22856 6944 Acked Read 8 Sent Done Done false
22857 6945 Acked Write 7 Sent Done Done false
22858 6946 Acked Flush 0 Sent Done Done false
22859 6947 Acked Flush 0 Sent Done Done false
22860 6948 Acked Flush 0 Sent Done Done false
22861 6949 Acked Write 2 Sent Done Done false
22862 6950 Acked Write 7 Sent Done Done false
22863 6951 Acked Flush 0 Sent Done Done false
22864 6952 Acked Read 7 Sent Done Done false
22865 6953 Acked Write 4 Sent Done Done false
22866 6954 Acked Read 8 Sent Done Done false
22867 6955 Acked Write 8 Sent Done Done false
22868 6956 Acked Read 1 Sent Done Done false
22869 6957 Acked Write 5 Sent Done Done false
22870 6958 Acked Read 9 Sent Done Done false
22871 6959 Acked Write 1 Sent Done Done false
22872 6960 Acked Write 4 Sent Done Done false
22873 6961 Acked Read 7 Sent Done Done false
22874 6962 Acked Write 6 Sent Done Done false
22875 6963 Acked Read 10 Sent Done Done false
22876 6964 Acked Read 6 Sent Done Done false
22877 6965 Acked Read 7 Sent Done Done false
22878 6966 Acked Read 10 Sent Done Done false
22879 6967 Acked Read 1 Sent Done Done false
22880 6968 Acked Write 2 Sent Done Done false
22881 6969 Acked Read 5 Sent Done Done false
22882 6970 Acked Read 5 Sent Done Done false
22883 6971 Acked Write 1 Sent Done Done false
22884 6972 Acked Read 8 Sent Done Done false
22885 6973 Acked Flush 0 Sent Done Done false
22886 6974 Acked Read 7 Sent Done Done false
22887 6975 Acked Read 5 Sent Done Done false
22888 6976 Acked Write 10 Sent Done Done false
22889 6977 Acked Read 5 Sent Done Done false
22890 6978 Acked Read 3 Sent Done Done false
22891 6979 Acked Write 7 Sent Done Done false
22892 6980 Acked Write 9 Sent Done Done false
22893 6981 Acked Write 1 Sent Done Done false
22894 6982 Acked Read 1 Sent Done Done false
22895 6983 Acked Write 4 Sent Done Done false
22896 6984 Acked Write 7 Sent Done Done false
22897 6985 Acked Read 8 Sent Done Done false
22898 6986 Acked Write 3 Sent Done Done false
22899 6987 Acked Read 2 Sent Done Done false
22900 6988 Acked Read 7 Sent Done Done false
22901 6989 Acked Write 9 Sent Done Done false
22902 6990 Acked Read 7 Sent Done Done false
22903 6991 Acked Write 9 Sent Done Done false
22904 6992 Acked Write 8 Sent Done Done false
22905 6993 Acked Read 9 Sent Done Done false
22906 6994 Acked Write 10 Sent Done Done false
22907 6995 Acked Read 2 Sent Done Done false
22908 6996 Acked Write 5 Sent Done Done false
22909 6997 Acked Read 8 Sent Done Done false
22910 6998 Acked Read 5 Sent Done Done false
22911 6999 Acked Read 10 Sent Done Done false
22912 7000 Acked Write 2 Sent Done Done false
22913 7001 Acked Read 9 Sent Done Done false
22914 7002 Acked Read 7 Sent Done Done false
22915 7003 Acked Read 1 Sent Done Done false
22916 7004 Acked Read 3 Sent Done Done false
22917 7005 Acked Read 1 Sent Done Done false
22918 7006 Acked Flush 0 Sent Done Done false
22919 7007 Acked Write 1 Sent Done Done false
22920 7008 Acked Write 6 Sent Done Done false
22921 7009 Acked Write 8 Sent Done Done false
22922 7010 Acked Read 8 Sent Done Done false
22923 7011 Acked Read 8 Sent Done Done false
22924 7012 Acked Write 9 Sent Done Done false
22925 7013 Acked Write 2 Sent Done Done false
22926 7014 Acked Write 10 Sent Done Done false
22927 7015 Acked Read 3 Sent Done Done false
22928 7016 Acked Read 5 Sent Done Done false
22929 7017 Acked Read 3 Sent Done Done false
22930 7018 Acked Write 8 Sent Done Done false
22931 7019 Acked Write 4 Sent Done Done false
22932 7020 Acked Read 4 Sent Done Done false
22933 7021 Acked Read 10 Sent Done Done false
22934 7022 Acked Write 6 Sent Done Done false
22935 7023 Acked Read 6 Sent Done Done false
22936 7024 Acked Read 7 Sent Done Done false
22937 7025 Acked Read 2 Sent Done Done false
22938 7026 Acked Read 9 Sent Done Done false
22939 7027 Acked Read 10 Sent Done Done false
22940 7028 Acked Write 9 Sent Done Done false
22941 7029 Acked Read 9 Sent Done Done false
22942 7030 Acked Flush 0 Sent Done Done false
22943 7031 Acked Write 9 Sent Done Done false
22944 7032 Acked Flush 0 Sent Done Done false
22945 7033 Acked Read 4 Sent Done Done false
22946 7034 Acked Flush 0 Sent Done Done false
22947 7035 Acked Write 8 Sent Done Done false
22948 7036 Acked Write 4 Sent Done Done false
22949 STATES DS:0 DS:1 DS:2 TOTAL
22950 Sent 301 0 0 301
22951 Done 0 301 301 602
22952 Skipped 0 0 0 0
22953 Error 0 0 0 0
22954 Last Flush: Some(JobId(6735)) Some(JobId(7034)) Some(JobId(7034))
22955 Downstairs last five completed: 6735 6734 6733 6732 6731
22956 Upstairs last five completed: 7036 7035 7034 7033 7032
22957 CLIENT: Up:0 ds:301 act:2
229582025-02-25T20:51:38.535ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
22959 [8810] got start action
22960 [8810] I am going to restart
22961 Starting downstairs at port 8810
22962 Make output file at "/tmp/dsc/downstairs-8810.txt"
22963 [8810][0] reports Starting
22964 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1226)
22965 [8810][0] reports Running
229662025-02-25T20:51:39.535ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
229672025-02-25T20:51:40.536ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
229682025-02-25T20:51:41.538ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
22969 ----------------------------------------------------------------
22970 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22971 States: Offline Active Active
22972 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22973 6736 Acked Read 1 Sent Done Done false
22974 6737 Acked Read 7 Sent Done Done false
22975 6738 Acked Write 1 Sent Done Done false
22976 6739 Acked Read 7 Sent Done Done false
22977 6740 Acked Write 10 Sent Done Done false
22978 6741 Acked Write 6 Sent Done Done false
22979 6742 Acked Write 1 Sent Done Done false
22980 6743 Acked Read 9 Sent Done Done false
22981 6744 Acked Read 10 Sent Done Done false
22982 6745 Acked Flush 0 Sent Done Done false
22983 6746 Acked Read 7 Sent Done Done false
22984 6747 Acked Read 9 Sent Done Done false
22985 6748 Acked Read 1 Sent Done Done false
22986 6749 Acked Read 2 Sent Done Done false
22987 6750 Acked Write 8 Sent Done Done false
22988 6751 Acked Write 5 Sent Done Done false
22989 6752 Acked Read 7 Sent Done Done false
22990 6753 Acked Read 8 Sent Done Done false
22991 6754 Acked Write 1 Sent Done Done false
22992 6755 Acked Read 4 Sent Done Done false
22993 6756 Acked Read 3 Sent Done Done false
22994 6757 Acked Read 4 Sent Done Done false
22995 6758 Acked Read 5 Sent Done Done false
22996 6759 Acked Write 9 Sent Done Done false
22997 6760 Acked Write 9 Sent Done Done false
22998 6761 Acked Read 7 Sent Done Done false
22999 6762 Acked Write 6 Sent Done Done false
23000 6763 Acked Read 10 Sent Done Done false
23001 6764 Acked Read 2 Sent Done Done false
23002 6765 Acked Write 5 Sent Done Done false
23003 6766 Acked Read 5 Sent Done Done false
23004 6767 Acked Write 2 Sent Done Done false
23005 6768 Acked Read 5 Sent Done Done false
23006 6769 Acked Write 8 Sent Done Done false
23007 6770 Acked Read 1 Sent Done Done false
23008 6771 Acked Read 8 Sent Done Done false
23009 6772 Acked Read 6 Sent Done Done false
23010 6773 Acked Read 7 Sent Done Done false
23011 6774 Acked Read 3 Sent Done Done false
23012 6775 Acked Write 10 Sent Done Done false
23013 6776 Acked Flush 0 Sent Done Done false
23014 6777 Acked Flush 0 Sent Done Done false
23015 6778 Acked Read 2 Sent Done Done false
23016 6779 Acked Write 5 Sent Done Done false
23017 6780 Acked Read 2 Sent Done Done false
23018 6781 Acked Read 9 Sent Done Done false
23019 6782 Acked Read 2 Sent Done Done false
23020 6783 Acked Read 7 Sent Done Done false
23021 6784 Acked Read 4 Sent Done Done false
23022 6785 Acked Write 8 Sent Done Done false
23023 6786 Acked Write 5 Sent Done Done false
23024 6787 Acked Read 9 Sent Done Done false
23025 6788 Acked Read 7 Sent Done Done false
23026 6789 Acked Read 4 Sent Done Done false
23027 6790 Acked Read 4 Sent Done Done false
23028 6791 Acked Write 9 Sent Done Done false
23029 6792 Acked Read 2 Sent Done Done false
23030 6793 Acked Write 2 Sent Done Done false
23031 6794 Acked Write 8 Sent Done Done false
23032 6795 Acked Read 7 Sent Done Done false
23033 6796 Acked Read 5 Sent Done Done false
23034 6797 Acked Read 7 Sent Done Done false
23035 6798 Acked Write 2 Sent Done Done false
23036 6799 Acked Write 5 Sent Done Done false
23037 6800 Acked Read 4 Sent Done Done false
23038 6801 Acked Read 1 Sent Done Done false
23039 6802 Acked Read 1 Sent Done Done false
23040 6803 Acked Read 2 Sent Done Done false
23041 6804 Acked Read 7 Sent Done Done false
23042 6805 Acked Write 9 Sent Done Done false
23043 6806 Acked Write 7 Sent Done Done false
23044 6807 Acked Write 6 Sent Done Done false
23045 6808 Acked Read 2 Sent Done Done false
23046 6809 Acked Write 7 Sent Done Done false
23047 6810 Acked Flush 0 Sent Done Done false
23048 6811 Acked Read 4 Sent Done Done false
23049 6812 Acked Read 4 Sent Done Done false
23050 6813 Acked Write 10 Sent Done Done false
23051 6814 Acked Write 4 Sent Done Done false
23052 6815 Acked Read 10 Sent Done Done false
23053 6816 Acked Read 5 Sent Done Done false
23054 6817 Acked Flush 0 Sent Done Done false
23055 6818 Acked Read 9 Sent Done Done false
23056 6819 Acked Write 4 Sent Done Done false
23057 6820 Acked Flush 0 Sent Done Done false
23058 6821 Acked Read 1 Sent Done Done false
23059 6822 Acked Write 2 Sent Done Done false
23060 6823 Acked Write 8 Sent Done Done false
23061 6824 Acked Write 4 Sent Done Done false
23062 6825 Acked Write 5 Sent Done Done false
23063 6826 Acked Flush 0 Sent Done Done false
23064 6827 Acked Read 8 Sent Done Done false
23065 6828 Acked Read 3 Sent Done Done false
23066 6829 Acked Write 1 Sent Done Done false
23067 6830 Acked Read 1 Sent Done Done false
23068 6831 Acked Read 7 Sent Done Done false
23069 6832 Acked Write 3 Sent Done Done false
23070 6833 Acked Read 2 Sent Done Done false
23071 6834 Acked Read 2 Sent Done Done false
23072 6835 Acked Write 2 Sent Done Done false
23073 6836 Acked Read 8 Sent Done Done false
23074 6837 Acked Write 6 Sent Done Done false
23075 6838 Acked Read 8 Sent Done Done false
23076 6839 Acked Flush 0 Sent Done Done false
23077 6840 Acked Write 9 Sent Done Done false
23078 6841 Acked Read 5 Sent Done Done false
23079 6842 Acked Read 3 Sent Done Done false
23080 6843 Acked Write 7 Sent Done Done false
23081 6844 Acked Write 3 Sent Done Done false
23082 6845 Acked Read 4 Sent Done Done false
23083 6846 Acked Write 3 Sent Done Done false
23084 6847 Acked Read 6 Sent Done Done false
23085 6848 Acked Write 9 Sent Done Done false
23086 6849 Acked Write 6 Sent Done Done false
23087 6850 Acked Read 9 Sent Done Done false
23088 6851 Acked Write 1 Sent Done Done false
23089 6852 Acked Write 8 Sent Done Done false
23090 6853 Acked Read 5 Sent Done Done false
23091 6854 Acked Write 1 Sent Done Done false
23092 6855 Acked Write 10 Sent Done Done false
23093 6856 Acked Read 4 Sent Done Done false
23094 6857 Acked Read 8 Sent Done Done false
23095 6858 Acked Read 7 Sent Done Done false
23096 6859 Acked Read 10 Sent Done Done false
23097 6860 Acked Read 10 Sent Done Done false
23098 6861 Acked Write 4 Sent Done Done false
23099 6862 Acked Write 6 Sent Done Done false
23100 6863 Acked Write 3 Sent Done Done false
23101 6864 Acked Read 4 Sent Done Done false
23102 6865 Acked Write 2 Sent Done Done false
23103 6866 Acked Read 4 Sent Done Done false
23104 6867 Acked Read 5 Sent Done Done false
23105 6868 Acked Read 5 Sent Done Done false
23106 6869 Acked Write 7 Sent Done Done false
23107 6870 Acked Read 3 Sent Done Done false
23108 6871 Acked Write 7 Sent Done Done false
23109 6872 Acked Read 6 Sent Done Done false
23110 6873 Acked Write 6 Sent Done Done false
23111 6874 Acked Read 8 Sent Done Done false
23112 6875 Acked Read 7 Sent Done Done false
23113 6876 Acked Read 9 Sent Done Done false
23114 6877 Acked Read 10 Sent Done Done false
23115 6878 Acked Write 7 Sent Done Done false
23116 6879 Acked Write 1 Sent Done Done false
23117 6880 Acked Write 6 Sent Done Done false
23118 6881 Acked Read 9 Sent Done Done false
23119 6882 Acked Write 3 Sent Done Done false
23120 6883 Acked Read 2 Sent Done Done false
23121 6884 Acked Write 1 Sent Done Done false
23122 6885 Acked Write 7 Sent Done Done false
23123 6886 Acked Read 8 Sent Done Done false
23124 6887 Acked Read 10 Sent Done Done false
23125 6888 Acked Write 1 Sent Done Done false
23126 6889 Acked Read 1 Sent Done Done false
23127 6890 Acked Flush 0 Sent Done Done false
23128 6891 Acked Read 4 Sent Done Done false
23129 6892 Acked Read 1 Sent Done Done false
23130 6893 Acked Write 7 Sent Done Done false
23131 6894 Acked Write 10 Sent Done Done false
23132 6895 Acked Read 5 Sent Done Done false
23133 6896 Acked Write 3 Sent Done Done false
23134 6897 Acked Flush 0 Sent Done Done false
23135 6898 Acked Read 8 Sent Done Done false
23136 6899 Acked Read 5 Sent Done Done false
23137 6900 Acked Write 1 Sent Done Done false
23138 6901 Acked Read 1 Sent Done Done false
23139 6902 Acked Read 9 Sent Done Done false
23140 6903 Acked Write 1 Sent Done Done false
23141 6904 Acked Read 2 Sent Done Done false
23142 6905 Acked Read 8 Sent Done Done false
23143 6906 Acked Write 10 Sent Done Done false
23144 6907 Acked Read 1 Sent Done Done false
23145 6908 Acked Write 10 Sent Done Done false
23146 6909 Acked Write 5 Sent Done Done false
23147 6910 Acked Read 2 Sent Done Done false
23148 6911 Acked Write 9 Sent Done Done false
23149 6912 Acked Write 1 Sent Done Done false
23150 6913 Acked Write 5 Sent Done Done false
23151 6914 Acked Write 3 Sent Done Done false
23152 6915 Acked Write 3 Sent Done Done false
23153 6916 Acked Write 6 Sent Done Done false
23154 6917 Acked Read 4 Sent Done Done false
23155 6918 Acked Write 2 Sent Done Done false
23156 6919 Acked Read 9 Sent Done Done false
23157 6920 Acked Read 6 Sent Done Done false
23158 6921 Acked Read 6 Sent Done Done false
23159 6922 Acked Write 10 Sent Done Done false
23160 6923 Acked Read 2 Sent Done Done false
23161 6924 Acked Read 9 Sent Done Done false
23162 6925 Acked Read 5 Sent Done Done false
23163 6926 Acked Read 3 Sent Done Done false
23164 6927 Acked Read 3 Sent Done Done false
23165 6928 Acked Write 1 Sent Done Done false
23166 6929 Acked Read 10 Sent Done Done false
23167 6930 Acked Write 5 Sent Done Done false
23168 6931 Acked Read 4 Sent Done Done false
23169 6932 Acked Write 1 Sent Done Done false
23170 6933 Acked Read 2 Sent Done Done false
23171 6934 Acked Write 5 Sent Done Done false
23172 6935 Acked Write 3 Sent Done Done false
23173 6936 Acked Read 6 Sent Done Done false
23174 6937 Acked Read 2 Sent Done Done false
23175 6938 Acked Read 7 Sent Done Done false
23176 6939 Acked Write 10 Sent Done Done false
23177 6940 Acked Read 3 Sent Done Done false
23178 6941 Acked Write 1 Sent Done Done false
23179 6942 Acked Read 9 Sent Done Done false
23180 6943 Acked Write 4 Sent Done Done false
23181 6944 Acked Read 8 Sent Done Done false
23182 6945 Acked Write 7 Sent Done Done false
23183 6946 Acked Flush 0 Sent Done Done false
23184 6947 Acked Flush 0 Sent Done Done false
23185 6948 Acked Flush 0 Sent Done Done false
23186 6949 Acked Write 2 Sent Done Done false
23187 6950 Acked Write 7 Sent Done Done false
23188 6951 Acked Flush 0 Sent Done Done false
23189 6952 Acked Read 7 Sent Done Done false
23190 6953 Acked Write 4 Sent Done Done false
23191 6954 Acked Read 8 Sent Done Done false
23192 6955 Acked Write 8 Sent Done Done false
23193 6956 Acked Read 1 Sent Done Done false
23194 6957 Acked Write 5 Sent Done Done false
23195 6958 Acked Read 9 Sent Done Done false
23196 6959 Acked Write 1 Sent Done Done false
23197 6960 Acked Write 4 Sent Done Done false
23198 6961 Acked Read 7 Sent Done Done false
23199 6962 Acked Write 6 Sent Done Done false
23200 6963 Acked Read 10 Sent Done Done false
23201 6964 Acked Read 6 Sent Done Done false
23202 6965 Acked Read 7 Sent Done Done false
23203 6966 Acked Read 10 Sent Done Done false
23204 6967 Acked Read 1 Sent Done Done false
23205 6968 Acked Write 2 Sent Done Done false
23206 6969 Acked Read 5 Sent Done Done false
23207 6970 Acked Read 5 Sent Done Done false
23208 6971 Acked Write 1 Sent Done Done false
23209 6972 Acked Read 8 Sent Done Done false
23210 6973 Acked Flush 0 Sent Done Done false
23211 6974 Acked Read 7 Sent Done Done false
23212 6975 Acked Read 5 Sent Done Done false
23213 6976 Acked Write 10 Sent Done Done false
23214 6977 Acked Read 5 Sent Done Done false
23215 6978 Acked Read 3 Sent Done Done false
23216 6979 Acked Write 7 Sent Done Done false
23217 6980 Acked Write 9 Sent Done Done false
23218 6981 Acked Write 1 Sent Done Done false
23219 6982 Acked Read 1 Sent Done Done false
23220 6983 Acked Write 4 Sent Done Done false
23221 6984 Acked Write 7 Sent Done Done false
23222 6985 Acked Read 8 Sent Done Done false
23223 6986 Acked Write 3 Sent Done Done false
23224 6987 Acked Read 2 Sent Done Done false
23225 6988 Acked Read 7 Sent Done Done false
23226 6989 Acked Write 9 Sent Done Done false
23227 6990 Acked Read 7 Sent Done Done false
23228 6991 Acked Write 9 Sent Done Done false
23229 6992 Acked Write 8 Sent Done Done false
23230 6993 Acked Read 9 Sent Done Done false
23231 6994 Acked Write 10 Sent Done Done false
23232 6995 Acked Read 2 Sent Done Done false
23233 6996 Acked Write 5 Sent Done Done false
23234 6997 Acked Read 8 Sent Done Done false
23235 6998 Acked Read 5 Sent Done Done false
23236 6999 Acked Read 10 Sent Done Done false
23237 7000 Acked Write 2 Sent Done Done false
23238 7001 Acked Read 9 Sent Done Done false
23239 7002 Acked Read 7 Sent Done Done false
23240 7003 Acked Read 1 Sent Done Done false
23241 7004 Acked Read 3 Sent Done Done false
23242 7005 Acked Read 1 Sent Done Done false
23243 7006 Acked Flush 0 Sent Done Done false
23244 7007 Acked Write 1 Sent Done Done false
23245 7008 Acked Write 6 Sent Done Done false
23246 7009 Acked Write 8 Sent Done Done false
23247 7010 Acked Read 8 Sent Done Done false
23248 7011 Acked Read 8 Sent Done Done false
23249 7012 Acked Write 9 Sent Done Done false
23250 7013 Acked Write 2 Sent Done Done false
23251 7014 Acked Write 10 Sent Done Done false
23252 7015 Acked Read 3 Sent Done Done false
23253 7016 Acked Read 5 Sent Done Done false
23254 7017 Acked Read 3 Sent Done Done false
23255 7018 Acked Write 8 Sent Done Done false
23256 7019 Acked Write 4 Sent Done Done false
23257 7020 Acked Read 4 Sent Done Done false
23258 7021 Acked Read 10 Sent Done Done false
23259 7022 Acked Write 6 Sent Done Done false
23260 7023 Acked Read 6 Sent Done Done false
23261 7024 Acked Read 7 Sent Done Done false
23262 7025 Acked Read 2 Sent Done Done false
23263 7026 Acked Read 9 Sent Done Done false
23264 7027 Acked Read 10 Sent Done Done false
23265 7028 Acked Write 9 Sent Done Done false
23266 7029 Acked Read 9 Sent Done Done false
23267 7030 Acked Flush 0 Sent Done Done false
23268 7031 Acked Write 9 Sent Done Done false
23269 7032 Acked Flush 0 Sent Done Done false
23270 7033 Acked Read 4 Sent Done Done false
23271 7034 Acked Flush 0 Sent Done Done false
23272 7035 Acked Write 8 Sent Done Done false
23273 7036 Acked Write 4 Sent Done Done false
23274 STATES DS:0 DS:1 DS:2 TOTAL
23275 Sent 301 0 0 301
23276 Done 0 301 301 602
23277 Skipped 0 0 0 0
23278 Error 0 0 0 0
23279 Last Flush: Some(JobId(6735)) Some(JobId(7034)) Some(JobId(7034))
23280 Downstairs last five completed: 6735 6734 6733 6732 6731
23281 Upstairs last five completed: 7036 7035 7034 7033 7032
23282 CLIENT: Up:0 ds:301 act:2
232832025-02-25T20:51:42.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232842025-02-25T20:51:43.497ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232852025-02-25T20:51:43.497ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34942) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232862025-02-25T20:51:43.498ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232872025-02-25T20:51:43.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232882025-02-25T20:51:43.499ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232892025-02-25T20:51:43.499ZINFOcrucible: send last flush ID to this DS: Some(JobId(6735)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232902025-02-25T20:51:43.499ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232912025-02-25T20:51:43.499ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6735)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
232922025-02-25T20:51:43.499ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6735)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
23293 ----------------------------------------------------------------
23294 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23295 Downstairs last five completed: 7037 7036 7035 7034 7033
23296 Upstairs last five completed: 7037 7036 7035 7034 7033
23297 CLIENT: Up:0 ds:0 act:3
23298 Replay: All jobs finished
23299 Main task has work to do, go find it
23300 got dsc Stop(1)
23301 stop 1
23302 Feb 25 20:51:45.758 INFO request completed, latency_us: 386, response_code: 204[8820] Got stop action so:false kr:false
23303 , uri: /stop/cid/1, method: POST, req_id: 0c21603a-7e8b-4ced-8a68-e1bcea2ed177, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
23304 Feb 25 20:51:45.759 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/1, method: GET, req_id: 60ad2ff5-5319-48fd-8079-a5f13b4242e5, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
233052025-02-25T20:51:45.759ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
233062025-02-25T20:51:45.759ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
233072025-02-25T20:51:45.759ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
233082025-02-25T20:51:45.759ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
23309 [8820] Exited with: ExitStatus(unix_wait_status(9))
23310 [8820][1] reports Exit
23311 Feb 25 20:51:49.762 INFO request completed, latency_us: 339, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2f83431f-3226-493f-83ff-d35d721eae30, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
23312 001/300 Write block 764 len 36864 data: 3 3 2 2 2 2 3 3 3
23313 002/300 Write block 2227 len 20480 data: 5 5 6 5 5
23314 003/300 Write block 4091 len 24576 data: 3 4 3 5 5 5
23315 004/300 Read block 3402 len 28672
23316 005/300 Flush
23317 006/300 Read block 1464 len 8192
23318 007/300 Flush
23319 008/300 Read block 4675 len 12288
23320 009/300 Write block 631 len 24576 data: 4 5 4 4 4 2
23321 010/300 Read block 4440 len 4096
23322 011/300 Read block 2064 len 12288
23323 012/300 Write block 2932 len 4096 data: 2
23324 013/300 Write block 1328 len 32768 data: 6 5 5 4 4 4 4 5
23325 014/300 Read block 1386 len 36864
23326 015/300 Read block 1951 len 32768
23327 016/300 Flush
23328 017/300 Write block 4982 len 40960 data: 2 3 4 4 4 5 5 6 5 6
23329 018/300 Write block 1954 len 16384 data: 6 5 5 5
23330 019/300 Read block 4893 len 12288
23331 020/300 Flush
23332 021/300 Write block 2517 len 24576 data: 7 8 8 6 5 5
23333 022/300 Write block 2221 len 12288 data: 6 6 6
23334 023/300 Write block 4125 len 4096 data: 7
23335 024/300 Flush
23336 025/300 Read block 359 len 28672
23337 026/300 Read block 3563 len 12288
23338 027/300 Read block 1848 len 32768
23339 028/300 Read block 4320 len 4096
23340 029/300 Read block 2452 len 40960
23341 030/300 Read block 1332 len 36864
23342 031/300 Write block 2577 len 20480 data: 3 2 4 4 4
23343 032/300 Write block 784 len 12288 data: 4 4 3
23344 033/300 Write block 3842 len 4096 data: 4
23345 034/300 Read block 3004 len 12288
23346 035/300 Flush
23347 036/300 Write block 3552 len 4096 data: 2
23348 037/300 Read block 4908 len 20480
23349 038/300 Read block 4062 len 40960
23350 039/300 Write block 3542 len 40960 data: 4 4 3 3 3 3 3 3 3 3
23351 040/300 Write block 3462 len 32768 data: 2 2 2 2 2 3 3 4
23352 041/300 Write block 4560 len 20480 data: 3 3 2 2 3
23353 042/300 Read block 1175 len 12288
23354 043/300 Read block 1974 len 24576
23355 044/300 Read block 1764 len 16384
23356 045/300 Write block 2966 len 16384 data: 3 3 3 4
23357 046/300 Read block 3816 len 4096
23358 047/300 Write block 3441 len 40960 data: 4 5 6 7 5 5 5 5 3 3
23359 048/300 Read block 4498 len 36864
23360 049/300 Write block 980 len 32768 data: 5 6 7 8 9 10 9 8
23361 050/300 Read block 4639 len 8192
23362 051/300 Write block 1869 len 16384 data: 4 5 4 3
23363 052/300 Flush
23364 053/300 Read block 4049 len 24576
23365 054/300 Read block 175 len 4096
23366 055/300 Write block 4927 len 40960 data: 2 2 2 2 2 4 3 3 4 6
23367 056/300 Write block 2160 len 16384 data: 10 8 7 6
23368 057/300 Read block 4471 len 20480
23369 058/300 Write block 3431 len 12288 data: 5 6 7
23370 059/300 Write block 3204 len 20480 data: 3 3 3 3 3
23371 060/300 Read block 182 len 36864
23372 061/300 Read block 3151 len 4096
23373 062/300 Write block 3328 len 40960 data: 5 5 5 5 4 4 5 5 5 5
23374 063/300 Write block 3140 len 24576 data: 5 4 5 5 7 7
23375 064/300 Flush
23376 065/300 Write block 1651 len 32768 data: 4 3 4 5 5 6 7 7
23377 066/300 Write block 2645 len 32768 data: 5 6 9 7 6 6 6 5
23378 067/300 Read block 886 len 16384
23379 068/300 Write block 16 len 24576 data: 4 3 3 3 3 3
23380 069/300 Write block 432 len 16384 data: 5 4 5 5
23381 070/300 Read block 391 len 20480
23382 071/300 Write block 2418 len 36864 data: 3 2 2 2 2 2 5 6 8
23383 072/300 Write block 1824 len 8192 data: 5 6
23384 073/300 Read block 236 len 36864
23385 074/300 Read block 4317 len 36864
23386 075/300 Read block 2490 len 20480
23387 076/300 Read block 3524 len 24576
23388 077/300 Read block 4789 len 20480
23389 078/300 Write block 1901 len 24576 data: 4 5 4 3 3 3
23390 079/300 Read block 3038 len 36864
23391 080/300 Read block 1042 len 20480
23392 081/300 Flush
23393 082/300 Write block 4122 len 36864 data: 7 8 6 8 6 5 6 6 6
23394 083/300 Read block 2368 len 4096
23395 084/300 Write block 4292 len 20480 data: 3 3 4 3 4
23396 085/300 Read block 225 len 32768
23397 086/300 Flush
23398 087/300 Write block 1153 len 28672 data: 4 4 2 3 3 3 3
23399 088/300 Write block 493 len 12288 data: 4 4 4
23400 089/300 Read block 908 len 12288
23401 090/300 Flush
23402 091/300 Read block 3449 len 16384
23403 092/300 Write block 4759 len 16384 data: 5 5 5 5
23404 093/300 Read block 3664 len 20480
23405 094/300 Write block 2338 len 36864 data: 8 8 8 10 10 10 5 5 4
23406 095/300 Flush
23407 096/300 Write block 3932 len 12288 data: 3 4 4
23408 097/300 Read block 1739 len 24576
23409 098/300 Flush
23410 099/300 Write block 2345 len 20480 data: 6 5 4 6 6
23411 100/300 Read block 3229 len 12288
23412 101/300 Write block 3610 len 40960 data: 5 6 7 6 4 4 5 6 5 6
23413 102/300 Read block 1958 len 24576
23414 103/300 Write block 2941 len 32768 data: 3 4 3 3 3 3 3 3
23415 104/300 Write block 107 len 32768 data: 4 5 5 5 4 2 2 2
23416 105/300 Write block 4635 len 36864 data: 6 6 5 3 4 4 2 2 2
23417 106/300 Read block 2965 len 40960
23418 107/300 Write block 1380 len 12288 data: 2 2 3
23419 108/300 Write block 4031 len 12288 data: 4 4 4
23420 109/300 Read block 1167 len 16384
23421 110/300 Read block 3007 len 36864
23422 111/300 Flush
23423 112/300 Write block 991 len 36864 data: 3 2 6 7 7 7 4 4 5
23424 113/300 Write block 2636 len 40960 data: 3 2 3 2 2 3 5 5 5 6
23425 114/300 Read block 2687 len 20480
23426 115/300 Read block 2460 len 24576
23427 116/300 Write block 1436 len 12288 data: 4 4 4
23428 117/300 Write block 2222 len 28672 data: 7 7 6 6 5 6 6
23429 118/300 Read block 3981 len 28672
23430 119/300 Write block 3599 len 4096 data: 4
23431 120/300 Read block 359 len 8192
23432 121/300 Read block 2497 len 24576
23433 122/300 Write block 2139 len 12288 data: 3 2 3
23434 123/300 Read block 2004 len 12288
23435 124/300 Write block 2975 len 12288 data: 3 3 3
23436 125/300 Read block 2074 len 32768
23437 126/300 Read block 3454 len 36864
23438 127/300 Read block 3910 len 16384
23439 128/300 Write block 2100 len 40960 data: 4 4 5 4 4 4 5 6 6 4
23440 129/300 Flush
23441 130/300 Read block 3712 len 4096
23442 131/300 Read block 2477 len 24576
23443 132/300 Write block 3377 len 24576 data: 4 4 4 6 7 6
23444 133/300 Read block 4717 len 16384
23445 134/300 Write block 695 len 8192 data: 5 4
23446 135/300 Read block 3655 len 32768
23447 136/300 Write block 2242 len 16384 data: 5 6 9 7
23448 137/300 Write block 577 len 4096 data: 6
23449 138/300 Read block 3231 len 32768
23450 139/300 Read block 3459 len 16384
23451 140/300 Read block 2225 len 16384
23452 141/300 Read block 1275 len 36864
23453 142/300 Read block 4827 len 20480
23454 143/300 Write block 2950 len 36864 data: 3 3 3 4 4 4 5 7 9
23455 144/300 Flush
23456 145/300 Write block 3218 len 28672 data: 4 4 4 4 3 3 3
23457 146/300 Read block 2910 len 12288
23458 147/300 Read block 2023 len 12288
23459 148/300 Flush
23460 149/300 Write block 1760 len 16384 data: 3 3 3 3
23461 150/300 Write block 4066 len 24576 data: 4 4 3 4 2 2
23462 151/300 Read block 4960 len 16384
23463 152/300 Read block 2504 len 36864
23464 153/300 Read block 4255 len 24576
23465 154/300 Read block 59 len 24576
23466 155/300 Write block 2374 len 40960 data: 4 5 6 8 7 8 5 4 5 6
23467 156/300 Read block 739 len 36864
23468 157/300 Read block 1186 len 28672
23469 158/300 Read block 213 len 4096
23470 159/300 Write block 3396 len 36864 data: 4 3 2 3 3 3 3 5 5
23471 160/300 Read block 2381 len 20480
23472 161/300 Write block 109 len 24576 data: 6 6 5 3 3 3
23473 162/300 Read block 2723 len 28672
23474 163/300 Read block 2267 len 40960
23475 164/300 Read block 3602 len 36864
23476 165/300 Read block 2047 len 16384
23477 166/300 Read block 3437 len 8192
23478 167/300 Read block 2250 len 28672
23479 168/300 Write block 2330 len 28672 data: 4 4 4 6 5 5 5
23480 169/300 Read block 2381 len 32768
23481 170/300 Write block 4918 len 20480 data: 5 5 7 7 6
23482 171/300 Read block 1230 len 32768
23483 172/300 Read block 2016 len 24576
23484 173/300 Write block 2590 len 4096 data: 4
23485 174/300 Write block 3792 len 24576 data: 4 4 4 4 4 5
23486 175/300 Write block 3485 len 20480 data: 4 4 4 5 5
23487 176/300 Flush
23488 177/300 Flush
23489 178/300 Flush
23490 179/300 Write block 4321 len 28672 data: 3 3 3 3 3 4 4
23491 180/300 Read block 2601 len 28672
23492 181/300 Write block 4589 len 32768 data: 4 4 4 4 4 4 5 5
23493 182/300 Read block 3806 len 36864
23494 183/300 Read block 4508 len 12288
23495 184/300 Read block 2229 len 16384
23496 185/300 Write block 3817 len 40960 data: 5 5 6 6 4 4 4 5 6 6
23497 186/300 Write block 4670 len 32768 data: 5 3 3 4 4 4 4 4
23498 187/300 Read block 1569 len 36864
23499 188/300 Read block 385 len 12288
23500 189/300 Write block 2612 len 16384 data: 6 5 4 5
23501 190/300 Write block 840 len 36864 data: 5 5 7 7 8 7 6 6 6
23502 191/300 Flush
23503 192/300 Write block 1555 len 40960 data: 5 3 5 6 7 8 8 9 5 5
23504 193/300 Read block 4873 len 36864
23505 194/300 Write block 2261 len 24576 data: 6 4 4 3 3 2
23506 195/300 Read block 1728 len 8192
23507 196/300 Write block 1458 len 12288 data: 8 8 7
23508 197/300 Read block 4757 len 4096
23509 198/300 Write block 181 len 32768 data: 4 4 4 4 4 5 5 5
23510 199/300 Write block 332 len 28672 data: 7 6 7 7 7 5 4
23511 200/300 Read block 4517 len 28672
23512 201/300 Read block 1376 len 28672
23513 202/300 Write block 2637 len 32768 data: 3 4 3 3 4 6 6 6
23514 203/300 Write block 2791 len 4096 data: 2
23515 204/300 Write block 4612 len 36864 data: 5 6 4 4 4 5 4 3 3
23516 205/300 Write block 1059 len 32768 data: 6 6 6 5 6 4 4 3
23517 206/300 Read block 4871 len 40960
23518 207/300 Write block 2471 len 12288 data: 7 6 6
23519 208/300 Read block 1646 len 40960
23520 209/300 Read block 777 len 24576
23521 210/300 Flush
23522 211/300 Read block 854 len 20480
23523 212/300 Write block 1022 len 12288 data: 6 6 8
23524 213/300 Write block 3660 len 32768 data: 8 7 7 7 5 5 5 3
23525 214/300 Write block 302 len 12288 data: 2 3 2
23526 215/300 Read block 3084 len 40960
23527 216/300 Read block 4830 len 12288
23528 217/300 Write block 588 len 40960 data: 5 5 7 6 5 6 4 4 4 4
23529 218/300 Write block 2892 len 16384 data: 3 6 6 7
23530 219/300 Read block 3559 len 40960
23531 220/300 Read block 545 len 8192
23532 221/300 Write block 553 len 36864 data: 7 7 8 8 5 5 5 6 6
23533 222/300 Write block 4643 len 24576 data: 3 2 2 2 2 2
23534 223/300 Write block 3397 len 12288 data: 4 3 4
23535 224/300 Write block 4294 len 40960 data: 5 4 5 4 4 4 6 5 5 4
23536 225/300 Read block 359 len 28672
23537 226/300 Write block 1438 len 4096 data: 5
23538 227/300 Read block 4492 len 12288
23539 228/300 Read block 4235 len 36864
23540 229/300 Read block 4533 len 36864
23541 230/300 Read block 2206 len 4096
23542 231/300 Write block 1342 len 16384 data: 4 4 4 5
23543 232/300 Read block 1834 len 8192
23544 233/300 Read block 1765 len 16384
23545 234/300 Write block 3718 len 12288 data: 5 6 7
23546 235/300 Write block 609 len 32768 data: 3 3 3 4 5 5 5 6
23547 236/300 Write block 2305 len 12288 data: 3 4 4
23548 237/300 Write block 3356 len 12288 data: 3 3 4
23549 238/300 Read block 2182 len 8192
23550 239/300 Read block 3421 len 40960
23551 240/300 Write block 256 len 32768 data: 5 5 4 4 3 3 4 4
23552 241/300 Read block 2074 len 24576
23553 242/300 Read block 4324 len 16384
23554 243/300 Read block 4310 len 8192
23555 244/300 Read block 2992 len 12288
23556 245/300 Write block 2150 len 20480 data: 4 5 6 5 5
23557 246/300 Write block 1786 len 32768 data: 7 6 6 7 8 7 7 7
23558 247/300 Read block 4216 len 32768
23559 248/300 Read block 1686 len 16384
23560 249/300 Write block 1317 len 40960 data: 5 6 7 10 10 9 10 9 10 8
23561 250/300 Write block 3022 len 24576 data: 4 4 3 3 4 5
23562 251/300 Write block 4461 len 8192 data: 9 10
23563 252/300 Read block 2181 len 24576
23564 253/300 Write block 2140 len 36864 data: 3 4 3 3 3 3 3 3 4
23565 254/300 Write block 4748 len 40960 data: 5 6 6 6 6 7 6 7 7 4
23566 255/300 Write block 1868 len 8192 data: 4 5
23567 256/300 Write block 2012 len 40960 data: 6 5 5 5 5 5 5 6 5 6
23568 257/300 Write block 1561 len 32768 data: 9 10 6 6 4 5 5 4
23569 258/300 Write block 3059 len 40960 data: 3 4 3 5 6 6 5 5 6 6
23570 259/300 Flush
23571 260/300 Read block 4097 len 8192
23572 261/300 Read block 3179 len 4096
23573 262/300 Read block 1334 len 20480
23574 263/300 Read block 1671 len 4096
23575 264/300 Read block 1298 len 40960
23576 265/300 Write block 3697 len 16384 data: 4 5 5 5
23577 266/300 Write block 4126 len 28672 data: 7 6 7 7 7 6 4
23578 267/300 Read block 1337 len 36864
23579 268/300 Write block 2414 len 36864 data: 5 5 4 5 4 3 3 3 3
23580 269/300 Read block 4787 len 24576
23581 270/300 Write block 1768 len 36864 data: 4 3 4 4 4 4 4 6 6
23582 271/300 Read block 366 len 12288
23583 272/300 Write block 3515 len 20480 data: 4 4 4 4 3
23584 273/300 Read block 613 len 16384
23585 274/300 Write block 2799 len 8192 data: 5 8
23586 275/300 Flush
23587 276/300 Read block 2882 len 20480
23588 277/300 Read block 3780 len 4096
23589 278/300 Read block 4202 len 4096
23590 279/300 Read block 1028 len 24576
23591 280/300 Read block 3690 len 36864
23592 281/300 Write block 2956 len 36864 data: 6 8 10 8 7 8 6 7 6
23593 282/300 Read block 4618 len 20480
23594 283/300 Write block 63 len 16384 data: 4 4 5 3
23595 284/300 Read block 4111 len 20480
23596 285/300 Write block 3200 len 40960 data: 6 6 4 3 4 4 4 4 4 4
23597 286/300 Read block 3234 len 16384
23598 287/300 Read block 2508 len 16384
23599 288/300 Read block 2782 len 4096
23600 289/300 Read block 2732 len 4096
23601 290/300 Write block 3418 len 24576 data: 8 8 8 8 9 8
23602 291/300 Read block 3033 len 28672
23603 292/300 Read block 4583 len 28672
23604 293/300 Read block 850 len 36864
23605 294/300 Read block 836 len 8192
23606 295/300 Read block 1006 len 4096
23607 296/300 Read block 2365 len 4096
23608 297/300 Read block 3727 len 12288
23609 298/300 Read block 2852 len 4096
23610 299/300 Write block 1964 len 28672 data: 4 4 5 5 5 4 5
23611 300/300 Read block 4731 len 20480
23612 301/300 Flush
23613 Main task has work to do, go find it
23614 got dsc Start(1)
23615 start 1
23616 Feb 25 20:51:50.010 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: c6af691a-21c5-4090-9cf9-00f8bc925b4a, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
23617 Replay: started 1, returned:Ok(())
23618 ----------------------------------------------------------------
23619 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23620 States: Active Offline Active
23621 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23622 7038 Acked Write 9 Done Sent Done false
23623 7039 Acked Write 5 Done Sent Done false
23624 7040 Acked Write 6 Done Sent Done false
23625 7041 Acked Read 7 Done Sent Done false
23626 7042 Acked Flush 0 Done Sent Done false
23627 7043 Acked Read 2 Done Sent Done false
23628 7044 Acked Flush 0 Done Sent Done false
23629 7045 Acked Read 3 Done Sent Done false
23630 7046 Acked Write 6 Done Sent Done false
23631 7047 Acked Read 1 Done Sent Done false
23632 7048 Acked Read 3 Done Sent Done false
23633 7049 Acked Write 1 Done Sent Done false
23634 7050 Acked Write 8 Done Sent Done false
23635 7051 Acked Read 9 Done Sent Done false
23636 7052 Acked Read 8 Done Sent Done false
23637 7053 Acked Flush 0 Done Sent Done false
23638 7054 Acked Write 10 Done Sent Done false
23639 7055 Acked Write 4 Done Sent Done false
23640 7056 Acked Read 3 Done Sent Done false
23641 7057 Acked Flush 0 Done Sent Done false
23642 7058 Acked Write 6 Done Sent Done false
23643 7059 Acked Write 3 Done Sent Done false
23644 7060 Acked Write 1 Done Sent Done false
23645 7061 Acked Flush 0 Done Sent Done false
23646 7062 Acked Read 7 Done Sent Done false
23647 7063 Acked Read 3 Done Sent Done false
23648 7064 Acked Read 8 Done Sent Done false
23649 7065 Acked Read 1 Done Sent Done false
23650 7066 Acked Read 10 Done Sent Done false
23651 7067 Acked Read 9 Done Sent Done false
23652 7068 Acked Write 5 Done Sent Done false
23653 7069 Acked Write 3 Done Sent Done false
23654 7070 Acked Write 1 Done Sent Done false
23655 7071 Acked Read 3 Done Sent Done false
23656 7072 Acked Flush 0 Done Sent Done false
23657 7073 Acked Write 1 Done Sent Done false
23658 7074 Acked Read 5 Done Sent Done false
23659 7075 Acked Read 10 Done Sent Done false
23660 7076 Acked Write 10 Done Sent Done false
23661 7077 Acked Write 8 Done Sent Done false
23662 7078 Acked Write 5 Done Sent Done false
23663 7079 Acked Read 3 Done Sent Done false
23664 7080 Acked Read 6 Done Sent Done false
23665 7081 Acked Read 4 Done Sent Done false
23666 7082 Acked Write 4 Done Sent Done false
23667 7083 Acked Read 1 Done Sent Done false
23668 7084 Acked Write 10 Done Sent Done false
23669 7085 Acked Read 9 Done Sent Done false
23670 7086 Acked Write 8 Done Sent Done false
23671 7087 Acked Read 2 Done Sent Done false
23672 7088 Acked Write 4 Done Sent Done false
23673 7089 Acked Flush 0 Done Sent Done false
23674 7090 Acked Read 6 Done Sent Done false
23675 7091 Acked Read 1 Done Sent Done false
23676 7092 Acked Write 10 Done Sent Done false
23677 7093 Acked Write 4 Done Sent Done false
23678 7094 Acked Read 5 Done Sent Done false
23679 7095 Acked Write 3 Done Sent Done false
23680 7096 Acked Write 5 Done Sent Done false
23681 7097 Acked Read 9 Done Sent Done false
23682 7098 Acked Read 1 Done Sent Done false
23683 7099 Acked Write 10 Done Sent Done false
23684 7100 Acked Write 6 Done Sent Done false
23685 7101 Acked Flush 0 Done Sent Done false
23686 7102 Acked Write 8 Done Sent Done false
23687 7103 Acked Write 8 Done Sent Done false
23688 7104 Acked Read 4 Done Sent Done false
23689 7105 Acked Write 6 Done Sent Done false
23690 7106 Acked Write 4 Done Sent Done false
23691 7107 Acked Read 5 Done Sent Done false
23692 7108 Acked Write 9 Done Sent Done false
23693 7109 Acked Write 2 Done Sent Done false
23694 7110 Acked Read 9 Done Sent Done false
23695 7111 Acked Read 9 Done Sent Done false
23696 7112 Acked Read 5 Done Sent Done false
23697 7113 Acked Read 6 Done Sent Done false
23698 7114 Acked Read 5 Done Sent Done false
23699 7115 Acked Write 6 Done Sent Done false
23700 7116 Acked Read 9 Done Sent Done false
23701 7117 Acked Read 5 Done Sent Done false
23702 7118 Acked Flush 0 Done Sent Done false
23703 7119 Acked Write 9 Done Sent Done false
23704 7120 Acked Read 1 Done Sent Done false
23705 7121 Acked Write 5 Done Sent Done false
23706 7122 Acked Read 8 Done Sent Done false
23707 7123 Acked Flush 0 Done Sent Done false
23708 7124 Acked Write 7 Done Sent Done false
23709 7125 Acked Write 3 Done Sent Done false
23710 7126 Acked Read 3 Done Sent Done false
23711 7127 Acked Flush 0 Done Sent Done false
23712 7128 Acked Read 4 Done Sent Done false
23713 7129 Acked Write 4 Done Sent Done false
23714 7130 Acked Read 5 Done Sent Done false
23715 7131 Acked Write 9 Done Sent Done false
23716 7132 Acked Flush 0 Done Sent Done false
23717 7133 Acked Write 3 Done Sent Done false
23718 7134 Acked Read 6 Done Sent Done false
23719 7135 Acked Flush 0 Done Sent Done false
23720 7136 Acked Write 5 Done Sent Done false
23721 7137 Acked Read 3 Done Sent Done false
23722 7138 Acked Write 10 Done Sent Done false
23723 7139 Acked Read 6 Done Sent Done false
23724 7140 Acked Write 8 Done Sent Done false
23725 7141 Acked Write 8 Done Sent Done false
23726 7142 Acked Write 9 Done Sent Done false
23727 7143 Acked Read 10 Done Sent Done false
23728 7144 Acked Write 3 Done Sent Done false
23729 7145 Acked Write 3 Done Sent Done false
23730 7146 Acked Read 4 Done Sent Done false
23731 7147 Acked Read 9 Done Sent Done false
23732 7148 Acked Flush 0 Done Sent Done false
23733 7149 Acked Write 9 Done Sent Done false
23734 7150 Acked Write 10 Done Sent Done false
23735 7151 Acked Read 5 Done Sent Done false
23736 7152 Acked Read 6 Done Sent Done false
23737 7153 Acked Write 3 Done Sent Done false
23738 7154 Acked Write 7 Done Sent Done false
23739 7155 Acked Read 7 Done Sent Done false
23740 7156 Acked Write 1 Done Sent Done false
23741 7157 Acked Read 2 Done Sent Done false
23742 7158 Acked Read 6 Done Sent Done false
23743 7159 Acked Write 3 Done Sent Done false
23744 7160 Acked Read 3 Done Sent Done false
23745 7161 Acked Write 3 Done Sent Done false
23746 7162 Acked Read 8 Done Sent Done false
23747 7163 Acked Read 9 Done Sent Done false
23748 7164 Acked Read 4 Done Sent Done false
23749 7165 Acked Write 10 Done Sent Done false
23750 7166 Acked Flush 0 Done Sent Done false
23751 7167 Acked Read 1 Done Sent Done false
23752 7168 Acked Read 6 Done Sent Done false
23753 7169 Acked Write 6 Done Sent Done false
23754 7170 Acked Read 4 Done Sent Done false
23755 7171 Acked Write 2 Done Sent Done false
23756 7172 Acked Read 8 Done Sent Done false
23757 7173 Acked Write 4 Done Sent Done false
23758 7174 Acked Write 1 Done Sent Done false
23759 7175 Acked Read 8 Done Sent Done false
23760 7176 Acked Read 4 Done Sent Done false
23761 7177 Acked Read 4 Done Sent Done false
23762 7178 Acked Read 9 Done Sent Done false
23763 7179 Acked Read 5 Done Sent Done false
23764 7180 Acked Write 9 Done Sent Done false
23765 7181 Acked Flush 0 Done Sent Done false
23766 7182 Acked Write 7 Done Sent Done false
23767 7183 Acked Read 3 Done Sent Done false
23768 7184 Acked Read 3 Done Sent Done false
23769 7185 Acked Flush 0 Done Sent Done false
23770 7186 Acked Write 4 Done Sent Done false
23771 7187 Acked Write 6 Done Sent Done false
23772 7188 Acked Read 4 Done Sent Done false
23773 7189 Acked Read 9 Done Sent Done false
23774 7190 Acked Read 6 Done Sent Done false
23775 7191 Acked Read 6 Done Sent Done false
23776 7192 Acked Write 10 Done Sent Done false
23777 7193 Acked Read 9 Done Sent Done false
23778 7194 Acked Read 7 Done Sent Done false
23779 7195 Acked Read 1 Done Sent Done false
23780 7196 Acked Write 9 Done Sent Done false
23781 7197 Acked Read 5 Done Sent Done false
23782 7198 Acked Write 6 Done Sent Done false
23783 7199 Acked Read 7 Done Sent Done false
23784 7200 Acked Read 10 Done Sent Done false
23785 7201 Acked Read 9 Done Sent Done false
23786 7202 Acked Read 4 Done Sent Done false
23787 7203 Acked Read 2 Done Sent Done false
23788 7204 Acked Read 7 Done Sent Done false
23789 7205 Acked Write 7 Done Sent Done false
23790 7206 Acked Read 8 Done Sent Done false
23791 7207 Acked Write 5 Done Sent Done false
23792 7208 Acked Read 8 Done Sent Done false
23793 7209 Acked Read 6 Done Sent Done false
23794 7210 Acked Write 1 Done Sent Done false
23795 7211 Acked Write 6 Done Sent Done false
23796 7212 Acked Write 5 Done Sent Done false
23797 7213 Acked Flush 0 Done Sent Done false
23798 7214 Acked Flush 0 Done Sent Done false
23799 7215 Acked Flush 0 Done Sent Done false
23800 7216 Acked Write 7 Done Sent Done false
23801 7217 Acked Read 7 Done Sent Done false
23802 7218 Acked Write 8 Done Sent Done false
23803 7219 Acked Read 9 Done Sent Done false
23804 7220 Acked Read 3 Done Sent Done false
23805 7221 Acked Read 4 Done Sent Done false
23806 7222 Acked Write 10 Done Sent Done false
23807 7223 Acked Write 8 Done Sent Done false
23808 7224 Acked Read 9 Done Sent Done false
23809 7225 Acked Read 3 Done Sent Done false
23810 7226 Acked Write 4 Done Sent Done false
23811 7227 Acked Write 9 Done Sent Done false
23812 7228 Acked Flush 0 Done Sent Done false
23813 7229 Acked Write 10 Done Sent Done false
23814 7230 Acked Read 9 Done Sent Done false
23815 7231 Acked Write 6 Done Sent Done false
23816 7232 Acked Read 2 Done Sent Done false
23817 7233 Acked Write 3 Done Sent Done false
23818 7234 Acked Read 1 Done Sent Done false
23819 7235 Acked Write 8 Done Sent Done false
23820 7236 Acked Write 7 Done Sent Done false
23821 7237 Acked Read 7 Done Sent Done false
23822 7238 Acked Read 7 Done Sent Done false
23823 7239 Acked Write 8 Done Sent Done false
23824 7240 Acked Write 1 Done Sent Done false
23825 7241 Acked Write 9 Done Sent Done false
23826 7242 Acked Write 8 Done Sent Done false
23827 7243 Acked Read 10 Done Sent Done false
23828 7244 Acked Write 3 Done Sent Done false
23829 7245 Acked Read 10 Done Sent Done false
23830 7246 Acked Read 6 Done Sent Done false
23831 7247 Acked Flush 0 Done Sent Done false
23832 7248 Acked Read 5 Done Sent Done false
23833 7249 Acked Write 3 Done Sent Done false
23834 7250 Acked Write 8 Done Sent Done false
23835 7251 Acked Write 3 Done Sent Done false
23836 7252 Acked Read 10 Done Sent Done false
23837 7253 Acked Read 3 Done Sent Done false
23838 7254 Acked Write 10 Done Sent Done false
23839 7255 Acked Write 4 Done Sent Done false
23840 7256 Acked Read 10 Done Sent Done false
23841 7257 Acked Read 2 Done Sent Done false
23842 7258 Acked Write 9 Done Sent Done false
23843 7259 Acked Write 6 Done Sent Done false
23844 7260 Acked Write 3 Done Sent Done false
23845 7261 Acked Write 10 Done Sent Done false
23846 7262 Acked Read 7 Done Sent Done false
23847 7263 Acked Write 1 Done Sent Done false
23848 7264 Acked Read 3 Done Sent Done false
23849 7265 Acked Read 9 Done Sent Done false
23850 7266 Acked Read 9 Done Sent Done false
23851 7267 Acked Read 1 Done Sent Done false
23852 7268 Acked Write 4 Done Sent Done false
23853 7269 Acked Read 2 Done Sent Done false
23854 7270 Acked Read 4 Done Sent Done false
23855 7271 Acked Write 3 Done Sent Done false
23856 7272 Acked Write 8 Done Sent Done false
23857 7273 Acked Write 3 Done Sent Done false
23858 7274 Acked Write 3 Done Sent Done false
23859 7275 Acked Read 2 Done Sent Done false
23860 7276 Acked Read 10 Done Sent Done false
23861 7277 Acked Write 8 Done Sent Done false
23862 7278 Acked Read 6 Done Sent Done false
23863 7279 Acked Read 4 Done Sent Done false
23864 7280 Acked Read 2 Done Sent Done false
23865 7281 Acked Read 3 Done Sent Done false
23866 7282 Acked Write 5 Done Sent Done false
23867 7283 Acked Write 8 Done Sent Done false
23868 7284 Acked Read 8 Done Sent Done false
23869 7285 Acked Read 4 Done Sent Done false
23870 7286 Acked Write 10 Done Sent Done false
23871 7287 Acked Write 6 Done Sent Done false
23872 7288 Acked Write 2 Done Sent Done false
23873 7289 Acked Read 6 Done Sent Done false
23874 7290 Acked Write 9 Done Sent Done false
23875 7291 Acked Write 10 Done Sent Done false
23876 7292 Acked Write 2 Done Sent Done false
23877 7293 Acked Write 10 Done Sent Done false
23878 7294 Acked Write 8 Done Sent Done false
23879 7295 Acked Write 10 Done Sent Done false
23880 7296 Acked Flush 0 Done Sent Done false
23881 7297 Acked Read 2 Done Sent Done false
23882 7298 Acked Read 1 Done Sent Done false
23883 7299 Acked Read 5 Done Sent Done false
23884 7300 Acked Read 1 Done Sent Done false
23885 7301 Acked Read 10 Done Sent Done false
23886 7302 Acked Write 4 Done Sent Done false
23887 7303 Acked Write 7 Done Sent Done false
23888 7304 Acked Read 9 Done Sent Done false
23889 7305 Acked Write 9 Done Sent Done false
23890 7306 Acked Read 6 Done Sent Done false
23891 7307 Acked Write 9 Done Sent Done false
23892 7308 Acked Read 3 Done Sent Done false
23893 7309 Acked Write 5 Done Sent Done false
23894 7310 Acked Read 4 Done Sent Done false
23895 7311 Acked Write 2 Done Sent Done false
23896 7312 Acked Flush 0 Done Sent Done false
23897 7313 Acked Read 5 Done Sent Done false
23898 7314 Acked Read 1 Done Sent Done false
23899 7315 Acked Read 1 Done Sent Done false
23900 7316 Acked Read 6 Done Sent Done false
23901 7317 Acked Read 9 Done Sent Done false
23902 7318 Acked Write 9 Done Sent Done false
23903 7319 Acked Read 5 Done Sent Done false
23904 7320 Acked Write 4 Done Sent Done false
23905 7321 Acked Read 5 Done Sent Done false
23906 7322 Acked Write 10 Done Sent Done false
23907 7323 Acked Read 4 Done Sent Done false
23908 7324 Acked Read 4 Done Sent Done false
23909 7325 Acked Read 1 Done Sent Done false
23910 7326 Acked Read 1 Done Sent Done false
23911 7327 Acked Write 6 Done Sent Done false
23912 7328 Acked Read 7 Done Sent Done false
23913 7329 Acked Read 7 Done Sent Done false
23914 7330 Acked Read 9 Done Sent Done false
23915 7331 Acked Read 2 Done Sent Done false
23916 7332 Acked Read 1 Done Sent Done false
23917 7333 Acked Read 1 Done Sent Done false
23918 7334 Acked Read 3 Done Sent Done false
23919 7335 Acked Read 1 Done Sent Done false
23920 7336 Acked Write 7 Done Sent Done false
23921 7337 Acked Read 5 Done Sent Done false
23922 7338 Acked Flush 0 Done Sent Done false
23923 STATES DS:0 DS:1 DS:2 TOTAL
23924 Sent 0 301 0 301
23925 Done 301 0 301 602
23926 Skipped 0 0 0 0
23927 Error 0 0 0 0
23928 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
23929 Downstairs last five completed: 7037 7036 7035 7034 7033
23930 Upstairs last five completed: 7338 7337 7336 7335 7334
23931 CLIENT: Up:0 ds:301 act:2
239322025-02-25T20:51:50.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
239332025-02-25T20:51:51.557ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
239342025-02-25T20:51:52.559ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
239352025-02-25T20:51:54.001ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
23936 ----------------------------------------------------------------
23937 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23938 States: Active Offline Active
23939 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23940 7038 Acked Write 9 Done Sent Done false
23941 7039 Acked Write 5 Done Sent Done false
23942 7040 Acked Write 6 Done Sent Done false
23943 7041 Acked Read 7 Done Sent Done false
23944 7042 Acked Flush 0 Done Sent Done false
23945 7043 Acked Read 2 Done Sent Done false
23946 7044 Acked Flush 0 Done Sent Done false
23947 7045 Acked Read 3 Done Sent Done false
23948 7046 Acked Write 6 Done Sent Done false
23949 7047 Acked Read 1 Done Sent Done false
23950 7048 Acked Read 3 Done Sent Done false
23951 7049 Acked Write 1 Done Sent Done false
23952 7050 Acked Write 8 Done Sent Done false
23953 7051 Acked Read 9 Done Sent Done false
23954 7052 Acked Read 8 Done Sent Done false
23955 7053 Acked Flush 0 Done Sent Done false
23956 7054 Acked Write 10 Done Sent Done false
23957 7055 Acked Write 4 Done Sent Done false
23958 7056 Acked Read 3 Done Sent Done false
23959 7057 Acked Flush 0 Done Sent Done false
23960 7058 Acked Write 6 Done Sent Done false
23961 7059 Acked Write 3 Done Sent Done false
23962 7060 Acked Write 1 Done Sent Done false
23963 7061 Acked Flush 0 Done Sent Done false
23964 7062 Acked Read 7 Done Sent Done false
23965 7063 Acked Read 3 Done Sent Done false
23966 7064 Acked Read 8 Done Sent Done false
23967 7065 Acked Read 1 Done Sent Done false
23968 7066 Acked Read 10 Done Sent Done false
23969 7067 Acked Read 9 Done Sent Done false
23970 7068 Acked Write 5 Done Sent Done false
23971 7069 Acked Write 3 Done Sent Done false
23972 7070 Acked Write 1 Done Sent Done false
23973 7071 Acked Read 3 Done Sent Done false
23974 7072 Acked Flush 0 Done Sent Done false
23975 7073 Acked Write 1 Done Sent Done false
23976 7074 Acked Read 5 Done Sent Done false
23977 7075 Acked Read 10 Done Sent Done false
23978 7076 Acked Write 10 Done Sent Done false
23979 7077 Acked Write 8 Done Sent Done false
23980 7078 Acked Write 5 Done Sent Done false
23981 7079 Acked Read 3 Done Sent Done false
23982 7080 Acked Read 6 Done Sent Done false
23983 7081 Acked Read 4 Done Sent Done false
23984 7082 Acked Write 4 Done Sent Done false
23985 7083 Acked Read 1 Done Sent Done false
23986 7084 Acked Write 10 Done Sent Done false
23987 7085 Acked Read 9 Done Sent Done false
23988 7086 Acked Write 8 Done Sent Done false
23989 7087 Acked Read 2 Done Sent Done false
23990 7088 Acked Write 4 Done Sent Done false
23991 7089 Acked Flush 0 Done Sent Done false
23992 7090 Acked Read 6 Done Sent Done false
23993 7091 Acked Read 1 Done Sent Done false
23994 7092 Acked Write 10 Done Sent Done false
23995 7093 Acked Write 4 Done Sent Done false
23996 7094 Acked Read 5 Done Sent Done false
23997 7095 Acked Write 3 Done Sent Done false
23998 7096 Acked Write 5 Done Sent Done false
23999 7097 Acked Read 9 Done Sent Done false
24000 7098 Acked Read 1 Done Sent Done false
24001 7099 Acked Write 10 Done Sent Done false
24002 7100 Acked Write 6 Done Sent Done false
24003 7101 Acked Flush 0 Done Sent Done false
24004 7102 Acked Write 8 Done Sent Done false
24005 7103 Acked Write 8 Done Sent Done false
24006 7104 Acked Read 4 Done Sent Done false
24007 7105 Acked Write 6 Done Sent Done false
24008 7106 Acked Write 4 Done Sent Done false
24009 7107 Acked Read 5 Done Sent Done false
24010 7108 Acked Write 9 Done Sent Done false
24011 7109 Acked Write 2 Done Sent Done false
24012 7110 Acked Read 9 Done Sent Done false
24013 7111 Acked Read 9 Done Sent Done false
24014 7112 Acked Read 5 Done Sent Done false
24015 7113 Acked Read 6 Done Sent Done false
24016 7114 Acked Read 5 Done Sent Done false
24017 7115 Acked Write 6 Done Sent Done false
24018 7116 Acked Read 9 Done Sent Done false
24019 7117 Acked Read 5 Done Sent Done false
24020 7118 Acked Flush 0 Done Sent Done false
24021 7119 Acked Write 9 Done Sent Done false
24022 7120 Acked Read 1 Done Sent Done false
24023 7121 Acked Write 5 Done Sent Done false
24024 7122 Acked Read 8 Done Sent Done false
24025 7123 Acked Flush 0 Done Sent Done false
24026 7124 Acked Write 7 Done Sent Done false
24027 7125 Acked Write 3 Done Sent Done false
24028 7126 Acked Read 3 Done Sent Done false
24029 7127 Acked Flush 0 Done Sent Done false
24030 7128 Acked Read 4 Done Sent Done false
24031 7129 Acked Write 4 Done Sent Done false
24032 7130 Acked Read 5 Done Sent Done false
24033 7131 Acked Write 9 Done Sent Done false
24034 7132 Acked Flush 0 Done Sent Done false
24035 7133 Acked Write 3 Done Sent Done false
24036 7134 Acked Read 6 Done Sent Done false
24037 7135 Acked Flush 0 Done Sent Done false
24038 7136 Acked Write 5 Done Sent Done false
24039 7137 Acked Read 3 Done Sent Done false
24040 7138 Acked Write 10 Done Sent Done false
24041 7139 Acked Read 6 Done Sent Done false
24042 7140 Acked Write 8 Done Sent Done false
24043 7141 Acked Write 8 Done Sent Done false
24044 7142 Acked Write 9 Done Sent Done false
24045 7143 Acked Read 10 Done Sent Done false
24046 7144 Acked Write 3 Done Sent Done false
24047 7145 Acked Write 3 Done Sent Done false
24048 7146 Acked Read 4 Done Sent Done false
24049 7147 Acked Read 9 Done Sent Done false
24050 7148 Acked Flush 0 Done Sent Done false
24051 7149 Acked Write 9 Done Sent Done false
24052 7150 Acked Write 10 Done Sent Done false
24053 7151 Acked Read 5 Done Sent Done false
24054 7152 Acked Read 6 Done Sent Done false
24055 7153 Acked Write 3 Done Sent Done false
24056 7154 Acked Write 7 Done Sent Done false
24057 7155 Acked Read 7 Done Sent Done false
24058 7156 Acked Write 1 Done Sent Done false
24059 7157 Acked Read 2 Done Sent Done false
24060 7158 Acked Read 6 Done Sent Done false
24061 7159 Acked Write 3 Done Sent Done false
24062 7160 Acked Read 3 Done Sent Done false
24063 7161 Acked Write 3 Done Sent Done false
24064 7162 Acked Read 8 Done Sent Done false
24065 7163 Acked Read 9 Done Sent Done false
24066 7164 Acked Read 4 Done Sent Done false
24067 7165 Acked Write 10 Done Sent Done false
24068 7166 Acked Flush 0 Done Sent Done false
24069 7167 Acked Read 1 Done Sent Done false
24070 7168 Acked Read 6 Done Sent Done false
24071 7169 Acked Write 6 Done Sent Done false
24072 7170 Acked Read 4 Done Sent Done false
24073 7171 Acked Write 2 Done Sent Done false
24074 7172 Acked Read 8 Done Sent Done false
24075 7173 Acked Write 4 Done Sent Done false
24076 7174 Acked Write 1 Done Sent Done false
24077 7175 Acked Read 8 Done Sent Done false
24078 7176 Acked Read 4 Done Sent Done false
24079 7177 Acked Read 4 Done Sent Done false
24080 7178 Acked Read 9 Done Sent Done false
24081 7179 Acked Read 5 Done Sent Done false
24082 7180 Acked Write 9 Done Sent Done false
24083 7181 Acked Flush 0 Done Sent Done false
24084 7182 Acked Write 7 Done Sent Done false
24085 7183 Acked Read 3 Done Sent Done false
24086 7184 Acked Read 3 Done Sent Done false
24087 7185 Acked Flush 0 Done Sent Done false
24088 7186 Acked Write 4 Done Sent Done false
24089 7187 Acked Write 6 Done Sent Done false
24090 7188 Acked Read 4 Done Sent Done false
24091 7189 Acked Read 9 Done Sent Done false
24092 7190 Acked Read 6 Done Sent Done false
24093 7191 Acked Read 6 Done Sent Done false
24094 7192 Acked Write 10 Done Sent Done false
24095 7193 Acked Read 9 Done Sent Done false
24096 7194 Acked Read 7 Done Sent Done false
24097 7195 Acked Read 1 Done Sent Done false
24098 7196 Acked Write 9 Done Sent Done false
24099 7197 Acked Read 5 Done Sent Done false
24100 7198 Acked Write 6 Done Sent Done false
24101 7199 Acked Read 7 Done Sent Done false
24102 7200 Acked Read 10 Done Sent Done false
24103 7201 Acked Read 9 Done Sent Done false
24104 7202 Acked Read 4 Done Sent Done false
24105 7203 Acked Read 2 Done Sent Done false
24106 7204 Acked Read 7 Done Sent Done false
24107 7205 Acked Write 7 Done Sent Done false
24108 7206 Acked Read 8 Done Sent Done false
24109 7207 Acked Write 5 Done Sent Done false
24110 7208 Acked Read 8 Done Sent Done false
24111 7209 Acked Read 6 Done Sent Done false
24112 7210 Acked Write 1 Done Sent Done false
24113 7211 Acked Write 6 Done Sent Done false
24114 7212 Acked Write 5 Done Sent Done false
24115 7213 Acked Flush 0 Done Sent Done false
24116 7214 Acked Flush 0 Done Sent Done false
24117 7215 Acked Flush 0 Done Sent Done false
24118 7216 Acked Write 7 Done Sent Done false
24119 7217 Acked Read 7 Done Sent Done false
24120 7218 Acked Write 8 Done Sent Done false
24121 7219 Acked Read 9 Done Sent Done false
24122 7220 Acked Read 3 Done Sent Done false
24123 7221 Acked Read 4 Done Sent Done false
24124 7222 Acked Write 10 Done Sent Done false
24125 7223 Acked Write 8 Done Sent Done false
24126 7224 Acked Read 9 Done Sent Done false
24127 7225 Acked Read 3 Done Sent Done false
24128 7226 Acked Write 4 Done Sent Done false
24129 7227 Acked Write 9 Done Sent Done false
24130 7228 Acked Flush 0 Done Sent Done false
24131 7229 Acked Write 10 Done Sent Done false
24132 7230 Acked Read 9 Done Sent Done false
24133 7231 Acked Write 6 Done Sent Done false
24134 7232 Acked Read 2 Done Sent Done false
24135 7233 Acked Write 3 Done Sent Done false
24136 7234 Acked Read 1 Done Sent Done false
24137 7235 Acked Write 8 Done Sent Done false
24138 7236 Acked Write 7 Done Sent Done false
24139 7237 Acked Read 7 Done Sent Done false
24140 7238 Acked Read 7 Done Sent Done false
24141 7239 Acked Write 8 Done Sent Done false
24142 7240 Acked Write 1 Done Sent Done false
24143 7241 Acked Write 9 Done Sent Done false
24144 7242 Acked Write 8 Done Sent Done false
24145 7243 Acked Read 10 Done Sent Done false
24146 7244 Acked Write 3 Done Sent Done false
24147 7245 Acked Read 10 Done Sent Done false
24148 7246 Acked Read 6 Done Sent Done false
24149 7247 Acked Flush 0 Done Sent Done false
24150 7248 Acked Read 5 Done Sent Done false
24151 7249 Acked Write 3 Done Sent Done false
24152 7250 Acked Write 8 Done Sent Done false
24153 7251 Acked Write 3 Done Sent Done false
24154 7252 Acked Read 10 Done Sent Done false
24155 7253 Acked Read 3 Done Sent Done false
24156 7254 Acked Write 10 Done Sent Done false
24157 7255 Acked Write 4 Done Sent Done false
24158 7256 Acked Read 10 Done Sent Done false
24159 7257 Acked Read 2 Done Sent Done false
24160 7258 Acked Write 9 Done Sent Done false
24161 7259 Acked Write 6 Done Sent Done false
24162 7260 Acked Write 3 Done Sent Done false
24163 7261 Acked Write 10 Done Sent Done false
24164 7262 Acked Read 7 Done Sent Done false
24165 7263 Acked Write 1 Done Sent Done false
24166 7264 Acked Read 3 Done Sent Done false
24167 7265 Acked Read 9 Done Sent Done false
24168 7266 Acked Read 9 Done Sent Done false
24169 7267 Acked Read 1 Done Sent Done false
24170 7268 Acked Write 4 Done Sent Done false
24171 7269 Acked Read 2 Done Sent Done false
24172 7270 Acked Read 4 Done Sent Done false
24173 7271 Acked Write 3 Done Sent Done false
24174 7272 Acked Write 8 Done Sent Done false
24175 7273 Acked Write 3 Done Sent Done false
24176 7274 Acked Write 3 Done Sent Done false
24177 7275 Acked Read 2 Done Sent Done false
24178 7276 Acked Read 10 Done Sent Done false
24179 7277 Acked Write 8 Done Sent Done false
24180 7278 Acked Read 6 Done Sent Done false
24181 7279 Acked Read 4 Done Sent Done false
24182 7280 Acked Read 2 Done Sent Done false
24183 7281 Acked Read 3 Done Sent Done false
24184 7282 Acked Write 5 Done Sent Done false
24185 7283 Acked Write 8 Done Sent Done false
24186 7284 Acked Read 8 Done Sent Done false
24187 7285 Acked Read 4 Done Sent Done false
24188 7286 Acked Write 10 Done Sent Done false
24189 7287 Acked Write 6 Done Sent Done false
24190 7288 Acked Write 2 Done Sent Done false
24191 7289 Acked Read 6 Done Sent Done false
24192 7290 Acked Write 9 Done Sent Done false
24193 7291 Acked Write 10 Done Sent Done false
24194 7292 Acked Write 2 Done Sent Done false
24195 7293 Acked Write 10 Done Sent Done false
24196 7294 Acked Write 8 Done Sent Done false
24197 7295 Acked Write 10 Done Sent Done false
24198 7296 Acked Flush 0 Done Sent Done false
24199 7297 Acked Read 2 Done Sent Done false
24200 7298 Acked Read 1 Done Sent Done false
24201 7299 Acked Read 5 Done Sent Done false
24202 7300 Acked Read 1 Done Sent Done false
24203 7301 Acked Read 10 Done Sent Done false
24204 7302 Acked Write 4 Done Sent Done false
24205 7303 Acked Write 7 Done Sent Done false
24206 7304 Acked Read 9 Done Sent Done false
24207 7305 Acked Write 9 Done Sent Done false
24208 7306 Acked Read 6 Done Sent Done false
24209 7307 Acked Write 9 Done Sent Done false
24210 7308 Acked Read 3 Done Sent Done false
24211 7309 Acked Write 5 Done Sent Done false
24212 7310 Acked Read 4 Done Sent Done false
24213 7311 Acked Write 2 Done Sent Done false
24214 7312 Acked Flush 0 Done Sent Done false
24215 7313 Acked Read 5 Done Sent Done false
24216 7314 Acked Read 1 Done Sent Done false
24217 7315 Acked Read 1 Done Sent Done false
24218 7316 Acked Read 6 Done Sent Done false
24219 7317 Acked Read 9 Done Sent Done false
24220 7318 Acked Write 9 Done Sent Done false
24221 7319 Acked Read 5 Done Sent Done false
24222 7320 Acked Write 4 Done Sent Done false
24223 7321 Acked Read 5 Done Sent Done false
24224 7322 Acked Write 10 Done Sent Done false
24225 7323 Acked Read 4 Done Sent Done false
24226 7324 Acked Read 4 Done Sent Done false
24227 7325 Acked Read 1 Done Sent Done false
24228 7326 Acked Read 1 Done Sent Done false
24229 7327 Acked Write 6 Done Sent Done false
24230 7328 Acked Read 7 Done Sent Done false
24231 7329 Acked Read 7 Done Sent Done false
24232 7330 Acked Read 9 Done Sent Done false
24233 7331 Acked Read 2 Done Sent Done false
24234 7332 Acked Read 1 Done Sent Done false
24235 7333 Acked Read 1 Done Sent Done false
24236 7334 Acked Read 3 Done Sent Done false
24237 7335 Acked Read 1 Done Sent Done false
24238 7336 Acked Write 7 Done Sent Done false
24239 7337 Acked Read 5 Done Sent Done false
24240 7338 Acked Flush 0 Done Sent Done false
24241 STATES DS:0 DS:1 DS:2 TOTAL
24242 Sent 0 301 0 301
24243 Done 301 0 301 602
24244 Skipped 0 0 0 0
24245 Error 0 0 0 0
24246 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
24247 Downstairs last five completed: 7037 7036 7035 7034 7033
24248 Upstairs last five completed: 7338 7337 7336 7335 7334
24249 CLIENT: Up:0 ds:301 act:2
242502025-02-25T20:51:54.563ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242512025-02-25T20:51:55.565ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242522025-02-25T20:51:55.761ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242532025-02-25T20:51:55.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242542025-02-25T20:51:55.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242552025-02-25T20:51:55.761ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242562025-02-25T20:51:55.761ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242572025-02-25T20:51:56.567ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
242582025-02-25T20:51:57.569ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
24259 [8820] got start action
24260 [8820] I am going to restart
24261 Starting downstairs at port 8820
24262 Make output file at "/tmp/dsc/downstairs-8820.txt"
24263 [8820][1] reports Starting
24264 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
24265 [8820][1] reports Running
24266 ----------------------------------------------------------------
24267 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24268 States: Active Offline Active
24269 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24270 7038 Acked Write 9 Done Sent Done false
24271 7039 Acked Write 5 Done Sent Done false
24272 7040 Acked Write 6 Done Sent Done false
24273 7041 Acked Read 7 Done Sent Done false
24274 7042 Acked Flush 0 Done Sent Done false
24275 7043 Acked Read 2 Done Sent Done false
24276 7044 Acked Flush 0 Done Sent Done false
24277 7045 Acked Read 3 Done Sent Done false
24278 7046 Acked Write 6 Done Sent Done false
24279 7047 Acked Read 1 Done Sent Done false
24280 7048 Acked Read 3 Done Sent Done false
24281 7049 Acked Write 1 Done Sent Done false
24282 7050 Acked Write 8 Done Sent Done false
24283 7051 Acked Read 9 Done Sent Done false
24284 7052 Acked Read 8 Done Sent Done false
24285 7053 Acked Flush 0 Done Sent Done false
24286 7054 Acked Write 10 Done Sent Done false
24287 7055 Acked Write 4 Done Sent Done false
24288 7056 Acked Read 3 Done Sent Done false
24289 7057 Acked Flush 0 Done Sent Done false
24290 7058 Acked Write 6 Done Sent Done false
24291 7059 Acked Write 3 Done Sent Done false
24292 7060 Acked Write 1 Done Sent Done false
24293 7061 Acked Flush 0 Done Sent Done false
24294 7062 Acked Read 7 Done Sent Done false
24295 7063 Acked Read 3 Done Sent Done false
24296 7064 Acked Read 8 Done Sent Done false
24297 7065 Acked Read 1 Done Sent Done false
24298 7066 Acked Read 10 Done Sent Done false
24299 7067 Acked Read 9 Done Sent Done false
24300 7068 Acked Write 5 Done Sent Done false
24301 7069 Acked Write 3 Done Sent Done false
24302 7070 Acked Write 1 Done Sent Done false
24303 7071 Acked Read 3 Done Sent Done false
24304 7072 Acked Flush 0 Done Sent Done false
24305 7073 Acked Write 1 Done Sent Done false
24306 7074 Acked Read 5 Done Sent Done false
24307 7075 Acked Read 10 Done Sent Done false
24308 7076 Acked Write 10 Done Sent Done false
24309 7077 Acked Write 8 Done Sent Done false
24310 7078 Acked Write 5 Done Sent Done false
24311 7079 Acked Read 3 Done Sent Done false
24312 7080 Acked Read 6 Done Sent Done false
24313 7081 Acked Read 4 Done Sent Done false
24314 7082 Acked Write 4 Done Sent Done false
24315 7083 Acked Read 1 Done Sent Done false
24316 7084 Acked Write 10 Done Sent Done false
24317 7085 Acked Read 9 Done Sent Done false
24318 7086 Acked Write 8 Done Sent Done false
24319 7087 Acked Read 2 Done Sent Done false
24320 7088 Acked Write 4 Done Sent Done false
24321 7089 Acked Flush 0 Done Sent Done false
24322 7090 Acked Read 6 Done Sent Done false
24323 7091 Acked Read 1 Done Sent Done false
24324 7092 Acked Write 10 Done Sent Done false
24325 7093 Acked Write 4 Done Sent Done false
24326 7094 Acked Read 5 Done Sent Done false
24327 7095 Acked Write 3 Done Sent Done false
24328 7096 Acked Write 5 Done Sent Done false
24329 7097 Acked Read 9 Done Sent Done false
24330 7098 Acked Read 1 Done Sent Done false
24331 7099 Acked Write 10 Done Sent Done false
24332 7100 Acked Write 6 Done Sent Done false
24333 7101 Acked Flush 0 Done Sent Done false
24334 7102 Acked Write 8 Done Sent Done false
24335 7103 Acked Write 8 Done Sent Done false
24336 7104 Acked Read 4 Done Sent Done false
24337 7105 Acked Write 6 Done Sent Done false
24338 7106 Acked Write 4 Done Sent Done false
24339 7107 Acked Read 5 Done Sent Done false
24340 7108 Acked Write 9 Done Sent Done false
24341 7109 Acked Write 2 Done Sent Done false
24342 7110 Acked Read 9 Done Sent Done false
24343 7111 Acked Read 9 Done Sent Done false
24344 7112 Acked Read 5 Done Sent Done false
24345 7113 Acked Read 6 Done Sent Done false
24346 7114 Acked Read 5 Done Sent Done false
24347 7115 Acked Write 6 Done Sent Done false
24348 7116 Acked Read 9 Done Sent Done false
24349 7117 Acked Read 5 Done Sent Done false
24350 7118 Acked Flush 0 Done Sent Done false
24351 7119 Acked Write 9 Done Sent Done false
24352 7120 Acked Read 1 Done Sent Done false
24353 7121 Acked Write 5 Done Sent Done false
24354 7122 Acked Read 8 Done Sent Done false
24355 7123 Acked Flush 0 Done Sent Done false
24356 7124 Acked Write 7 Done Sent Done false
24357 7125 Acked Write 3 Done Sent Done false
24358 7126 Acked Read 3 Done Sent Done false
24359 7127 Acked Flush 0 Done Sent Done false
24360 7128 Acked Read 4 Done Sent Done false
24361 7129 Acked Write 4 Done Sent Done false
24362 7130 Acked Read 5 Done Sent Done false
24363 7131 Acked Write 9 Done Sent Done false
24364 7132 Acked Flush 0 Done Sent Done false
24365 7133 Acked Write 3 Done Sent Done false
24366 7134 Acked Read 6 Done Sent Done false
24367 7135 Acked Flush 0 Done Sent Done false
24368 7136 Acked Write 5 Done Sent Done false
24369 7137 Acked Read 3 Done Sent Done false
24370 7138 Acked Write 10 Done Sent Done false
24371 7139 Acked Read 6 Done Sent Done false
24372 7140 Acked Write 8 Done Sent Done false
24373 7141 Acked Write 8 Done Sent Done false
24374 7142 Acked Write 9 Done Sent Done false
24375 7143 Acked Read 10 Done Sent Done false
24376 7144 Acked Write 3 Done Sent Done false
24377 7145 Acked Write 3 Done Sent Done false
24378 7146 Acked Read 4 Done Sent Done false
24379 7147 Acked Read 9 Done Sent Done false
24380 7148 Acked Flush 0 Done Sent Done false
24381 7149 Acked Write 9 Done Sent Done false
24382 7150 Acked Write 10 Done Sent Done false
24383 7151 Acked Read 5 Done Sent Done false
24384 7152 Acked Read 6 Done Sent Done false
24385 7153 Acked Write 3 Done Sent Done false
24386 7154 Acked Write 7 Done Sent Done false
24387 7155 Acked Read 7 Done Sent Done false
24388 7156 Acked Write 1 Done Sent Done false
24389 7157 Acked Read 2 Done Sent Done false
24390 7158 Acked Read 6 Done Sent Done false
24391 7159 Acked Write 3 Done Sent Done false
24392 7160 Acked Read 3 Done Sent Done false
24393 7161 Acked Write 3 Done Sent Done false
24394 7162 Acked Read 8 Done Sent Done false
24395 7163 Acked Read 9 Done Sent Done false
24396 7164 Acked Read 4 Done Sent Done false
24397 7165 Acked Write 10 Done Sent Done false
24398 7166 Acked Flush 0 Done Sent Done false
24399 7167 Acked Read 1 Done Sent Done false
24400 7168 Acked Read 6 Done Sent Done false
24401 7169 Acked Write 6 Done Sent Done false
24402 7170 Acked Read 4 Done Sent Done false
24403 7171 Acked Write 2 Done Sent Done false
24404 7172 Acked Read 8 Done Sent Done false
24405 7173 Acked Write 4 Done Sent Done false
24406 7174 Acked Write 1 Done Sent Done false
24407 7175 Acked Read 8 Done Sent Done false
24408 7176 Acked Read 4 Done Sent Done false
24409 7177 Acked Read 4 Done Sent Done false
24410 7178 Acked Read 9 Done Sent Done false
24411 7179 Acked Read 5 Done Sent Done false
24412 7180 Acked Write 9 Done Sent Done false
24413 7181 Acked Flush 0 Done Sent Done false
24414 7182 Acked Write 7 Done Sent Done false
24415 7183 Acked Read 3 Done Sent Done false
24416 7184 Acked Read 3 Done Sent Done false
24417 7185 Acked Flush 0 Done Sent Done false
24418 7186 Acked Write 4 Done Sent Done false
24419 7187 Acked Write 6 Done Sent Done false
24420 7188 Acked Read 4 Done Sent Done false
24421 7189 Acked Read 9 Done Sent Done false
24422 7190 Acked Read 6 Done Sent Done false
24423 7191 Acked Read 6 Done Sent Done false
24424 7192 Acked Write 10 Done Sent Done false
24425 7193 Acked Read 9 Done Sent Done false
24426 7194 Acked Read 7 Done Sent Done false
24427 7195 Acked Read 1 Done Sent Done false
24428 7196 Acked Write 9 Done Sent Done false
24429 7197 Acked Read 5 Done Sent Done false
24430 7198 Acked Write 6 Done Sent Done false
24431 7199 Acked Read 7 Done Sent Done false
24432 7200 Acked Read 10 Done Sent Done false
24433 7201 Acked Read 9 Done Sent Done false
24434 7202 Acked Read 4 Done Sent Done false
24435 7203 Acked Read 2 Done Sent Done false
24436 7204 Acked Read 7 Done Sent Done false
24437 7205 Acked Write 7 Done Sent Done false
24438 7206 Acked Read 8 Done Sent Done false
24439 7207 Acked Write 5 Done Sent Done false
24440 7208 Acked Read 8 Done Sent Done false
24441 7209 Acked Read 6 Done Sent Done false
24442 7210 Acked Write 1 Done Sent Done false
24443 7211 Acked Write 6 Done Sent Done false
24444 7212 Acked Write 5 Done Sent Done false
24445 7213 Acked Flush 0 Done Sent Done false
24446 7214 Acked Flush 0 Done Sent Done false
24447 7215 Acked Flush 0 Done Sent Done false
24448 7216 Acked Write 7 Done Sent Done false
24449 7217 Acked Read 7 Done Sent Done false
24450 7218 Acked Write 8 Done Sent Done false
24451 7219 Acked Read 9 Done Sent Done false
24452 7220 Acked Read 3 Done Sent Done false
24453 7221 Acked Read 4 Done Sent Done false
24454 7222 Acked Write 10 Done Sent Done false
24455 7223 Acked Write 8 Done Sent Done false
24456 7224 Acked Read 9 Done Sent Done false
24457 7225 Acked Read 3 Done Sent Done false
24458 7226 Acked Write 4 Done Sent Done false
24459 7227 Acked Write 9 Done Sent Done false
24460 7228 Acked Flush 0 Done Sent Done false
24461 7229 Acked Write 10 Done Sent Done false
24462 7230 Acked Read 9 Done Sent Done false
24463 7231 Acked Write 6 Done Sent Done false
24464 7232 Acked Read 2 Done Sent Done false
24465 7233 Acked Write 3 Done Sent Done false
24466 7234 Acked Read 1 Done Sent Done false
24467 7235 Acked Write 8 Done Sent Done false
24468 7236 Acked Write 7 Done Sent Done false
24469 7237 Acked Read 7 Done Sent Done false
24470 7238 Acked Read 7 Done Sent Done false
24471 7239 Acked Write 8 Done Sent Done false
24472 7240 Acked Write 1 Done Sent Done false
24473 7241 Acked Write 9 Done Sent Done false
24474 7242 Acked Write 8 Done Sent Done false
24475 7243 Acked Read 10 Done Sent Done false
24476 7244 Acked Write 3 Done Sent Done false
24477 7245 Acked Read 10 Done Sent Done false
24478 7246 Acked Read 6 Done Sent Done false
24479 7247 Acked Flush 0 Done Sent Done false
24480 7248 Acked Read 5 Done Sent Done false
24481 7249 Acked Write 3 Done Sent Done false
24482 7250 Acked Write 8 Done Sent Done false
24483 7251 Acked Write 3 Done Sent Done false
24484 7252 Acked Read 10 Done Sent Done false
24485 7253 Acked Read 3 Done Sent Done false
24486 7254 Acked Write 10 Done Sent Done false
24487 7255 Acked Write 4 Done Sent Done false
24488 7256 Acked Read 10 Done Sent Done false
24489 7257 Acked Read 2 Done Sent Done false
24490 7258 Acked Write 9 Done Sent Done false
24491 7259 Acked Write 6 Done Sent Done false
24492 7260 Acked Write 3 Done Sent Done false
24493 7261 Acked Write 10 Done Sent Done false
24494 7262 Acked Read 7 Done Sent Done false
24495 7263 Acked Write 1 Done Sent Done false
24496 7264 Acked Read 3 Done Sent Done false
24497 7265 Acked Read 9 Done Sent Done false
24498 7266 Acked Read 9 Done Sent Done false
24499 7267 Acked Read 1 Done Sent Done false
24500 7268 Acked Write 4 Done Sent Done false
24501 7269 Acked Read 2 Done Sent Done false
24502 7270 Acked Read 4 Done Sent Done false
24503 7271 Acked Write 3 Done Sent Done false
24504 7272 Acked Write 8 Done Sent Done false
24505 7273 Acked Write 3 Done Sent Done false
24506 7274 Acked Write 3 Done Sent Done false
24507 7275 Acked Read 2 Done Sent Done false
24508 7276 Acked Read 10 Done Sent Done false
24509 7277 Acked Write 8 Done Sent Done false
24510 7278 Acked Read 6 Done Sent Done false
24511 7279 Acked Read 4 Done Sent Done false
24512 7280 Acked Read 2 Done Sent Done false
24513 7281 Acked Read 3 Done Sent Done false
24514 7282 Acked Write 5 Done Sent Done false
24515 7283 Acked Write 8 Done Sent Done false
24516 7284 Acked Read 8 Done Sent Done false
24517 7285 Acked Read 4 Done Sent Done false
24518 7286 Acked Write 10 Done Sent Done false
24519 7287 Acked Write 6 Done Sent Done false
24520 7288 Acked Write 2 Done Sent Done false
24521 7289 Acked Read 6 Done Sent Done false
24522 7290 Acked Write 9 Done Sent Done false
24523 7291 Acked Write 10 Done Sent Done false
24524 7292 Acked Write 2 Done Sent Done false
24525 7293 Acked Write 10 Done Sent Done false
24526 7294 Acked Write 8 Done Sent Done false
24527 7295 Acked Write 10 Done Sent Done false
24528 7296 Acked Flush 0 Done Sent Done false
24529 7297 Acked Read 2 Done Sent Done false
24530 7298 Acked Read 1 Done Sent Done false
24531 7299 Acked Read 5 Done Sent Done false
24532 7300 Acked Read 1 Done Sent Done false
24533 7301 Acked Read 10 Done Sent Done false
24534 7302 Acked Write 4 Done Sent Done false
24535 7303 Acked Write 7 Done Sent Done false
24536 7304 Acked Read 9 Done Sent Done false
24537 7305 Acked Write 9 Done Sent Done false
24538 7306 Acked Read 6 Done Sent Done false
24539 7307 Acked Write 9 Done Sent Done false
24540 7308 Acked Read 3 Done Sent Done false
24541 7309 Acked Write 5 Done Sent Done false
24542 7310 Acked Read 4 Done Sent Done false
24543 7311 Acked Write 2 Done Sent Done false
24544 7312 Acked Flush 0 Done Sent Done false
24545 7313 Acked Read 5 Done Sent Done false
24546 7314 Acked Read 1 Done Sent Done false
24547 7315 Acked Read 1 Done Sent Done false
24548 7316 Acked Read 6 Done Sent Done false
24549 7317 Acked Read 9 Done Sent Done false
24550 7318 Acked Write 9 Done Sent Done false
24551 7319 Acked Read 5 Done Sent Done false
24552 7320 Acked Write 4 Done Sent Done false
24553 7321 Acked Read 5 Done Sent Done false
24554 7322 Acked Write 10 Done Sent Done false
24555 7323 Acked Read 4 Done Sent Done false
24556 7324 Acked Read 4 Done Sent Done false
24557 7325 Acked Read 1 Done Sent Done false
24558 7326 Acked Read 1 Done Sent Done false
24559 7327 Acked Write 6 Done Sent Done false
24560 7328 Acked Read 7 Done Sent Done false
24561 7329 Acked Read 7 Done Sent Done false
24562 7330 Acked Read 9 Done Sent Done false
24563 7331 Acked Read 2 Done Sent Done false
24564 7332 Acked Read 1 Done Sent Done false
24565 7333 Acked Read 1 Done Sent Done false
24566 7334 Acked Read 3 Done Sent Done false
24567 7335 Acked Read 1 Done Sent Done false
24568 7336 Acked Write 7 Done Sent Done false
24569 7337 Acked Read 5 Done Sent Done false
24570 7338 Acked Flush 0 Done Sent Done false
24571 STATES DS:0 DS:1 DS:2 TOTAL
24572 Sent 0 301 0 301
24573 Done 301 0 301 602
24574 Skipped 0 0 0 0
24575 Error 0 0 0 0
24576 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
24577 Downstairs last five completed: 7037 7036 7035 7034 7033
24578 Upstairs last five completed: 7338 7337 7336 7335 7334
24579 CLIENT: Up:0 ds:301 act:2
245802025-02-25T20:51:59.001ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
245812025-02-25T20:51:59.573ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
245822025-02-25T20:52:00.574ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
245832025-02-25T20:52:01.576ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
24584 ----------------------------------------------------------------
24585 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24586 States: Active Offline Active
24587 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24588 7038 Acked Write 9 Done Sent Done false
24589 7039 Acked Write 5 Done Sent Done false
24590 7040 Acked Write 6 Done Sent Done false
24591 7041 Acked Read 7 Done Sent Done false
24592 7042 Acked Flush 0 Done Sent Done false
24593 7043 Acked Read 2 Done Sent Done false
24594 7044 Acked Flush 0 Done Sent Done false
24595 7045 Acked Read 3 Done Sent Done false
24596 7046 Acked Write 6 Done Sent Done false
24597 7047 Acked Read 1 Done Sent Done false
24598 7048 Acked Read 3 Done Sent Done false
24599 7049 Acked Write 1 Done Sent Done false
24600 7050 Acked Write 8 Done Sent Done false
24601 7051 Acked Read 9 Done Sent Done false
24602 7052 Acked Read 8 Done Sent Done false
24603 7053 Acked Flush 0 Done Sent Done false
24604 7054 Acked Write 10 Done Sent Done false
24605 7055 Acked Write 4 Done Sent Done false
24606 7056 Acked Read 3 Done Sent Done false
24607 7057 Acked Flush 0 Done Sent Done false
24608 7058 Acked Write 6 Done Sent Done false
24609 7059 Acked Write 3 Done Sent Done false
24610 7060 Acked Write 1 Done Sent Done false
24611 7061 Acked Flush 0 Done Sent Done false
24612 7062 Acked Read 7 Done Sent Done false
24613 7063 Acked Read 3 Done Sent Done false
24614 7064 Acked Read 8 Done Sent Done false
24615 7065 Acked Read 1 Done Sent Done false
24616 7066 Acked Read 10 Done Sent Done false
24617 7067 Acked Read 9 Done Sent Done false
24618 7068 Acked Write 5 Done Sent Done false
24619 7069 Acked Write 3 Done Sent Done false
24620 7070 Acked Write 1 Done Sent Done false
24621 7071 Acked Read 3 Done Sent Done false
24622 7072 Acked Flush 0 Done Sent Done false
24623 7073 Acked Write 1 Done Sent Done false
24624 7074 Acked Read 5 Done Sent Done false
24625 7075 Acked Read 10 Done Sent Done false
24626 7076 Acked Write 10 Done Sent Done false
24627 7077 Acked Write 8 Done Sent Done false
24628 7078 Acked Write 5 Done Sent Done false
24629 7079 Acked Read 3 Done Sent Done false
24630 7080 Acked Read 6 Done Sent Done false
24631 7081 Acked Read 4 Done Sent Done false
24632 7082 Acked Write 4 Done Sent Done false
24633 7083 Acked Read 1 Done Sent Done false
24634 7084 Acked Write 10 Done Sent Done false
24635 7085 Acked Read 9 Done Sent Done false
24636 7086 Acked Write 8 Done Sent Done false
24637 7087 Acked Read 2 Done Sent Done false
24638 7088 Acked Write 4 Done Sent Done false
24639 7089 Acked Flush 0 Done Sent Done false
24640 7090 Acked Read 6 Done Sent Done false
24641 7091 Acked Read 1 Done Sent Done false
24642 7092 Acked Write 10 Done Sent Done false
24643 7093 Acked Write 4 Done Sent Done false
24644 7094 Acked Read 5 Done Sent Done false
24645 7095 Acked Write 3 Done Sent Done false
24646 7096 Acked Write 5 Done Sent Done false
24647 7097 Acked Read 9 Done Sent Done false
24648 7098 Acked Read 1 Done Sent Done false
24649 7099 Acked Write 10 Done Sent Done false
24650 7100 Acked Write 6 Done Sent Done false
24651 7101 Acked Flush 0 Done Sent Done false
24652 7102 Acked Write 8 Done Sent Done false
24653 7103 Acked Write 8 Done Sent Done false
24654 7104 Acked Read 4 Done Sent Done false
24655 7105 Acked Write 6 Done Sent Done false
24656 7106 Acked Write 4 Done Sent Done false
24657 7107 Acked Read 5 Done Sent Done false
24658 7108 Acked Write 9 Done Sent Done false
24659 7109 Acked Write 2 Done Sent Done false
24660 7110 Acked Read 9 Done Sent Done false
24661 7111 Acked Read 9 Done Sent Done false
24662 7112 Acked Read 5 Done Sent Done false
24663 7113 Acked Read 6 Done Sent Done false
24664 7114 Acked Read 5 Done Sent Done false
24665 7115 Acked Write 6 Done Sent Done false
24666 7116 Acked Read 9 Done Sent Done false
24667 7117 Acked Read 5 Done Sent Done false
24668 7118 Acked Flush 0 Done Sent Done false
24669 7119 Acked Write 9 Done Sent Done false
24670 7120 Acked Read 1 Done Sent Done false
24671 7121 Acked Write 5 Done Sent Done false
24672 7122 Acked Read 8 Done Sent Done false
24673 7123 Acked Flush 0 Done Sent Done false
24674 7124 Acked Write 7 Done Sent Done false
24675 7125 Acked Write 3 Done Sent Done false
24676 7126 Acked Read 3 Done Sent Done false
24677 7127 Acked Flush 0 Done Sent Done false
24678 7128 Acked Read 4 Done Sent Done false
24679 7129 Acked Write 4 Done Sent Done false
24680 7130 Acked Read 5 Done Sent Done false
24681 7131 Acked Write 9 Done Sent Done false
24682 7132 Acked Flush 0 Done Sent Done false
24683 7133 Acked Write 3 Done Sent Done false
24684 7134 Acked Read 6 Done Sent Done false
24685 7135 Acked Flush 0 Done Sent Done false
24686 7136 Acked Write 5 Done Sent Done false
24687 7137 Acked Read 3 Done Sent Done false
24688 7138 Acked Write 10 Done Sent Done false
24689 7139 Acked Read 6 Done Sent Done false
24690 7140 Acked Write 8 Done Sent Done false
24691 7141 Acked Write 8 Done Sent Done false
24692 7142 Acked Write 9 Done Sent Done false
24693 7143 Acked Read 10 Done Sent Done false
24694 7144 Acked Write 3 Done Sent Done false
24695 7145 Acked Write 3 Done Sent Done false
24696 7146 Acked Read 4 Done Sent Done false
24697 7147 Acked Read 9 Done Sent Done false
24698 7148 Acked Flush 0 Done Sent Done false
24699 7149 Acked Write 9 Done Sent Done false
24700 7150 Acked Write 10 Done Sent Done false
24701 7151 Acked Read 5 Done Sent Done false
24702 7152 Acked Read 6 Done Sent Done false
24703 7153 Acked Write 3 Done Sent Done false
24704 7154 Acked Write 7 Done Sent Done false
24705 7155 Acked Read 7 Done Sent Done false
24706 7156 Acked Write 1 Done Sent Done false
24707 7157 Acked Read 2 Done Sent Done false
24708 7158 Acked Read 6 Done Sent Done false
24709 7159 Acked Write 3 Done Sent Done false
24710 7160 Acked Read 3 Done Sent Done false
24711 7161 Acked Write 3 Done Sent Done false
24712 7162 Acked Read 8 Done Sent Done false
24713 7163 Acked Read 9 Done Sent Done false
24714 7164 Acked Read 4 Done Sent Done false
24715 7165 Acked Write 10 Done Sent Done false
24716 7166 Acked Flush 0 Done Sent Done false
24717 7167 Acked Read 1 Done Sent Done false
24718 7168 Acked Read 6 Done Sent Done false
24719 7169 Acked Write 6 Done Sent Done false
24720 7170 Acked Read 4 Done Sent Done false
24721 7171 Acked Write 2 Done Sent Done false
24722 7172 Acked Read 8 Done Sent Done false
24723 7173 Acked Write 4 Done Sent Done false
24724 7174 Acked Write 1 Done Sent Done false
24725 7175 Acked Read 8 Done Sent Done false
24726 7176 Acked Read 4 Done Sent Done false
24727 7177 Acked Read 4 Done Sent Done false
24728 7178 Acked Read 9 Done Sent Done false
24729 7179 Acked Read 5 Done Sent Done false
24730 7180 Acked Write 9 Done Sent Done false
24731 7181 Acked Flush 0 Done Sent Done false
24732 7182 Acked Write 7 Done Sent Done false
24733 7183 Acked Read 3 Done Sent Done false
24734 7184 Acked Read 3 Done Sent Done false
24735 7185 Acked Flush 0 Done Sent Done false
24736 7186 Acked Write 4 Done Sent Done false
24737 7187 Acked Write 6 Done Sent Done false
24738 7188 Acked Read 4 Done Sent Done false
24739 7189 Acked Read 9 Done Sent Done false
24740 7190 Acked Read 6 Done Sent Done false
24741 7191 Acked Read 6 Done Sent Done false
24742 7192 Acked Write 10 Done Sent Done false
24743 7193 Acked Read 9 Done Sent Done false
24744 7194 Acked Read 7 Done Sent Done false
24745 7195 Acked Read 1 Done Sent Done false
24746 7196 Acked Write 9 Done Sent Done false
24747 7197 Acked Read 5 Done Sent Done false
24748 7198 Acked Write 6 Done Sent Done false
24749 7199 Acked Read 7 Done Sent Done false
24750 7200 Acked Read 10 Done Sent Done false
24751 7201 Acked Read 9 Done Sent Done false
24752 7202 Acked Read 4 Done Sent Done false
24753 7203 Acked Read 2 Done Sent Done false
24754 7204 Acked Read 7 Done Sent Done false
24755 7205 Acked Write 7 Done Sent Done false
24756 7206 Acked Read 8 Done Sent Done false
24757 7207 Acked Write 5 Done Sent Done false
24758 7208 Acked Read 8 Done Sent Done false
24759 7209 Acked Read 6 Done Sent Done false
24760 7210 Acked Write 1 Done Sent Done false
24761 7211 Acked Write 6 Done Sent Done false
24762 7212 Acked Write 5 Done Sent Done false
24763 7213 Acked Flush 0 Done Sent Done false
24764 7214 Acked Flush 0 Done Sent Done false
24765 7215 Acked Flush 0 Done Sent Done false
24766 7216 Acked Write 7 Done Sent Done false
24767 7217 Acked Read 7 Done Sent Done false
24768 7218 Acked Write 8 Done Sent Done false
24769 7219 Acked Read 9 Done Sent Done false
24770 7220 Acked Read 3 Done Sent Done false
24771 7221 Acked Read 4 Done Sent Done false
24772 7222 Acked Write 10 Done Sent Done false
24773 7223 Acked Write 8 Done Sent Done false
24774 7224 Acked Read 9 Done Sent Done false
24775 7225 Acked Read 3 Done Sent Done false
24776 7226 Acked Write 4 Done Sent Done false
24777 7227 Acked Write 9 Done Sent Done false
24778 7228 Acked Flush 0 Done Sent Done false
24779 7229 Acked Write 10 Done Sent Done false
24780 7230 Acked Read 9 Done Sent Done false
24781 7231 Acked Write 6 Done Sent Done false
24782 7232 Acked Read 2 Done Sent Done false
24783 7233 Acked Write 3 Done Sent Done false
24784 7234 Acked Read 1 Done Sent Done false
24785 7235 Acked Write 8 Done Sent Done false
24786 7236 Acked Write 7 Done Sent Done false
24787 7237 Acked Read 7 Done Sent Done false
24788 7238 Acked Read 7 Done Sent Done false
24789 7239 Acked Write 8 Done Sent Done false
24790 7240 Acked Write 1 Done Sent Done false
24791 7241 Acked Write 9 Done Sent Done false
24792 7242 Acked Write 8 Done Sent Done false
24793 7243 Acked Read 10 Done Sent Done false
24794 7244 Acked Write 3 Done Sent Done false
24795 7245 Acked Read 10 Done Sent Done false
24796 7246 Acked Read 6 Done Sent Done false
24797 7247 Acked Flush 0 Done Sent Done false
24798 7248 Acked Read 5 Done Sent Done false
24799 7249 Acked Write 3 Done Sent Done false
24800 7250 Acked Write 8 Done Sent Done false
24801 7251 Acked Write 3 Done Sent Done false
24802 7252 Acked Read 10 Done Sent Done false
24803 7253 Acked Read 3 Done Sent Done false
24804 7254 Acked Write 10 Done Sent Done false
24805 7255 Acked Write 4 Done Sent Done false
24806 7256 Acked Read 10 Done Sent Done false
24807 7257 Acked Read 2 Done Sent Done false
24808 7258 Acked Write 9 Done Sent Done false
24809 7259 Acked Write 6 Done Sent Done false
24810 7260 Acked Write 3 Done Sent Done false
24811 7261 Acked Write 10 Done Sent Done false
24812 7262 Acked Read 7 Done Sent Done false
24813 7263 Acked Write 1 Done Sent Done false
24814 7264 Acked Read 3 Done Sent Done false
24815 7265 Acked Read 9 Done Sent Done false
24816 7266 Acked Read 9 Done Sent Done false
24817 7267 Acked Read 1 Done Sent Done false
24818 7268 Acked Write 4 Done Sent Done false
24819 7269 Acked Read 2 Done Sent Done false
24820 7270 Acked Read 4 Done Sent Done false
24821 7271 Acked Write 3 Done Sent Done false
24822 7272 Acked Write 8 Done Sent Done false
24823 7273 Acked Write 3 Done Sent Done false
24824 7274 Acked Write 3 Done Sent Done false
24825 7275 Acked Read 2 Done Sent Done false
24826 7276 Acked Read 10 Done Sent Done false
24827 7277 Acked Write 8 Done Sent Done false
24828 7278 Acked Read 6 Done Sent Done false
24829 7279 Acked Read 4 Done Sent Done false
24830 7280 Acked Read 2 Done Sent Done false
24831 7281 Acked Read 3 Done Sent Done false
24832 7282 Acked Write 5 Done Sent Done false
24833 7283 Acked Write 8 Done Sent Done false
24834 7284 Acked Read 8 Done Sent Done false
24835 7285 Acked Read 4 Done Sent Done false
24836 7286 Acked Write 10 Done Sent Done false
24837 7287 Acked Write 6 Done Sent Done false
24838 7288 Acked Write 2 Done Sent Done false
24839 7289 Acked Read 6 Done Sent Done false
24840 7290 Acked Write 9 Done Sent Done false
24841 7291 Acked Write 10 Done Sent Done false
24842 7292 Acked Write 2 Done Sent Done false
24843 7293 Acked Write 10 Done Sent Done false
24844 7294 Acked Write 8 Done Sent Done false
24845 7295 Acked Write 10 Done Sent Done false
24846 7296 Acked Flush 0 Done Sent Done false
24847 7297 Acked Read 2 Done Sent Done false
24848 7298 Acked Read 1 Done Sent Done false
24849 7299 Acked Read 5 Done Sent Done false
24850 7300 Acked Read 1 Done Sent Done false
24851 7301 Acked Read 10 Done Sent Done false
24852 7302 Acked Write 4 Done Sent Done false
24853 7303 Acked Write 7 Done Sent Done false
24854 7304 Acked Read 9 Done Sent Done false
24855 7305 Acked Write 9 Done Sent Done false
24856 7306 Acked Read 6 Done Sent Done false
24857 7307 Acked Write 9 Done Sent Done false
24858 7308 Acked Read 3 Done Sent Done false
24859 7309 Acked Write 5 Done Sent Done false
24860 7310 Acked Read 4 Done Sent Done false
24861 7311 Acked Write 2 Done Sent Done false
24862 7312 Acked Flush 0 Done Sent Done false
24863 7313 Acked Read 5 Done Sent Done false
24864 7314 Acked Read 1 Done Sent Done false
24865 7315 Acked Read 1 Done Sent Done false
24866 7316 Acked Read 6 Done Sent Done false
24867 7317 Acked Read 9 Done Sent Done false
24868 7318 Acked Write 9 Done Sent Done false
24869 7319 Acked Read 5 Done Sent Done false
24870 7320 Acked Write 4 Done Sent Done false
24871 7321 Acked Read 5 Done Sent Done false
24872 7322 Acked Write 10 Done Sent Done false
24873 7323 Acked Read 4 Done Sent Done false
24874 7324 Acked Read 4 Done Sent Done false
24875 7325 Acked Read 1 Done Sent Done false
24876 7326 Acked Read 1 Done Sent Done false
24877 7327 Acked Write 6 Done Sent Done false
24878 7328 Acked Read 7 Done Sent Done false
24879 7329 Acked Read 7 Done Sent Done false
24880 7330 Acked Read 9 Done Sent Done false
24881 7331 Acked Read 2 Done Sent Done false
24882 7332 Acked Read 1 Done Sent Done false
24883 7333 Acked Read 1 Done Sent Done false
24884 7334 Acked Read 3 Done Sent Done false
24885 7335 Acked Read 1 Done Sent Done false
24886 7336 Acked Write 7 Done Sent Done false
24887 7337 Acked Read 5 Done Sent Done false
24888 7338 Acked Flush 0 Done Sent Done false
24889 STATES DS:0 DS:1 DS:2 TOTAL
24890 Sent 0 301 0 301
24891 Done 301 0 301 602
24892 Skipped 0 0 0 0
24893 Error 0 0 0 0
24894 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
24895 Downstairs last five completed: 7037 7036 7035 7034 7033
24896 Upstairs last five completed: 7338 7337 7336 7335 7334
24897 CLIENT: Up:0 ds:301 act:2
248982025-02-25T20:52:02.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
248992025-02-25T20:52:04.003ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249002025-02-25T20:52:04.579ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249012025-02-25T20:52:05.581ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249022025-02-25T20:52:05.762ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249032025-02-25T20:52:05.762ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51487) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249042025-02-25T20:52:05.763ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249052025-02-25T20:52:05.764ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249062025-02-25T20:52:05.764ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249072025-02-25T20:52:05.764ZINFOcrucible: send last flush ID to this DS: Some(JobId(7037)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249082025-02-25T20:52:05.764ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249092025-02-25T20:52:05.764ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7037)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249102025-02-25T20:52:05.764ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7037)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
24911 ----------------------------------------------------------------
24912 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24913 Downstairs last five completed: 7338 7337 7336 7335 7334
24914 Upstairs last five completed: 7338 7337 7336 7335 7334
24915 CLIENT: Up:0 ds:0 act:3
24916 Replay: All jobs finished
24917 Main task has work to do, go find it
24918 got dsc Stop(1)
24919 stop 1
24920 Feb 25 20:52:06.021 INFO[8820] Got stop action so:false kr:false
24921 request completed, latency_us: 341, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b43dece5-0726-44ff-8662-377f9c374857, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
24922 Feb 25 20:52:06.022 INFO request completed, latency_us: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:52:06.02202337Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
24923 145, response_code: 200, uri: /state/cid/1, method: GET, req_id: e4e4859f-1e1d-41ea-967d-6e19959bab4a, remote_addr: 127{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:52:06.022096345Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
24924 .0.0.1:43930, local_addr: 127.0.0.1:9998
249252025-02-25T20:52:06.022ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
249262025-02-25T20:52:06.022ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
24927 [8820] Exited with: ExitStatus(unix_wait_status(9))
24928 [8820][1] reports Exit
24929 Feb 25 20:52:10.024 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 182bceb7-9fe7-4543-ac90-1913da952d68, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
24930 001/300 Write block 4752 len 24576 data: 7 8 7 8 8 5
24931 002/300 Write block 4202 len 40960 data: 2 2 2 2 2 2 2 2 2 3
24932 003/300 Write block 3388 len 4096 data: 3
24933 004/300 Write block 3476 len 12288 data: 4 4 4
24934 005/300 Read block 3090 len 36864
24935 006/300 Read block 3713 len 40960
24936 007/300 Read block 1369 len 12288
24937 008/300 Flush
24938 009/300 Read block 594 len 8192
24939 010/300 Read block 1321 len 24576
24940 011/300 Write block 3901 len 36864 data: 5 4 5 6 6 5 5 4 3
24941 012/300 Read block 3377 len 16384
24942 013/300 Read block 685 len 32768
24943 014/300 Read block 578 len 4096
24944 015/300 Read block 3370 len 16384
24945 016/300 Write block 4624 len 32768 data: 2 3 3 3 3 4 5 5
24946 017/300 Read block 2001 len 12288
24947 018/300 Write block 896 len 40960 data: 3 4 4 4 4 4 4 4 4 4
24948 019/300 Read block 4544 len 32768
24949 020/300 Write block 1059 len 28672 data: 7 7 7 6 7 5 5
24950 021/300 Write block 4017 len 8192 data: 3 3
24951 022/300 Write block 873 len 32768 data: 3 3 4 4 4 4 4 3
24952 023/300 Read block 3016 len 24576
24953 024/300 Read block 278 len 40960
24954 025/300 Read block 3781 len 4096
24955 026/300 Write block 2028 len 4096 data: 5
24956 027/300 Read block 4460 len 4096
24957 028/300 Read block 2261 len 36864
24958 029/300 Write block 4294 len 4096 data: 6
24959 030/300 Write block 917 len 28672 data: 5 6 7 7 7 6 6
24960 031/300 Read block 2873 len 36864
24961 032/300 Write block 3784 len 28672 data: 4 5 6 5 4 6 5
24962 033/300 Read block 1145 len 40960
24963 034/300 Write block 4906 len 12288 data: 4 3 3
24964 035/300 Read block 2076 len 24576
24965 036/300 Read block 4381 len 4096
24966 037/300 Read block 1591 len 28672
24967 038/300 Write block 4776 len 24576 data: 4 5 2 2 2 4
24968 039/300 Write block 926 len 8192 data: 5 4
24969 040/300 Read block 2537 len 20480
24970 041/300 Write block 3901 len 36864 data: 6 5 6 7 7 6 6 5 4
24971 042/300 Read block 3208 len 4096
24972 043/300 Read block 1069 len 8192
24973 044/300 Write block 3787 len 16384 data: 6 5 7 6
24974 045/300 Read block 3740 len 36864
24975 046/300 Read block 2257 len 36864
24976 047/300 Read block 2788 len 36864
24977 048/300 Read block 188 len 20480
24978 049/300 Read block 1396 len 36864
24979 050/300 Write block 2148 len 28672 data: 5 4 5 6 7 6 6
24980 051/300 Read block 1748 len 4096
24981 052/300 Read block 3866 len 32768
24982 053/300 Read block 1222 len 40960
24983 054/300 Read block 3964 len 28672
24984 055/300 Read block 875 len 16384
24985 056/300 Write block 2047 len 16384 data: 3 4 4 4
24986 057/300 Read block 3008 len 40960
24987 058/300 Read block 3247 len 20480
24988 059/300 Write block 4013 len 12288 data: 4 4 5
24989 060/300 Read block 683 len 28672
24990 061/300 Write block 3483 len 8192 data: 3 4
24991 062/300 Flush
24992 063/300 Read block 4866 len 28672
24993 064/300 Write block 2101 len 20480 data: 5 6 5 5 5
24994 065/300 Flush
24995 066/300 Read block 3364 len 24576
24996 067/300 Write block 2664 len 24576 data: 4 5 4 4 5 3
24997 068/300 Read block 1232 len 36864
24998 069/300 Read block 4563 len 32768
24999 070/300 Read block 695 len 40960
25000 071/300 Read block 2083 len 40960
25001 072/300 Write block 4174 len 4096 data: 7
25002 073/300 Write block 1387 len 8192 data: 6 7
25003 074/300 Read block 469 len 40960
25004 075/300 Read block 4666 len 12288
25005 076/300 Write block 3807 len 32768 data: 3 3 3 4 4 4 4 4
25006 077/300 Read block 286 len 16384
25007 078/300 Read block 1262 len 36864
25008 079/300 Write block 1544 len 4096 data: 2
25009 080/300 Read block 3091 len 4096
25010 081/300 Read block 2379 len 28672
25011 082/300 Read block 1941 len 20480
25012 083/300 Read block 1498 len 28672
25013 084/300 Read block 2153 len 28672
25014 085/300 Read block 2236 len 12288
25015 086/300 Write block 2589 len 16384 data: 2 5 4 4
25016 087/300 Read block 419 len 24576
25017 088/300 Read block 547 len 24576
25018 089/300 Read block 3307 len 12288
25019 090/300 Write block 1157 len 16384 data: 4 4 4 3
25020 091/300 Read block 2359 len 28672
25021 092/300 Read block 1103 len 8192
25022 093/300 Read block 2621 len 4096
25023 094/300 Read block 1302 len 20480
25024 095/300 Read block 695 len 20480
25025 096/300 Read block 1683 len 16384
25026 097/300 Read block 190 len 36864
25027 098/300 Write block 4792 len 24576 data: 4 4 7 8 8 8
25028 099/300 Read block 3549 len 32768
25029 100/300 Read block 4201 len 12288
25030 101/300 Read block 2327 len 12288
25031 102/300 Read block 2934 len 40960
25032 103/300 Read block 155 len 16384
25033 104/300 Write block 4173 len 24576 data: 8 8 6 5 4 4
25034 105/300 Write block 4345 len 12288 data: 3 2 3
25035 106/300 Read block 3333 len 32768
25036 107/300 Write block 3484 len 8192 data: 5 5
25037 108/300 Write block 4208 len 20480 data: 3 3 3 4 3
25038 109/300 Write block 3036 len 40960 data: 3 3 2 2 2 3 4 4 4 3
25039 110/300 Flush
25040 111/300 Read block 2768 len 20480
25041 112/300 Write block 143 len 40960 data: 3 3 3 4 4 4 5 4 4 4
25042 113/300 Write block 2579 len 12288 data: 5 5 5
25043 114/300 Read block 1863 len 12288
25044 115/300 Read block 912 len 12288
25045 116/300 Write block 2335 len 24576 data: 6 6 7 9 9 9
25046 117/300 Write block 1134 len 36864 data: 4 4 3 3 2 4 4 4 3
25047 118/300 Read block 4243 len 4096
25048 119/300 Read block 1404 len 8192
25049 120/300 Read block 1763 len 28672
25050 121/300 Read block 2358 len 24576
25051 122/300 Read block 1121 len 36864
25052 123/300 Read block 1709 len 20480
25053 124/300 Write block 239 len 32768 data: 5 4 4 3 3 4 4 3
25054 125/300 Read block 4725 len 24576
25055 126/300 Write block 2480 len 12288 data: 3 4 4
25056 127/300 Read block 4357 len 4096
25057 128/300 Write block 2036 len 32768 data: 6 5 4 4 3 3 3 3
25058 129/300 Flush
25059 130/300 Read block 2054 len 16384
25060 131/300 Read block 1949 len 32768
25061 132/300 Write block 4 len 36864 data: 4 4 5 5 5 6 6 6 4
25062 133/300 Write block 3025 len 4096 data: 4
25063 134/300 Read block 4684 len 40960
25064 135/300 Read block 4079 len 20480
25065 136/300 Read block 1736 len 36864
25066 137/300 Write block 435 len 20480 data: 6 5 8 7 5
25067 138/300 Write block 4389 len 4096 data: 5
25068 139/300 Write block 996 len 32768 data: 8 5 5 6 5 5 5 5
25069 140/300 Write block 2574 len 28672 data: 5 3 3 4 3 6 6
25070 141/300 Read block 1314 len 28672
25071 142/300 Write block 318 len 32768 data: 3 3 4 6 7 5 5 4
25072 143/300 Read block 1515 len 24576
25073 144/300 Write block 343 len 28672 data: 5 5 6 6 6 6 5
25074 145/300 Read block 1537 len 12288
25075 146/300 Flush
25076 147/300 Write block 2792 len 16384 data: 4 4 4 4
25077 148/300 Read block 3794 len 36864
25078 149/300 Read block 1571 len 12288
25079 150/300 Write block 1985 len 28672 data: 6 4 5 6 6 5 6
25080 151/300 Read block 4090 len 16384
25081 152/300 Write block 602 len 28672 data: 4 4 4 5 4 4 3
25082 153/300 Read block 614 len 36864
25083 154/300 Read block 1747 len 16384
25084 155/300 Write block 73 len 36864 data: 5 5 5 5 5 5 5 6 6
25085 156/300 Read block 89 len 8192
25086 157/300 Write block 3961 len 12288 data: 5 5 5
25087 158/300 Read block 1255 len 12288
25088 159/300 Read block 3653 len 4096
25089 160/300 Write block 2324 len 20480 data: 6 6 5 3 3
25090 161/300 Write block 1974 len 28672 data: 5 4 3 3 3 4 5
25091 162/300 Read block 2331 len 32768
25092 163/300 Flush
25093 164/300 Read block 945 len 40960
25094 165/300 Flush
25095 166/300 Read block 3136 len 20480
25096 167/300 Write block 4607 len 4096 data: 3
25097 168/300 Write block 4264 len 8192 data: 4 4
25098 169/300 Read block 1637 len 16384
25099 170/300 Read block 4944 len 24576
25100 171/300 Read block 4773 len 8192
25101 172/300 Write block 1044 len 40960 data: 6 5 5 4 4 4 6 6 8 7
25102 173/300 Read block 4390 len 20480
25103 174/300 Flush
25104 175/300 Write block 2770 len 4096 data: 7
25105 176/300 Write block 1632 len 4096 data: 6
25106 177/300 Write block 735 len 4096 data: 3
25107 178/300 Flush
25108 179/300 Write block 861 len 28672 data: 6 6 7 5 5 4 4
25109 180/300 Write block 4166 len 36864 data: 11 9 7 8 8 6 6 9 9
25110 181/300 Read block 3243 len 28672
25111 182/300 Write block 1028 len 16384 data: 6 5 5 3
25112 183/300 Flush
25113 184/300 Write block 4716 len 28672 data: 6 5 4 3 4 5 5
25114 185/300 Write block 1545 len 36864 data: 2 4 3 4 4 4 4 3 4
25115 186/300 Write block 3530 len 16384 data: 4 5 5 4
25116 187/300 Read block 168 len 4096
25117 188/300 Read block 4078 len 24576
25118 189/300 Read block 4306 len 24576
25119 190/300 Read block 1449 len 32768
25120 191/300 Read block 3235 len 32768
25121 192/300 Read block 897 len 40960
25122 193/300 Write block 2874 len 20480 data: 3 2 3 3 3
25123 194/300 Read block 3109 len 20480
25124 195/300 Write block 1238 len 12288 data: 6 4 3
25125 196/300 Write block 964 len 36864 data: 4 3 5 5 5 3 3 3 4
25126 197/300 Write block 1360 len 24576 data: 5 6 5 7 7 6
25127 198/300 Read block 3559 len 24576
25128 199/300 Flush
25129 200/300 Write block 4361 len 24576 data: 5 5 6 5 4 4
25130 201/300 Flush
25131 202/300 Write block 3305 len 16384 data: 8 6 5 6
25132 203/300 Read block 3887 len 36864
25133 204/300 Write block 1272 len 4096 data: 3
25134 205/300 Flush
25135 206/300 Flush
25136 207/300 Read block 3775 len 4096
25137 208/300 Write block 4442 len 28672 data: 5 4 3 3 3 4 3
25138 209/300 Read block 1515 len 12288
25139 210/300 Read block 3750 len 28672
25140 211/300 Read block 633 len 36864
25141 212/300 Write block 3374 len 24576 data: 5 3 4 5 5 5
25142 213/300 Write block 414 len 40960 data: 7 6 6 5 4 4 4 7 7 7
25143 214/300 Read block 724 len 4096
25144 215/300 Write block 448 len 32768 data: 5 4 4 3 3 3 3 3
25145 216/300 Write block 3137 len 20480 data: 6 4 5 6 5
25146 217/300 Write block 2591 len 24576 data: 5 5 4 5 5 6
25147 218/300 Read block 2307 len 40960
25148 219/300 Write block 1208 len 24576 data: 4 4 5 5 5 4
25149 220/300 Write block 4774 len 16384 data: 5 5 5 6
25150 221/300 Read block 192 len 32768
25151 222/300 Write block 3781 len 4096 data: 5
25152 223/300 Read block 390 len 4096
25153 224/300 Read block 1149 len 12288
25154 225/300 Read block 3056 len 16384
25155 226/300 Read block 4212 len 12288
25156 227/300 Write block 1427 len 16384 data: 2 4 5 5
25157 228/300 Read block 2484 len 24576
25158 229/300 Read block 2298 len 40960
25159 230/300 Read block 1630 len 8192
25160 231/300 Read block 4634 len 20480
25161 232/300 Read block 218 len 28672
25162 233/300 Write block 2814 len 8192 data: 3 3
25163 234/300 Read block 1325 len 16384
25164 235/300 Write block 4752 len 4096 data: 8
25165 236/300 Write block 4603 len 40960 data: 6 5 5 5 4 3 4 4 5 6
25166 237/300 Write block 4148 len 32768 data: 3 5 5 5 5 4 5 4
25167 238/300 Read block 1903 len 12288
25168 239/300 Read block 4393 len 4096
25169 240/300 Read block 2481 len 28672
25170 241/300 Read block 3261 len 32768
25171 242/300 Read block 4004 len 8192
25172 243/300 Read block 374 len 32768
25173 244/300 Read block 2701 len 20480
25174 245/300 Read block 103 len 20480
25175 246/300 Read block 4533 len 16384
25176 247/300 Read block 2877 len 20480
25177 248/300 Write block 4249 len 36864 data: 3 3 3 3 4 5 4 3 3
25178 249/300 Write block 1621 len 4096 data: 5
25179 250/300 Write block 10 len 36864 data: 7 7 5 4 3 3 5 4 4
25180 251/300 Flush
25181 252/300 Write block 975 len 28672 data: 6 5 5 5 4 6 7
25182 253/300 Flush
25183 254/300 Read block 3080 len 28672
25184 255/300 Read block 354 len 32768
25185 256/300 Read block 2438 len 16384
25186 257/300 Read block 605 len 28672
25187 258/300 Read block 780 len 36864
25188 259/300 Write block 2796 len 12288 data: 4 5 4
25189 260/300 Read block 3698 len 4096
25190 261/300 Write block 451 len 8192 data: 4 4
25191 262/300 Write block 1371 len 40960 data: 6 6 6 3 3 2 2 2 2 3
25192 263/300 Read block 2788 len 4096
25193 264/300 Read block 1711 len 12288
25194 265/300 Read block 914 len 4096
25195 266/300 Read block 2377 len 32768
25196 267/300 Read block 4584 len 32768
25197 268/300 Read block 108 len 8192
25198 269/300 Write block 2945 len 8192 data: 4 4
25199 270/300 Read block 2619 len 32768
25200 271/300 Flush
25201 272/300 Flush
25202 273/300 Write block 4378 len 16384 data: 5 4 4 4
25203 274/300 Flush
25204 275/300 Read block 1761 len 36864
25205 276/300 Read block 1273 len 32768
25206 277/300 Read block 1218 len 16384
25207 278/300 Read block 3399 len 4096
25208 279/300 Read block 3752 len 20480
25209 280/300 Write block 4482 len 24576 data: 4 3 4 4 4 3
25210 281/300 Read block 647 len 24576
25211 282/300 Read block 960 len 36864
25212 283/300 Read block 4715 len 16384
25213 284/300 Write block 498 len 20480 data: 5 5 5 6 5
25214 285/300 Read block 3302 len 40960
25215 286/300 Read block 4663 len 28672
25216 287/300 Read block 4104 len 36864
25217 288/300 Read block 2905 len 8192
25218 289/300 Write block 4595 len 8192 data: 6 6
25219 290/300 Write block 4195 len 16384 data: 4 3 3 3
25220 291/300 Read block 3151 len 32768
25221 292/300 Read block 2928 len 12288
25222 293/300 Read block 573 len 20480
25223 294/300 Read block 3875 len 4096
25224 295/300 Flush
25225 296/300 Read block 4271 len 28672
25226 297/300 Read block 3219 len 24576
25227 298/300 Flush
25228 299/300 Write block 2366 len 4096 data: 6
25229 300/300 Write block 2873 len 12288 data: 4 4 3
25230 301/300 Read block 3468 len 24576
25231 Main task has work to do, go find it
25232 got dsc Start(1)
25233 start 1
25234 Feb 25 20:52:10.283 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: 04eabec3-d4e6-4ff6-a14f-f04089451355, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
25235 Replay: started 1, returned:Ok(())
25236 ----------------------------------------------------------------
25237 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25238 States: Active Offline Active
25239 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25240 7339 Acked Write 6 Done Sent Done false
25241 7340 Acked Write 10 Done Sent Done false
25242 7341 Acked Write 1 Done Sent Done false
25243 7342 Acked Write 3 Done Sent Done false
25244 7343 Acked Read 9 Done Sent Done false
25245 7344 Acked Read 10 Done Sent Done false
25246 7345 Acked Read 3 Done Sent Done false
25247 7346 Acked Flush 0 Done Sent Done false
25248 7347 Acked Read 2 Done Sent Done false
25249 7348 Acked Read 6 Done Sent Done false
25250 7349 Acked Write 9 Done Sent Done false
25251 7350 Acked Read 4 Done Sent Done false
25252 7351 Acked Read 8 Done Sent Done false
25253 7352 Acked Read 1 Done Sent Done false
25254 7353 Acked Read 4 Done Sent Done false
25255 7354 Acked Write 8 Done Sent Done false
25256 7355 Acked Read 3 Done Sent Done false
25257 7356 Acked Write 10 Done Sent Done false
25258 7357 Acked Read 8 Done Sent Done false
25259 7358 Acked Write 7 Done Sent Done false
25260 7359 Acked Write 2 Done Sent Done false
25261 7360 Acked Write 8 Done Sent Done false
25262 7361 Acked Read 6 Done Sent Done false
25263 7362 Acked Read 10 Done Sent Done false
25264 7363 Acked Read 1 Done Sent Done false
25265 7364 Acked Write 1 Done Sent Done false
25266 7365 Acked Read 1 Done Sent Done false
25267 7366 Acked Read 9 Done Sent Done false
25268 7367 Acked Write 1 Done Sent Done false
25269 7368 Acked Write 7 Done Sent Done false
25270 7369 Acked Read 9 Done Sent Done false
25271 7370 Acked Write 7 Done Sent Done false
25272 7371 Acked Read 10 Done Sent Done false
25273 7372 Acked Write 3 Done Sent Done false
25274 7373 Acked Read 6 Done Sent Done false
25275 7374 Acked Read 1 Done Sent Done false
25276 7375 Acked Read 7 Done Sent Done false
25277 7376 Acked Write 6 Done Sent Done false
25278 7377 Acked Write 2 Done Sent Done false
25279 7378 Acked Read 5 Done Sent Done false
25280 7379 Acked Write 9 Done Sent Done false
25281 7380 Acked Read 1 Done Sent Done false
25282 7381 Acked Read 2 Done Sent Done false
25283 7382 Acked Write 4 Done Sent Done false
25284 7383 Acked Read 9 Done Sent Done false
25285 7384 Acked Read 9 Done Sent Done false
25286 7385 Acked Read 9 Done Sent Done false
25287 7386 Acked Read 5 Done Sent Done false
25288 7387 Acked Read 9 Done Sent Done false
25289 7388 Acked Write 7 Done Sent Done false
25290 7389 Acked Read 1 Done Sent Done false
25291 7390 Acked Read 8 Done Sent Done false
25292 7391 Acked Read 10 Done Sent Done false
25293 7392 Acked Read 7 Done Sent Done false
25294 7393 Acked Read 4 Done Sent Done false
25295 7394 Acked Write 4 Done Sent Done false
25296 7395 Acked Read 10 Done Sent Done false
25297 7396 Acked Read 5 Done Sent Done false
25298 7397 Acked Write 3 Done Sent Done false
25299 7398 Acked Read 7 Done Sent Done false
25300 7399 Acked Write 2 Done Sent Done false
25301 7400 Acked Flush 0 Done Sent Done false
25302 7401 Acked Read 7 Done Sent Done false
25303 7402 Acked Write 5 Done Sent Done false
25304 7403 Acked Flush 0 Done Sent Done false
25305 7404 Acked Read 6 Done Sent Done false
25306 7405 Acked Write 6 Done Sent Done false
25307 7406 Acked Read 9 Done Sent Done false
25308 7407 Acked Read 8 Done Sent Done false
25309 7408 Acked Read 10 Done Sent Done false
25310 7409 Acked Read 10 Done Sent Done false
25311 7410 Acked Write 1 Done Sent Done false
25312 7411 Acked Write 2 Done Sent Done false
25313 7412 Acked Read 10 Done Sent Done false
25314 7413 Acked Read 3 Done Sent Done false
25315 7414 Acked Write 8 Done Sent Done false
25316 7415 Acked Read 4 Done Sent Done false
25317 7416 Acked Read 9 Done Sent Done false
25318 7417 Acked Write 1 Done Sent Done false
25319 7418 Acked Read 1 Done Sent Done false
25320 7419 Acked Read 7 Done Sent Done false
25321 7420 Acked Read 5 Done Sent Done false
25322 7421 Acked Read 7 Done Sent Done false
25323 7422 Acked Read 7 Done Sent Done false
25324 7423 Acked Read 3 Done Sent Done false
25325 7424 Acked Write 4 Done Sent Done false
25326 7425 Acked Read 6 Done Sent Done false
25327 7426 Acked Read 6 Done Sent Done false
25328 7427 Acked Read 3 Done Sent Done false
25329 7428 Acked Write 4 Done Sent Done false
25330 7429 Acked Read 7 Done Sent Done false
25331 7430 Acked Read 2 Done Sent Done false
25332 7431 Acked Read 1 Done Sent Done false
25333 7432 Acked Read 5 Done Sent Done false
25334 7433 Acked Read 5 Done Sent Done false
25335 7434 Acked Read 4 Done Sent Done false
25336 7435 Acked Read 9 Done Sent Done false
25337 7436 Acked Write 6 Done Sent Done false
25338 7437 Acked Read 8 Done Sent Done false
25339 7438 Acked Read 3 Done Sent Done false
25340 7439 Acked Read 3 Done Sent Done false
25341 7440 Acked Read 10 Done Sent Done false
25342 7441 Acked Read 4 Done Sent Done false
25343 7442 Acked Write 6 Done Sent Done false
25344 7443 Acked Write 3 Done Sent Done false
25345 7444 Acked Read 8 Done Sent Done false
25346 7445 Acked Write 2 Done Sent Done false
25347 7446 Acked Write 5 Done Sent Done false
25348 7447 Acked Write 10 Done Sent Done false
25349 7448 Acked Flush 0 Done Sent Done false
25350 7449 Acked Read 5 Done Sent Done false
25351 7450 Acked Write 10 Done Sent Done false
25352 7451 Acked Write 3 Done Sent Done false
25353 7452 Acked Read 3 Done Sent Done false
25354 7453 Acked Read 3 Done Sent Done false
25355 7454 Acked Write 6 Done Sent Done false
25356 7455 Acked Write 9 Done Sent Done false
25357 7456 Acked Read 1 Done Sent Done false
25358 7457 Acked Read 2 Done Sent Done false
25359 7458 Acked Read 7 Done Sent Done false
25360 7459 Acked Read 6 Done Sent Done false
25361 7460 Acked Read 9 Done Sent Done false
25362 7461 Acked Read 5 Done Sent Done false
25363 7462 Acked Write 8 Done Sent Done false
25364 7463 Acked Read 6 Done Sent Done false
25365 7464 Acked Write 3 Done Sent Done false
25366 7465 Acked Read 1 Done Sent Done false
25367 7466 Acked Write 8 Done Sent Done false
25368 7467 Acked Flush 0 Done Sent Done false
25369 7468 Acked Read 4 Done Sent Done false
25370 7469 Acked Read 8 Done Sent Done false
25371 7470 Acked Write 9 Done Sent Done false
25372 7471 Acked Write 1 Done Sent Done false
25373 7472 Acked Read 10 Done Sent Done false
25374 7473 Acked Read 5 Done Sent Done false
25375 7474 Acked Read 9 Done Sent Done false
25376 7475 Acked Write 5 Done Sent Done false
25377 7476 Acked Write 1 Done Sent Done false
25378 7477 Acked Write 8 Done Sent Done false
25379 7478 Acked Write 7 Done Sent Done false
25380 7479 Acked Read 7 Done Sent Done false
25381 7480 Acked Write 8 Done Sent Done false
25382 7481 Acked Read 6 Done Sent Done false
25383 7482 Acked Write 7 Done Sent Done false
25384 7483 Acked Read 3 Done Sent Done false
25385 7484 Acked Flush 0 Done Sent Done false
25386 7485 Acked Write 4 Done Sent Done false
25387 7486 Acked Read 9 Done Sent Done false
25388 7487 Acked Read 3 Done Sent Done false
25389 7488 Acked Write 7 Done Sent Done false
25390 7489 Acked Read 4 Done Sent Done false
25391 7490 Acked Write 7 Done Sent Done false
25392 7491 Acked Read 9 Done Sent Done false
25393 7492 Acked Read 4 Done Sent Done false
25394 7493 Acked Write 9 Done Sent Done false
25395 7494 Acked Read 2 Done Sent Done false
25396 7495 Acked Write 3 Done Sent Done false
25397 7496 Acked Read 3 Done Sent Done false
25398 7497 Acked Read 1 Done Sent Done false
25399 7498 Acked Write 5 Done Sent Done false
25400 7499 Acked Write 7 Done Sent Done false
25401 7500 Acked Read 8 Done Sent Done false
25402 7501 Acked Flush 0 Done Sent Done false
25403 7502 Acked Read 10 Done Sent Done false
25404 7503 Acked Flush 0 Done Sent Done false
25405 7504 Acked Read 5 Done Sent Done false
25406 7505 Acked Write 1 Done Sent Done false
25407 7506 Acked Write 2 Done Sent Done false
25408 7507 Acked Read 4 Done Sent Done false
25409 7508 Acked Read 6 Done Sent Done false
25410 7509 Acked Read 2 Done Sent Done false
25411 7510 Acked Write 10 Done Sent Done false
25412 7511 Acked Read 5 Done Sent Done false
25413 7512 Acked Flush 0 Done Sent Done false
25414 7513 Acked Write 1 Done Sent Done false
25415 7514 Acked Write 1 Done Sent Done false
25416 7515 Acked Write 1 Done Sent Done false
25417 7516 Acked Flush 0 Done Sent Done false
25418 7517 Acked Write 7 Done Sent Done false
25419 7518 Acked Write 9 Done Sent Done false
25420 7519 Acked Read 7 Done Sent Done false
25421 7520 Acked Write 4 Done Sent Done false
25422 7521 Acked Flush 0 Done Sent Done false
25423 7522 Acked Write 7 Done Sent Done false
25424 7523 Acked Write 9 Done Sent Done false
25425 7524 Acked Write 4 Done Sent Done false
25426 7525 Acked Read 1 Done Sent Done false
25427 7526 Acked Read 6 Done Sent Done false
25428 7527 Acked Read 6 Done Sent Done false
25429 7528 Acked Read 8 Done Sent Done false
25430 7529 Acked Read 8 Done Sent Done false
25431 7530 Acked Read 10 Done Sent Done false
25432 7531 Acked Write 5 Done Sent Done false
25433 7532 Acked Read 5 Done Sent Done false
25434 7533 Acked Write 3 Done Sent Done false
25435 7534 Acked Write 9 Done Sent Done false
25436 7535 Acked Write 6 Done Sent Done false
25437 7536 Acked Read 6 Done Sent Done false
25438 7537 Acked Flush 0 Done Sent Done false
25439 7538 Acked Write 6 Done Sent Done false
25440 7539 Acked Flush 0 Done Sent Done false
25441 7540 Acked Write 4 Done Sent Done false
25442 7541 Acked Read 9 Done Sent Done false
25443 7542 Acked Write 1 Done Sent Done false
25444 7543 Acked Flush 0 Done Sent Done false
25445 7544 Acked Flush 0 Done Sent Done false
25446 7545 Acked Read 1 Done Sent Done false
25447 7546 Acked Write 7 Done Sent Done false
25448 7547 Acked Read 3 Done Sent Done false
25449 7548 Acked Read 7 Done Sent Done false
25450 7549 Acked Read 9 Done Sent Done false
25451 7550 Acked Write 6 Done Sent Done false
25452 7551 Acked Write 10 Done Sent Done false
25453 7552 Acked Read 1 Done Sent Done false
25454 7553 Acked Write 8 Done Sent Done false
25455 7554 Acked Write 5 Done Sent Done false
25456 7555 Acked Write 6 Done Sent Done false
25457 7556 Acked Read 10 Done Sent Done false
25458 7557 Acked Write 6 Done Sent Done false
25459 7558 Acked Write 4 Done Sent Done false
25460 7559 Acked Read 8 Done Sent Done false
25461 7560 Acked Write 1 Done Sent Done false
25462 7561 Acked Read 1 Done Sent Done false
25463 7562 Acked Read 3 Done Sent Done false
25464 7563 Acked Read 4 Done Sent Done false
25465 7564 Acked Read 3 Done Sent Done false
25466 7565 Acked Write 4 Done Sent Done false
25467 7566 Acked Read 6 Done Sent Done false
25468 7567 Acked Read 10 Done Sent Done false
25469 7568 Acked Read 2 Done Sent Done false
25470 7569 Acked Read 5 Done Sent Done false
25471 7570 Acked Read 7 Done Sent Done false
25472 7571 Acked Write 2 Done Sent Done false
25473 7572 Acked Read 4 Done Sent Done false
25474 7573 Acked Write 1 Done Sent Done false
25475 7574 Acked Write 10 Done Sent Done false
25476 7575 Acked Write 8 Done Sent Done false
25477 7576 Acked Read 3 Done Sent Done false
25478 7577 Acked Read 1 Done Sent Done false
25479 7578 Acked Read 7 Done Sent Done false
25480 7579 Acked Read 8 Done Sent Done false
25481 7580 Acked Read 2 Done Sent Done false
25482 7581 Acked Read 8 Done Sent Done false
25483 7582 Acked Read 5 Done Sent Done false
25484 7583 Acked Read 5 Done Sent Done false
25485 7584 Acked Read 4 Done Sent Done false
25486 7585 Acked Read 5 Done Sent Done false
25487 7586 Acked Write 9 Done Sent Done false
25488 7587 Acked Write 1 Done Sent Done false
25489 7588 Acked Write 9 Done Sent Done false
25490 7589 Acked Flush 0 Done Sent Done false
25491 7590 Acked Write 7 Done Sent Done false
25492 7591 Acked Flush 0 Done Sent Done false
25493 7592 Acked Read 7 Done Sent Done false
25494 7593 Acked Read 8 Done Sent Done false
25495 7594 Acked Read 4 Done Sent Done false
25496 7595 Acked Read 7 Done Sent Done false
25497 7596 Acked Read 9 Done Sent Done false
25498 7597 Acked Write 3 Done Sent Done false
25499 7598 Acked Read 1 Done Sent Done false
25500 7599 Acked Write 2 Done Sent Done false
25501 7600 Acked Write 10 Done Sent Done false
25502 7601 Acked Read 1 Done Sent Done false
25503 7602 Acked Read 3 Done Sent Done false
25504 7603 Acked Read 1 Done Sent Done false
25505 7604 Acked Read 8 Done Sent Done false
25506 7605 Acked Read 8 Done Sent Done false
25507 7606 Acked Read 2 Done Sent Done false
25508 7607 Acked Write 2 Done Sent Done false
25509 7608 Acked Read 8 Done Sent Done false
25510 7609 Acked Flush 0 Done Sent Done false
25511 7610 Acked Flush 0 Done Sent Done false
25512 7611 Acked Write 4 Done Sent Done false
25513 7612 Acked Flush 0 Done Sent Done false
25514 7613 Acked Read 9 Done Sent Done false
25515 7614 Acked Read 8 Done Sent Done false
25516 7615 Acked Read 4 Done Sent Done false
25517 7616 Acked Read 1 Done Sent Done false
25518 7617 Acked Read 5 Done Sent Done false
25519 7618 Acked Write 6 Done Sent Done false
25520 7619 Acked Read 6 Done Sent Done false
25521 7620 Acked Read 9 Done Sent Done false
25522 7621 Acked Read 4 Done Sent Done false
25523 7622 Acked Write 5 Done Sent Done false
25524 7623 Acked Read 10 Done Sent Done false
25525 7624 Acked Read 7 Done Sent Done false
25526 7625 Acked Read 9 Done Sent Done false
25527 7626 Acked Read 2 Done Sent Done false
25528 7627 Acked Write 2 Done Sent Done false
25529 7628 Acked Write 4 Done Sent Done false
25530 7629 Acked Read 8 Done Sent Done false
25531 7630 Acked Read 3 Done Sent Done false
25532 7631 Acked Read 5 Done Sent Done false
25533 7632 Acked Read 1 Done Sent Done false
25534 7633 Acked Flush 0 Done Sent Done false
25535 7634 Acked Read 7 Done Sent Done false
25536 7635 Acked Read 6 Done Sent Done false
25537 7636 Acked Flush 0 Done Sent Done false
25538 7637 Acked Write 1 Done Sent Done false
25539 7638 Acked Write 3 Done Sent Done false
25540 7639 Acked Read 6 Done Sent Done false
25541 STATES DS:0 DS:1 DS:2 TOTAL
25542 Sent 0 301 0 301
25543 Done 301 0 301 602
25544 Skipped 0 0 0 0
25545 Error 0 0 0 0
25546 Last Flush: Some(JobId(7636)) Some(JobId(7338)) Some(JobId(7636))
25547 Downstairs last five completed: 7338 7337 7336 7335 7334
25548 Upstairs last five completed: 7639 7638 7637 7636 7635
25549 CLIENT: Up:0 ds:301 act:2
255502025-02-25T20:52:10.785ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
255512025-02-25T20:52:11.588ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
255522025-02-25T20:52:12.589ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
255532025-02-25T20:52:14.005ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
25554 [8820] got start action
25555 [8820] I am going to restart
25556 Starting downstairs at port 8820
25557 Make output file at "/tmp/dsc/downstairs-8820.txt"
25558 [8820][1] reports Starting
25559 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1228)
25560 [8820][1] reports Running
25561 ----------------------------------------------------------------
25562 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25563 States: Active Offline Active
25564 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25565 7339 Acked Write 6 Done Sent Done false
25566 7340 Acked Write 10 Done Sent Done false
25567 7341 Acked Write 1 Done Sent Done false
25568 7342 Acked Write 3 Done Sent Done false
25569 7343 Acked Read 9 Done Sent Done false
25570 7344 Acked Read 10 Done Sent Done false
25571 7345 Acked Read 3 Done Sent Done false
25572 7346 Acked Flush 0 Done Sent Done false
25573 7347 Acked Read 2 Done Sent Done false
25574 7348 Acked Read 6 Done Sent Done false
25575 7349 Acked Write 9 Done Sent Done false
25576 7350 Acked Read 4 Done Sent Done false
25577 7351 Acked Read 8 Done Sent Done false
25578 7352 Acked Read 1 Done Sent Done false
25579 7353 Acked Read 4 Done Sent Done false
25580 7354 Acked Write 8 Done Sent Done false
25581 7355 Acked Read 3 Done Sent Done false
25582 7356 Acked Write 10 Done Sent Done false
25583 7357 Acked Read 8 Done Sent Done false
25584 7358 Acked Write 7 Done Sent Done false
25585 7359 Acked Write 2 Done Sent Done false
25586 7360 Acked Write 8 Done Sent Done false
25587 7361 Acked Read 6 Done Sent Done false
25588 7362 Acked Read 10 Done Sent Done false
25589 7363 Acked Read 1 Done Sent Done false
25590 7364 Acked Write 1 Done Sent Done false
25591 7365 Acked Read 1 Done Sent Done false
25592 7366 Acked Read 9 Done Sent Done false
25593 7367 Acked Write 1 Done Sent Done false
25594 7368 Acked Write 7 Done Sent Done false
25595 7369 Acked Read 9 Done Sent Done false
25596 7370 Acked Write 7 Done Sent Done false
25597 7371 Acked Read 10 Done Sent Done false
25598 7372 Acked Write 3 Done Sent Done false
25599 7373 Acked Read 6 Done Sent Done false
25600 7374 Acked Read 1 Done Sent Done false
25601 7375 Acked Read 7 Done Sent Done false
25602 7376 Acked Write 6 Done Sent Done false
25603 7377 Acked Write 2 Done Sent Done false
25604 7378 Acked Read 5 Done Sent Done false
25605 7379 Acked Write 9 Done Sent Done false
25606 7380 Acked Read 1 Done Sent Done false
25607 7381 Acked Read 2 Done Sent Done false
25608 7382 Acked Write 4 Done Sent Done false
25609 7383 Acked Read 9 Done Sent Done false
25610 7384 Acked Read 9 Done Sent Done false
25611 7385 Acked Read 9 Done Sent Done false
25612 7386 Acked Read 5 Done Sent Done false
25613 7387 Acked Read 9 Done Sent Done false
25614 7388 Acked Write 7 Done Sent Done false
25615 7389 Acked Read 1 Done Sent Done false
25616 7390 Acked Read 8 Done Sent Done false
25617 7391 Acked Read 10 Done Sent Done false
25618 7392 Acked Read 7 Done Sent Done false
25619 7393 Acked Read 4 Done Sent Done false
25620 7394 Acked Write 4 Done Sent Done false
25621 7395 Acked Read 10 Done Sent Done false
25622 7396 Acked Read 5 Done Sent Done false
25623 7397 Acked Write 3 Done Sent Done false
25624 7398 Acked Read 7 Done Sent Done false
25625 7399 Acked Write 2 Done Sent Done false
25626 7400 Acked Flush 0 Done Sent Done false
25627 7401 Acked Read 7 Done Sent Done false
25628 7402 Acked Write 5 Done Sent Done false
25629 7403 Acked Flush 0 Done Sent Done false
25630 7404 Acked Read 6 Done Sent Done false
25631 7405 Acked Write 6 Done Sent Done false
25632 7406 Acked Read 9 Done Sent Done false
25633 7407 Acked Read 8 Done Sent Done false
25634 7408 Acked Read 10 Done Sent Done false
25635 7409 Acked Read 10 Done Sent Done false
25636 7410 Acked Write 1 Done Sent Done false
25637 7411 Acked Write 2 Done Sent Done false
25638 7412 Acked Read 10 Done Sent Done false
25639 7413 Acked Read 3 Done Sent Done false
25640 7414 Acked Write 8 Done Sent Done false
25641 7415 Acked Read 4 Done Sent Done false
25642 7416 Acked Read 9 Done Sent Done false
25643 7417 Acked Write 1 Done Sent Done false
25644 7418 Acked Read 1 Done Sent Done false
25645 7419 Acked Read 7 Done Sent Done false
25646 7420 Acked Read 5 Done Sent Done false
25647 7421 Acked Read 7 Done Sent Done false
25648 7422 Acked Read 7 Done Sent Done false
25649 7423 Acked Read 3 Done Sent Done false
25650 7424 Acked Write 4 Done Sent Done false
25651 7425 Acked Read 6 Done Sent Done false
25652 7426 Acked Read 6 Done Sent Done false
25653 7427 Acked Read 3 Done Sent Done false
25654 7428 Acked Write 4 Done Sent Done false
25655 7429 Acked Read 7 Done Sent Done false
25656 7430 Acked Read 2 Done Sent Done false
25657 7431 Acked Read 1 Done Sent Done false
25658 7432 Acked Read 5 Done Sent Done false
25659 7433 Acked Read 5 Done Sent Done false
25660 7434 Acked Read 4 Done Sent Done false
25661 7435 Acked Read 9 Done Sent Done false
25662 7436 Acked Write 6 Done Sent Done false
25663 7437 Acked Read 8 Done Sent Done false
25664 7438 Acked Read 3 Done Sent Done false
25665 7439 Acked Read 3 Done Sent Done false
25666 7440 Acked Read 10 Done Sent Done false
25667 7441 Acked Read 4 Done Sent Done false
25668 7442 Acked Write 6 Done Sent Done false
25669 7443 Acked Write 3 Done Sent Done false
25670 7444 Acked Read 8 Done Sent Done false
25671 7445 Acked Write 2 Done Sent Done false
25672 7446 Acked Write 5 Done Sent Done false
25673 7447 Acked Write 10 Done Sent Done false
25674 7448 Acked Flush 0 Done Sent Done false
25675 7449 Acked Read 5 Done Sent Done false
25676 7450 Acked Write 10 Done Sent Done false
25677 7451 Acked Write 3 Done Sent Done false
25678 7452 Acked Read 3 Done Sent Done false
25679 7453 Acked Read 3 Done Sent Done false
25680 7454 Acked Write 6 Done Sent Done false
25681 7455 Acked Write 9 Done Sent Done false
25682 7456 Acked Read 1 Done Sent Done false
25683 7457 Acked Read 2 Done Sent Done false
25684 7458 Acked Read 7 Done Sent Done false
25685 7459 Acked Read 6 Done Sent Done false
25686 7460 Acked Read 9 Done Sent Done false
25687 7461 Acked Read 5 Done Sent Done false
25688 7462 Acked Write 8 Done Sent Done false
25689 7463 Acked Read 6 Done Sent Done false
25690 7464 Acked Write 3 Done Sent Done false
25691 7465 Acked Read 1 Done Sent Done false
25692 7466 Acked Write 8 Done Sent Done false
25693 7467 Acked Flush 0 Done Sent Done false
25694 7468 Acked Read 4 Done Sent Done false
25695 7469 Acked Read 8 Done Sent Done false
25696 7470 Acked Write 9 Done Sent Done false
25697 7471 Acked Write 1 Done Sent Done false
25698 7472 Acked Read 10 Done Sent Done false
25699 7473 Acked Read 5 Done Sent Done false
25700 7474 Acked Read 9 Done Sent Done false
25701 7475 Acked Write 5 Done Sent Done false
25702 7476 Acked Write 1 Done Sent Done false
25703 7477 Acked Write 8 Done Sent Done false
25704 7478 Acked Write 7 Done Sent Done false
25705 7479 Acked Read 7 Done Sent Done false
25706 7480 Acked Write 8 Done Sent Done false
25707 7481 Acked Read 6 Done Sent Done false
25708 7482 Acked Write 7 Done Sent Done false
25709 7483 Acked Read 3 Done Sent Done false
25710 7484 Acked Flush 0 Done Sent Done false
25711 7485 Acked Write 4 Done Sent Done false
25712 7486 Acked Read 9 Done Sent Done false
25713 7487 Acked Read 3 Done Sent Done false
25714 7488 Acked Write 7 Done Sent Done false
25715 7489 Acked Read 4 Done Sent Done false
25716 7490 Acked Write 7 Done Sent Done false
25717 7491 Acked Read 9 Done Sent Done false
25718 7492 Acked Read 4 Done Sent Done false
25719 7493 Acked Write 9 Done Sent Done false
25720 7494 Acked Read 2 Done Sent Done false
25721 7495 Acked Write 3 Done Sent Done false
25722 7496 Acked Read 3 Done Sent Done false
25723 7497 Acked Read 1 Done Sent Done false
25724 7498 Acked Write 5 Done Sent Done false
25725 7499 Acked Write 7 Done Sent Done false
25726 7500 Acked Read 8 Done Sent Done false
25727 7501 Acked Flush 0 Done Sent Done false
25728 7502 Acked Read 10 Done Sent Done false
25729 7503 Acked Flush 0 Done Sent Done false
25730 7504 Acked Read 5 Done Sent Done false
25731 7505 Acked Write 1 Done Sent Done false
25732 7506 Acked Write 2 Done Sent Done false
25733 7507 Acked Read 4 Done Sent Done false
25734 7508 Acked Read 6 Done Sent Done false
25735 7509 Acked Read 2 Done Sent Done false
25736 7510 Acked Write 10 Done Sent Done false
25737 7511 Acked Read 5 Done Sent Done false
25738 7512 Acked Flush 0 Done Sent Done false
25739 7513 Acked Write 1 Done Sent Done false
25740 7514 Acked Write 1 Done Sent Done false
25741 7515 Acked Write 1 Done Sent Done false
25742 7516 Acked Flush 0 Done Sent Done false
25743 7517 Acked Write 7 Done Sent Done false
25744 7518 Acked Write 9 Done Sent Done false
25745 7519 Acked Read 7 Done Sent Done false
25746 7520 Acked Write 4 Done Sent Done false
25747 7521 Acked Flush 0 Done Sent Done false
25748 7522 Acked Write 7 Done Sent Done false
25749 7523 Acked Write 9 Done Sent Done false
25750 7524 Acked Write 4 Done Sent Done false
25751 7525 Acked Read 1 Done Sent Done false
25752 7526 Acked Read 6 Done Sent Done false
25753 7527 Acked Read 6 Done Sent Done false
25754 7528 Acked Read 8 Done Sent Done false
25755 7529 Acked Read 8 Done Sent Done false
25756 7530 Acked Read 10 Done Sent Done false
25757 7531 Acked Write 5 Done Sent Done false
25758 7532 Acked Read 5 Done Sent Done false
25759 7533 Acked Write 3 Done Sent Done false
25760 7534 Acked Write 9 Done Sent Done false
25761 7535 Acked Write 6 Done Sent Done false
25762 7536 Acked Read 6 Done Sent Done false
25763 7537 Acked Flush 0 Done Sent Done false
25764 7538 Acked Write 6 Done Sent Done false
25765 7539 Acked Flush 0 Done Sent Done false
25766 7540 Acked Write 4 Done Sent Done false
25767 7541 Acked Read 9 Done Sent Done false
25768 7542 Acked Write 1 Done Sent Done false
25769 7543 Acked Flush 0 Done Sent Done false
25770 7544 Acked Flush 0 Done Sent Done false
25771 7545 Acked Read 1 Done Sent Done false
25772 7546 Acked Write 7 Done Sent Done false
25773 7547 Acked Read 3 Done Sent Done false
25774 7548 Acked Read 7 Done Sent Done false
25775 7549 Acked Read 9 Done Sent Done false
25776 7550 Acked Write 6 Done Sent Done false
25777 7551 Acked Write 10 Done Sent Done false
25778 7552 Acked Read 1 Done Sent Done false
25779 7553 Acked Write 8 Done Sent Done false
25780 7554 Acked Write 5 Done Sent Done false
25781 7555 Acked Write 6 Done Sent Done false
25782 7556 Acked Read 10 Done Sent Done false
25783 7557 Acked Write 6 Done Sent Done false
25784 7558 Acked Write 4 Done Sent Done false
25785 7559 Acked Read 8 Done Sent Done false
25786 7560 Acked Write 1 Done Sent Done false
25787 7561 Acked Read 1 Done Sent Done false
25788 7562 Acked Read 3 Done Sent Done false
25789 7563 Acked Read 4 Done Sent Done false
25790 7564 Acked Read 3 Done Sent Done false
25791 7565 Acked Write 4 Done Sent Done false
25792 7566 Acked Read 6 Done Sent Done false
25793 7567 Acked Read 10 Done Sent Done false
25794 7568 Acked Read 2 Done Sent Done false
25795 7569 Acked Read 5 Done Sent Done false
25796 7570 Acked Read 7 Done Sent Done false
25797 7571 Acked Write 2 Done Sent Done false
25798 7572 Acked Read 4 Done Sent Done false
25799 7573 Acked Write 1 Done Sent Done false
25800 7574 Acked Write 10 Done Sent Done false
25801 7575 Acked Write 8 Done Sent Done false
25802 7576 Acked Read 3 Done Sent Done false
25803 7577 Acked Read 1 Done Sent Done false
25804 7578 Acked Read 7 Done Sent Done false
25805 7579 Acked Read 8 Done Sent Done false
25806 7580 Acked Read 2 Done Sent Done false
25807 7581 Acked Read 8 Done Sent Done false
25808 7582 Acked Read 5 Done Sent Done false
25809 7583 Acked Read 5 Done Sent Done false
25810 7584 Acked Read 4 Done Sent Done false
25811 7585 Acked Read 5 Done Sent Done false
25812 7586 Acked Write 9 Done Sent Done false
25813 7587 Acked Write 1 Done Sent Done false
25814 7588 Acked Write 9 Done Sent Done false
25815 7589 Acked Flush 0 Done Sent Done false
25816 7590 Acked Write 7 Done Sent Done false
25817 7591 Acked Flush 0 Done Sent Done false
25818 7592 Acked Read 7 Done Sent Done false
25819 7593 Acked Read 8 Done Sent Done false
25820 7594 Acked Read 4 Done Sent Done false
25821 7595 Acked Read 7 Done Sent Done false
25822 7596 Acked Read 9 Done Sent Done false
25823 7597 Acked Write 3 Done Sent Done false
25824 7598 Acked Read 1 Done Sent Done false
25825 7599 Acked Write 2 Done Sent Done false
25826 7600 Acked Write 10 Done Sent Done false
25827 7601 Acked Read 1 Done Sent Done false
25828 7602 Acked Read 3 Done Sent Done false
25829 7603 Acked Read 1 Done Sent Done false
25830 7604 Acked Read 8 Done Sent Done false
25831 7605 Acked Read 8 Done Sent Done false
25832 7606 Acked Read 2 Done Sent Done false
25833 7607 Acked Write 2 Done Sent Done false
25834 7608 Acked Read 8 Done Sent Done false
25835 7609 Acked Flush 0 Done Sent Done false
25836 7610 Acked Flush 0 Done Sent Done false
25837 7611 Acked Write 4 Done Sent Done false
25838 7612 Acked Flush 0 Done Sent Done false
25839 7613 Acked Read 9 Done Sent Done false
25840 7614 Acked Read 8 Done Sent Done false
25841 7615 Acked Read 4 Done Sent Done false
25842 7616 Acked Read 1 Done Sent Done false
25843 7617 Acked Read 5 Done Sent Done false
25844 7618 Acked Write 6 Done Sent Done false
25845 7619 Acked Read 6 Done Sent Done false
25846 7620 Acked Read 9 Done Sent Done false
25847 7621 Acked Read 4 Done Sent Done false
25848 7622 Acked Write 5 Done Sent Done false
25849 7623 Acked Read 10 Done Sent Done false
25850 7624 Acked Read 7 Done Sent Done false
25851 7625 Acked Read 9 Done Sent Done false
25852 7626 Acked Read 2 Done Sent Done false
25853 7627 Acked Write 2 Done Sent Done false
25854 7628 Acked Write 4 Done Sent Done false
25855 7629 Acked Read 8 Done Sent Done false
25856 7630 Acked Read 3 Done Sent Done false
25857 7631 Acked Read 5 Done Sent Done false
25858 7632 Acked Read 1 Done Sent Done false
25859 7633 Acked Flush 0 Done Sent Done false
25860 7634 Acked Read 7 Done Sent Done false
25861 7635 Acked Read 6 Done Sent Done false
25862 7636 Acked Flush 0 Done Sent Done false
25863 7637 Acked Write 1 Done Sent Done false
25864 7638 Acked Write 3 Done Sent Done false
25865 7639 Acked Read 6 Done Sent Done false
25866 STATES DS:0 DS:1 DS:2 TOTAL
25867 Sent 0 301 0 301
25868 Done 301 0 301 602
25869 Skipped 0 0 0 0
25870 Error 0 0 0 0
25871 Last Flush: Some(JobId(7636)) Some(JobId(7338)) Some(JobId(7636))
25872 Downstairs last five completed: 7338 7337 7336 7335 7334
25873 Upstairs last five completed: 7639 7638 7637 7636 7635
25874 CLIENT: Up:0 ds:301 act:2
258752025-02-25T20:52:15.594ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258762025-02-25T20:52:16.023ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258772025-02-25T20:52:16.023ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61525) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258782025-02-25T20:52:16.025ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258792025-02-25T20:52:16.025ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258802025-02-25T20:52:16.025ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258812025-02-25T20:52:16.025ZINFOcrucible: send last flush ID to this DS: Some(JobId(7338)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258822025-02-25T20:52:16.025ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258832025-02-25T20:52:16.025ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7338)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258842025-02-25T20:52:16.025ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7338)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
25885 ----------------------------------------------------------------
25886 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25887 Downstairs last five completed: 7640 7639 7638 7637 7636
25888 Upstairs last five completed: 7640 7639 7638 7637 7636
25889 CLIENT: Up:0 ds:0 act:3
25890 Replay: All jobs finished
25891 Main task has work to do, go find it
25892 got dsc Stop(1)
25893 stop 1
25894 Feb 25 20:52:18.289 INFO request completed, latency_us: 535[8820] Got stop action so:false kr:false
25895 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: ae75a9a3-8df4-477e-b9de-805774cbf5e8, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
25896 Feb 25 20:52:18.290 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/1, method: GET, req_id: d517a800-2fe2-4df7-bdaa-241e3d3fe1c3, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
258972025-02-25T20:52:18.290ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258982025-02-25T20:52:18.290ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
258992025-02-25T20:52:18.290ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
259002025-02-25T20:52:18.290ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
25901 [8820] Exited with: ExitStatus(unix_wait_status(9))
25902 [8820][1] reports Exit
25903 Feb 25 20:52:22.292 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/1, method: GET, req_id: a13a6974-c94b-4617-8624-4b91be127bf8, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
25904 001/300 Write block 612 len 28672 data: 5 6 6 6 7 7 7
25905 002/300 Read block 3968 len 28672
25906 003/300 Read block 2527 len 32768
25907 004/300 Write block 1132 len 20480 data: 3 4 5 5 4
25908 005/300 Write block 571 len 12288 data: 2 2 3
25909 006/300 Read block 4023 len 40960
25910 007/300 Write block 2366 len 12288 data: 7 6 4
25911 008/300 Read block 1711 len 24576
25912 009/300 Read block 1185 len 32768
25913 010/300 Read block 4238 len 28672
25914 011/300 Read block 2355 len 40960
25915 012/300 Flush
25916 013/300 Read block 143 len 32768
25917 014/300 Write block 4027 len 4096 data: 4
25918 015/300 Write block 1950 len 16384 data: 6 6 4 5
25919 016/300 Write block 1279 len 12288 data: 2 2 2
25920 017/300 Write block 1296 len 4096 data: 3
25921 018/300 Read block 2458 len 36864
25922 019/300 Write block 172 len 24576 data: 5 7 7 6 7 6
25923 020/300 Read block 636 len 16384
25924 021/300 Read block 1778 len 36864
25925 022/300 Read block 1885 len 32768
25926 023/300 Write block 813 len 12288 data: 5 5 6
25927 024/300 Read block 1194 len 28672
25928 025/300 Write block 4717 len 24576 data: 6 5 4 5 6 6
25929 026/300 Write block 3711 len 40960 data: 7 7 6 7 7 6 5 6 7 8
25930 027/300 Write block 4136 len 16384 data: 5 5 5 6
25931 028/300 Write block 2470 len 12288 data: 6 8 7
25932 029/300 Read block 2769 len 12288
25933 030/300 Write block 3477 len 4096 data: 5
25934 031/300 Write block 673 len 32768 data: 7 9 11 8 5 4 4 2
25935 032/300 Read block 4636 len 4096
25936 033/300 Read block 3668 len 36864
25937 034/300 Read block 3806 len 28672
25938 035/300 Write block 1383 len 36864 data: 3 3 5 5 7 8 6 6 6
25939 036/300 Read block 880 len 32768
25940 037/300 Read block 3633 len 12288
25941 038/300 Write block 4160 len 12288 data: 6 9 11
25942 039/300 Write block 2027 len 12288 data: 4 6 6
25943 040/300 Read block 2218 len 40960
25944 041/300 Write block 1957 len 16384 data: 6 6 6 5
25945 042/300 Write block 1862 len 8192 data: 6 6
25946 043/300 Read block 3056 len 32768
25947 044/300 Read block 3022 len 8192
25948 045/300 Write block 4835 len 32768 data: 4 4 3 5 6 4 4 4
25949 046/300 Write block 2357 len 32768 data: 6 4 6 3 3 4 6 6
25950 047/300 Read block 3594 len 24576
25951 048/300 Read block 3947 len 16384
25952 049/300 Write block 4167 len 12288 data: 10 8 9
25953 050/300 Read block 449 len 4096
25954 051/300 Read block 2879 len 16384
25955 052/300 Write block 851 len 16384 data: 4 4 4 6
25956 053/300 Write block 1242 len 20480 data: 3 3 3 5 5
25957 054/300 Write block 2307 len 24576 data: 5 4 5 5 5 5
25958 055/300 Write block 2170 len 36864 data: 5 4 3 3 3 3 3 2 4
25959 056/300 Read block 3434 len 12288
25960 057/300 Read block 3859 len 4096
25961 058/300 Flush
25962 059/300 Write block 218 len 28672 data: 6 7 8 7 7 6 5
25963 060/300 Read block 4655 len 16384
25964 061/300 Write block 4206 len 40960 data: 3 3 4 4 4 5 4 3 3 5
25965 062/300 Write block 2900 len 12288 data: 8 8 5
25966 063/300 Read block 4300 len 32768
25967 064/300 Write block 3465 len 36864 data: 3 3 4 4 5 4 3 3 3
25968 065/300 Flush
25969 066/300 Read block 3709 len 36864
25970 067/300 Flush
25971 068/300 Read block 2525 len 24576
25972 069/300 Flush
25973 070/300 Write block 4139 len 36864 data: 7 6 6 7 5 5 6 4 3
25974 071/300 Read block 2170 len 40960
25975 072/300 Write block 4700 len 4096 data: 6
25976 073/300 Read block 2834 len 8192
25977 074/300 Write block 4690 len 16384 data: 7 7 6 5
25978 075/300 Read block 3804 len 8192
25979 076/300 Read block 4335 len 40960
25980 077/300 Read block 3800 len 4096
25981 078/300 Read block 2162 len 40960
25982 079/300 Read block 348 len 32768
25983 080/300 Write block 4041 len 36864 data: 4 2 2 2 3 4 4 4 4
25984 081/300 Write block 1745 len 20480 data: 3 3 3 4 4
25985 082/300 Read block 886 len 16384
25986 083/300 Read block 2720 len 8192
25987 084/300 Read block 1743 len 20480
25988 085/300 Read block 3063 len 36864
25989 086/300 Read block 2126 len 32768
25990 087/300 Write block 4483 len 24576 data: 4 5 5 5 4 3
25991 088/300 Write block 1148 len 4096 data: 3
25992 089/300 Write block 1694 len 8192 data: 3 3
25993 090/300 Read block 230 len 40960
25994 091/300 Write block 588 len 24576 data: 6 6 8 7 6 7
25995 092/300 Read block 1973 len 20480
25996 093/300 Write block 702 len 40960 data: 5 5 7 8 9 7 5 5 5 5
25997 094/300 Flush
25998 095/300 Read block 3303 len 40960
25999 096/300 Write block 3352 len 32768 data: 6 5 4 3 4 4 5 3
26000 097/300 Write block 1417 len 32768 data: 5 5 5 6 6 6 4 4
26001 098/300 Read block 2150 len 16384
26002 099/300 Read block 230 len 12288
26003 100/300 Read block 4853 len 16384
26004 101/300 Flush
26005 102/300 Write block 6 len 36864 data: 6 6 6 7 8 8 6 5 4
26006 103/300 Write block 174 len 36864 data: 8 7 8 7 5 4 4 5 5
26007 104/300 Write block 1848 len 32768 data: 3 5 5 6 4 4 3 5
26008 105/300 Write block 36 len 28672 data: 4 4 2 4 4 4 2
26009 106/300 Write block 4196 len 8192 data: 4 4
26010 107/300 Write block 1614 len 28672 data: 6 5 6 8 6 5 4
26011 108/300 Read block 3350 len 36864
26012 109/300 Write block 2142 len 40960 data: 4 4 4 4 4 4 6 5 6 7
26013 110/300 Read block 2108 len 20480
26014 111/300 Write block 3339 len 28672 data: 3 3 2 2 3 4 4
26015 112/300 Write block 1065 len 40960 data: 6 4 5 5 3 3 4 5 7 7
26016 113/300 Read block 1563 len 40960
26017 114/300 Write block 1042 len 20480 data: 5 6 7 6 6
26018 115/300 Read block 3445 len 32768
26019 116/300 Read block 664 len 24576
26020 117/300 Read block 1298 len 32768
26021 118/300 Read block 4862 len 16384
26022 119/300 Read block 1811 len 36864
26023 120/300 Write block 1903 len 40960 data: 5 4 4 4 3 3 4 4 4 4
26024 121/300 Write block 2961 len 16384 data: 9 7 8 7
26025 122/300 Read block 2001 len 28672
26026 123/300 Read block 499 len 32768
26027 124/300 Flush
26028 125/300 Read block 4047 len 4096
26029 126/300 Write block 1014 len 16384 data: 6 5 5 4
26030 127/300 Read block 4764 len 28672
26031 128/300 Read block 4914 len 40960
26032 129/300 Write block 4598 len 16384 data: 4 3 5 5
26033 130/300 Write block 249 len 24576 data: 3 5 5 6 6 6
26034 131/300 Write block 4201 len 36864 data: 2 3 3 3 3 4 4 5 5
26035 132/300 Read block 2361 len 8192
26036 133/300 Read block 4896 len 32768
26037 134/300 Read block 3694 len 20480
26038 135/300 Read block 4130 len 20480
26039 136/300 Write block 2895 len 20480 data: 8 7 7 7 8
26040 137/300 Read block 1305 len 24576
26041 138/300 Read block 2862 len 8192
26042 139/300 Write block 488 len 32768 data: 5 6 4 4 4 5 5 5
26043 140/300 Read block 1997 len 36864
26044 141/300 Write block 3743 len 12288 data: 2 2 4
26045 142/300 Flush
26046 143/300 Read block 3137 len 36864
26047 144/300 Write block 2780 len 32768 data: 3 5 5 6 4 3 2 3
26048 145/300 Write block 4702 len 36864 data: 4 4 5 4 3 4 4 5 5
26049 146/300 Read block 989 len 16384
26050 147/300 Write block 94 len 40960 data: 7 6 4 4 4 4 3 4 3 5
26051 148/300 Read block 3092 len 28672
26052 149/300 Write block 4974 len 8192 data: 4 4
26053 150/300 Read block 3210 len 8192
26054 151/300 Flush
26055 152/300 Read block 2666 len 40960
26056 153/300 Flush
26057 154/300 Write block 3803 len 4096 data: 3
26058 155/300 Write block 3564 len 20480 data: 4 4 4 5 4
26059 156/300 Read block 4358 len 4096
26060 157/300 Read block 4632 len 12288
26061 158/300 Write block 2442 len 20480 data: 3 4 4 7 7
26062 159/300 Write block 3022 len 12288 data: 5 5 4
26063 160/300 Write block 653 len 32768 data: 3 4 5 7 7 8 8 10
26064 161/300 Write block 3785 len 8192 data: 6 7
26065 162/300 Read block 2862 len 20480
26066 163/300 Read block 3588 len 40960
26067 164/300 Read block 4422 len 40960
26068 165/300 Flush
26069 166/300 Read block 2549 len 4096
26070 167/300 Read block 4804 len 28672
26071 168/300 Write block 2320 len 4096 data: 7
26072 169/300 Write block 1662 len 36864 data: 5 4 3 2 2 2 3 5 5
26073 170/300 Read block 3329 len 16384
26074 171/300 Read block 2920 len 40960
26075 172/300 Write block 2417 len 12288 data: 6 5 4
26076 173/300 Read block 4985 len 16384
26077 174/300 Read block 1399 len 12288
26078 175/300 Write block 4255 len 24576 data: 5 4 4 4 5 4
26079 176/300 Write block 4261 len 28672 data: 3 3 4 5 5 4 4
26080 177/300 Write block 958 len 12288 data: 8 6 8
26081 178/300 Read block 4719 len 24576
26082 179/300 Write block 3636 len 8192 data: 8 8
26083 180/300 Read block 3364 len 12288
26084 181/300 Read block 483 len 28672
26085 182/300 Write block 378 len 32768 data: 3 2 2 2 2 5 5 5
26086 183/300 Write block 3101 len 36864 data: 5 5 7 7 8 7 6 4 5
26087 184/300 Read block 4895 len 24576
26088 185/300 Flush
26089 186/300 Write block 2459 len 32768 data: 5 4 3 3 3 6 4 3
26090 187/300 Write block 3034 len 24576 data: 3 4 4 4 3 3
26091 188/300 Read block 761 len 16384
26092 189/300 Read block 1530 len 16384
26093 190/300 Flush
26094 191/300 Write block 3753 len 16384 data: 4 4 6 6
26095 192/300 Write block 18 len 16384 data: 5 4 4 4
26096 193/300 Write block 577 len 40960 data: 7 6 7 7 6 5 4 5 4 3
26097 194/300 Read block 2695 len 28672
26098 195/300 Read block 571 len 20480
26099 196/300 Flush
26100 197/300 Write block 1645 len 32768 data: 2 2 2 3 4 3 5 4
26101 198/300 Read block 3659 len 8192
26102 199/300 Read block 3633 len 24576
26103 200/300 Read block 4367 len 36864
26104 201/300 Write block 882 len 4096 data: 4
26105 202/300 Read block 1197 len 16384
26106 203/300 Write block 2412 len 36864 data: 4 5 6 6 5 7 6 5 4
26107 204/300 Flush
26108 205/300 Write block 2362 len 8192 data: 5 7
26109 206/300 Read block 3873 len 16384
26110 207/300 Write block 2220 len 32768 data: 5 7 8 8 7 7 6 7
26111 208/300 Write block 68 len 12288 data: 3 4 5
26112 209/300 Write block 1434 len 4096 data: 4
26113 210/300 Read block 489 len 24576
26114 211/300 Write block 1010 len 28672 data: 5 5 6 6 7 6 6
26115 212/300 Read block 4044 len 16384
26116 213/300 Read block 4242 len 4096
26117 214/300 Write block 3051 len 36864 data: 4 4 3 4 4 3 3 3 4
26118 215/300 Read block 2334 len 16384
26119 216/300 Write block 1115 len 20480 data: 5 7 6 6 5
26120 217/300 Read block 4405 len 32768
26121 218/300 Read block 635 len 20480
26122 219/300 Write block 4875 len 12288 data: 5 6 6
26123 220/300 Read block 1710 len 28672
26124 221/300 Read block 2551 len 8192
26125 222/300 Read block 2889 len 36864
26126 223/300 Write block 4568 len 24576 data: 4 4 5 5 5 4
26127 224/300 Read block 2380 len 4096
26128 225/300 Write block 2036 len 40960 data: 7 6 5 5 4 4 4 4 3 4
26129 226/300 Write block 4090 len 24576 data: 4 4 5 4 6 6
26130 227/300 Read block 773 len 40960
26131 228/300 Read block 4405 len 32768
26132 229/300 Write block 1872 len 12288 data: 4 3 6
26133 230/300 Write block 1985 len 32768 data: 7 5 6 7 7 6 7 7
26134 231/300 Read block 4975 len 8192
26135 232/300 Write block 661 len 8192 data: 9 8
26136 233/300 Read block 1649 len 16384
26137 234/300 Write block 3179 len 32768 data: 5 6 6 6 6 5 5 6
26138 235/300 Write block 2071 len 4096 data: 2
26139 236/300 Read block 1978 len 24576
26140 237/300 Read block 900 len 28672
26141 238/300 Write block 1141 len 32768 data: 5 4 3 3 3 3 3 4
26142 239/300 Write block 3537 len 16384 data: 3 3 3 3
26143 240/300 Flush
26144 241/300 Read block 4721 len 8192
26145 242/300 Read block 4916 len 8192
26146 243/300 Read block 1123 len 36864
26147 244/300 Write block 3802 len 16384 data: 3 4 3 3
26148 245/300 Read block 1488 len 24576
26149 246/300 Write block 3294 len 16384 data: 5 6 5 5
26150 247/300 Write block 4627 len 28672 data: 4 4 5 6 6 4 4
26151 248/300 Write block 4201 len 32768 data: 3 4 4 4 4 5 5 6
26152 249/300 Write block 2061 len 40960 data: 2 2 4 4 4 4 4 3 3 2
26153 250/300 Write block 4393 len 32768 data: 5 6 6 5 5 6 6 7
26154 251/300 Write block 1639 len 4096 data: 4
26155 252/300 Read block 2801 len 8192
26156 253/300 Write block 1122 len 24576 data: 6 6 5 4 4 4
26157 254/300 Read block 1028 len 8192
26158 255/300 Write block 4099 len 4096 data: 8
26159 256/300 Flush
26160 257/300 Flush
26161 258/300 Write block 1570 len 36864 data: 5 4 3 3 3 2 3 5 5
26162 259/300 Write block 1903 len 16384 data: 6 5 5 5
26163 260/300 Write block 922 len 40960 data: 7 7 6 5 6 5 4 4 3 4
26164 261/300 Read block 4139 len 8192
26165 262/300 Flush
26166 263/300 Write block 4921 len 28672 data: 8 7 4 5 5 4 3
26167 264/300 Read block 165 len 40960
26168 265/300 Read block 2679 len 28672
26169 266/300 Write block 3889 len 40960 data: 5 5 5 5 4 4 4 4 4 4
26170 267/300 Read block 369 len 12288
26171 268/300 Write block 2590 len 8192 data: 6 6
26172 269/300 Read block 2990 len 20480
26173 270/300 Read block 3450 len 16384
26174 271/300 Read block 2587 len 36864
26175 272/300 Write block 4279 len 40960 data: 7 7 6 4 4 4 4 4 2 3
26176 273/300 Read block 3382 len 16384
26177 274/300 Write block 1109 len 16384 data: 5 6 5 6
26178 275/300 Read block 3723 len 20480
26179 276/300 Read block 4432 len 16384
26180 277/300 Read block 3124 len 20480
26181 278/300 Flush
26182 279/300 Read block 3475 len 16384
26183 280/300 Write block 1221 len 8192 data: 5 4
26184 281/300 Read block 2488 len 40960
26185 282/300 Flush
26186 283/300 Read block 3962 len 24576
26187 284/300 Write block 4636 len 4096 data: 7
26188 285/300 Read block 442 len 40960
26189 286/300 Read block 630 len 12288
26190 287/300 Read block 695 len 8192
26191 288/300 Write block 3041 len 40960 data: 4 5 5 5 4 3 4 3 4 4
26192 289/300 Read block 667 len 32768
26193 290/300 Write block 4264 len 36864 data: 6 6 5 5 3 3 4 3 3
26194 291/300 Write block 3528 len 28672 data: 3 4 5 6 6 5 4
26195 292/300 Read block 4703 len 32768
26196 293/300 Write block 3440 len 8192 data: 4 5
26197 294/300 Read block 4741 len 20480
26198 295/300 Read block 3049 len 32768
26199 296/300 Read block 1955 len 4096
26200 297/300 Read block 1166 len 32768
26201 298/300 Read block 204 len 8192
26202 299/300 Read block 3146 len 24576
26203 300/300 Write block 713 len 40960 data: 6 5 5 4 3 3 3 4 6 6
26204 301/300 Flush
26205 Main task has work to do, go find it
26206 got dsc Start(1)
26207 start 1
26208 Feb 25 20:52:22.542 INFO request completed, latency_us: 257, response_code: 204, uri: /start/cid/1, method: POST, req_id: e33ea126-09b6-4563-a640-43e6aeca1e21, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
26209 Replay: started 1, returned:Ok(())
26210 ----------------------------------------------------------------
26211 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26212 States: Active Offline Active
26213 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26214 7641 Acked Write 7 Done Sent Done false
26215 7642 Acked Read 7 Done Sent Done false
26216 7643 Acked Read 8 Done Sent Done false
26217 7644 Acked Write 5 Done Sent Done false
26218 7645 Acked Write 3 Done Sent Done false
26219 7646 Acked Read 10 Done Sent Done false
26220 7647 Acked Write 3 Done Sent Done false
26221 7648 Acked Read 6 Done Sent Done false
26222 7649 Acked Read 8 Done Sent Done false
26223 7650 Acked Read 7 Done Sent Done false
26224 7651 Acked Read 10 Done Sent Done false
26225 7652 Acked Flush 0 Done Sent Done false
26226 7653 Acked Read 8 Done Sent Done false
26227 7654 Acked Write 1 Done Sent Done false
26228 7655 Acked Write 4 Done Sent Done false
26229 7656 Acked Write 3 Done Sent Done false
26230 7657 Acked Write 1 Done Sent Done false
26231 7658 Acked Read 9 Done Sent Done false
26232 7659 Acked Write 6 Done Sent Done false
26233 7660 Acked Read 4 Done Sent Done false
26234 7661 Acked Read 9 Done Sent Done false
26235 7662 Acked Read 8 Done Sent Done false
26236 7663 Acked Write 3 Done Sent Done false
26237 7664 Acked Read 7 Done Sent Done false
26238 7665 Acked Write 6 Done Sent Done false
26239 7666 Acked Write 10 Done Sent Done false
26240 7667 Acked Write 4 Done Sent Done false
26241 7668 Acked Write 3 Done Sent Done false
26242 7669 Acked Read 3 Done Sent Done false
26243 7670 Acked Write 1 Done Sent Done false
26244 7671 Acked Write 8 Done Sent Done false
26245 7672 Acked Read 1 Done Sent Done false
26246 7673 Acked Read 9 Done Sent Done false
26247 7674 Acked Read 7 Done Sent Done false
26248 7675 Acked Write 9 Done Sent Done false
26249 7676 Acked Read 8 Done Sent Done false
26250 7677 Acked Read 3 Done Sent Done false
26251 7678 Acked Write 3 Done Sent Done false
26252 7679 Acked Write 3 Done Sent Done false
26253 7680 Acked Read 10 Done Sent Done false
26254 7681 Acked Write 4 Done Sent Done false
26255 7682 Acked Write 2 Done Sent Done false
26256 7683 Acked Read 8 Done Sent Done false
26257 7684 Acked Read 2 Done Sent Done false
26258 7685 Acked Write 8 Done Sent Done false
26259 7686 Acked Write 8 Done Sent Done false
26260 7687 Acked Read 6 Done Sent Done false
26261 7688 Acked Read 4 Done Sent Done false
26262 7689 Acked Write 3 Done Sent Done false
26263 7690 Acked Read 1 Done Sent Done false
26264 7691 Acked Read 4 Done Sent Done false
26265 7692 Acked Write 4 Done Sent Done false
26266 7693 Acked Write 5 Done Sent Done false
26267 7694 Acked Write 6 Done Sent Done false
26268 7695 Acked Write 9 Done Sent Done false
26269 7696 Acked Read 3 Done Sent Done false
26270 7697 Acked Read 1 Done Sent Done false
26271 7698 Acked Flush 0 Done Sent Done false
26272 7699 Acked Write 7 Done Sent Done false
26273 7700 Acked Read 4 Done Sent Done false
26274 7701 Acked Write 10 Done Sent Done false
26275 7702 Acked Write 3 Done Sent Done false
26276 7703 Acked Read 8 Done Sent Done false
26277 7704 Acked Write 9 Done Sent Done false
26278 7705 Acked Flush 0 Done Sent Done false
26279 7706 Acked Read 9 Done Sent Done false
26280 7707 Acked Flush 0 Done Sent Done false
26281 7708 Acked Read 6 Done Sent Done false
26282 7709 Acked Flush 0 Done Sent Done false
26283 7710 Acked Write 9 Done Sent Done false
26284 7711 Acked Read 10 Done Sent Done false
26285 7712 Acked Write 1 Done Sent Done false
26286 7713 Acked Read 2 Done Sent Done false
26287 7714 Acked Write 4 Done Sent Done false
26288 7715 Acked Read 2 Done Sent Done false
26289 7716 Acked Read 10 Done Sent Done false
26290 7717 Acked Read 1 Done Sent Done false
26291 7718 Acked Read 10 Done Sent Done false
26292 7719 Acked Read 8 Done Sent Done false
26293 7720 Acked Write 9 Done Sent Done false
26294 7721 Acked Write 5 Done Sent Done false
26295 7722 Acked Read 4 Done Sent Done false
26296 7723 Acked Read 2 Done Sent Done false
26297 7724 Acked Read 5 Done Sent Done false
26298 7725 Acked Read 9 Done Sent Done false
26299 7726 Acked Read 8 Done Sent Done false
26300 7727 Acked Write 6 Done Sent Done false
26301 7728 Acked Write 1 Done Sent Done false
26302 7729 Acked Write 2 Done Sent Done false
26303 7730 Acked Read 10 Done Sent Done false
26304 7731 Acked Write 6 Done Sent Done false
26305 7732 Acked Read 5 Done Sent Done false
26306 7733 Acked Write 10 Done Sent Done false
26307 7734 Acked Flush 0 Done Sent Done false
26308 7735 Acked Read 10 Done Sent Done false
26309 7736 Acked Write 8 Done Sent Done false
26310 7737 Acked Write 8 Done Sent Done false
26311 7738 Acked Read 4 Done Sent Done false
26312 7739 Acked Read 3 Done Sent Done false
26313 7740 Acked Read 4 Done Sent Done false
26314 7741 Acked Flush 0 Done Sent Done false
26315 7742 Acked Write 9 Done Sent Done false
26316 7743 Acked Write 9 Done Sent Done false
26317 7744 Acked Write 8 Done Sent Done false
26318 7745 Acked Write 7 Done Sent Done false
26319 7746 Acked Write 2 Done Sent Done false
26320 7747 Acked Write 7 Done Sent Done false
26321 7748 Acked Read 9 Done Sent Done false
26322 7749 Acked Write 10 Done Sent Done false
26323 7750 Acked Read 5 Done Sent Done false
26324 7751 Acked Write 7 Done Sent Done false
26325 7752 Acked Write 10 Done Sent Done false
26326 7753 Acked Read 10 Done Sent Done false
26327 7754 Acked Write 5 Done Sent Done false
26328 7755 Acked Read 8 Done Sent Done false
26329 7756 Acked Read 6 Done Sent Done false
26330 7757 Acked Read 8 Done Sent Done false
26331 7758 Acked Read 4 Done Sent Done false
26332 7759 Acked Read 9 Done Sent Done false
26333 7760 Acked Write 10 Done Sent Done false
26334 7761 Acked Write 4 Done Sent Done false
26335 7762 Acked Read 7 Done Sent Done false
26336 7763 Acked Read 8 Done Sent Done false
26337 7764 Acked Flush 0 Done Sent Done false
26338 7765 Acked Read 1 Done Sent Done false
26339 7766 Acked Write 4 Done Sent Done false
26340 7767 Acked Read 7 Done Sent Done false
26341 7768 Acked Read 10 Done Sent Done false
26342 7769 Acked Write 4 Done Sent Done false
26343 7770 Acked Write 6 Done Sent Done false
26344 7771 Acked Write 9 Done Sent Done false
26345 7772 Acked Read 2 Done Sent Done false
26346 7773 Acked Read 8 Done Sent Done false
26347 7774 Acked Read 5 Done Sent Done false
26348 7775 Acked Read 5 Done Sent Done false
26349 7776 Acked Write 5 Done Sent Done false
26350 7777 Acked Read 6 Done Sent Done false
26351 7778 Acked Read 2 Done Sent Done false
26352 7779 Acked Write 8 Done Sent Done false
26353 7780 Acked Read 9 Done Sent Done false
26354 7781 Acked Write 3 Done Sent Done false
26355 7782 Acked Flush 0 Done Sent Done false
26356 7783 Acked Read 9 Done Sent Done false
26357 7784 Acked Write 8 Done Sent Done false
26358 7785 Acked Write 9 Done Sent Done false
26359 7786 Acked Read 4 Done Sent Done false
26360 7787 Acked Write 10 Done Sent Done false
26361 7788 Acked Read 7 Done Sent Done false
26362 7789 Acked Write 2 Done Sent Done false
26363 7790 Acked Read 2 Done Sent Done false
26364 7791 Acked Flush 0 Done Sent Done false
26365 7792 Acked Read 10 Done Sent Done false
26366 7793 Acked Flush 0 Done Sent Done false
26367 7794 Acked Write 1 Done Sent Done false
26368 7795 Acked Write 5 Done Sent Done false
26369 7796 Acked Read 1 Done Sent Done false
26370 7797 Acked Read 3 Done Sent Done false
26371 7798 Acked Write 5 Done Sent Done false
26372 7799 Acked Write 3 Done Sent Done false
26373 7800 Acked Write 8 Done Sent Done false
26374 7801 Acked Write 2 Done Sent Done false
26375 7802 Acked Read 5 Done Sent Done false
26376 7803 Acked Read 10 Done Sent Done false
26377 7804 Acked Read 10 Done Sent Done false
26378 7805 Acked Flush 0 Done Sent Done false
26379 7806 Acked Read 1 Done Sent Done false
26380 7807 Acked Read 7 Done Sent Done false
26381 7808 Acked Write 1 Done Sent Done false
26382 7809 Acked Write 9 Done Sent Done false
26383 7810 Acked Read 4 Done Sent Done false
26384 7811 Acked Read 10 Done Sent Done false
26385 7812 Acked Write 3 Done Sent Done false
26386 7813 Acked Read 4 Done Sent Done false
26387 7814 Acked Read 3 Done Sent Done false
26388 7815 Acked Write 6 Done Sent Done false
26389 7816 Acked Write 7 Done Sent Done false
26390 7817 Acked Write 3 Done Sent Done false
26391 7818 Acked Read 6 Done Sent Done false
26392 7819 Acked Write 2 Done Sent Done false
26393 7820 Acked Read 3 Done Sent Done false
26394 7821 Acked Read 7 Done Sent Done false
26395 7822 Acked Write 8 Done Sent Done false
26396 7823 Acked Write 9 Done Sent Done false
26397 7824 Acked Read 6 Done Sent Done false
26398 7825 Acked Flush 0 Done Sent Done false
26399 7826 Acked Write 8 Done Sent Done false
26400 7827 Acked Write 6 Done Sent Done false
26401 7828 Acked Read 4 Done Sent Done false
26402 7829 Acked Read 4 Done Sent Done false
26403 7830 Acked Flush 0 Done Sent Done false
26404 7831 Acked Write 4 Done Sent Done false
26405 7832 Acked Write 4 Done Sent Done false
26406 7833 Acked Write 10 Done Sent Done false
26407 7834 Acked Read 7 Done Sent Done false
26408 7835 Acked Read 5 Done Sent Done false
26409 7836 Acked Flush 0 Done Sent Done false
26410 7837 Acked Write 8 Done Sent Done false
26411 7838 Acked Read 2 Done Sent Done false
26412 7839 Acked Read 6 Done Sent Done false
26413 7840 Acked Read 9 Done Sent Done false
26414 7841 Acked Write 1 Done Sent Done false
26415 7842 Acked Read 4 Done Sent Done false
26416 7843 Acked Write 9 Done Sent Done false
26417 7844 Acked Flush 0 Done Sent Done false
26418 7845 Acked Write 2 Done Sent Done false
26419 7846 Acked Read 4 Done Sent Done false
26420 7847 Acked Write 8 Done Sent Done false
26421 7848 Acked Write 3 Done Sent Done false
26422 7849 Acked Write 1 Done Sent Done false
26423 7850 Acked Read 6 Done Sent Done false
26424 7851 Acked Write 7 Done Sent Done false
26425 7852 Acked Read 4 Done Sent Done false
26426 7853 Acked Read 1 Done Sent Done false
26427 7854 Acked Write 9 Done Sent Done false
26428 7855 Acked Read 4 Done Sent Done false
26429 7856 Acked Write 5 Done Sent Done false
26430 7857 Acked Read 8 Done Sent Done false
26431 7858 Acked Read 5 Done Sent Done false
26432 7859 Acked Write 3 Done Sent Done false
26433 7860 Acked Read 7 Done Sent Done false
26434 7861 Acked Read 2 Done Sent Done false
26435 7862 Acked Read 9 Done Sent Done false
26436 7863 Acked Write 6 Done Sent Done false
26437 7864 Acked Read 1 Done Sent Done false
26438 7865 Acked Write 10 Done Sent Done false
26439 7866 Acked Write 6 Done Sent Done false
26440 7867 Acked Read 10 Done Sent Done false
26441 7868 Acked Read 8 Done Sent Done false
26442 7869 Acked Write 3 Done Sent Done false
26443 7870 Acked Write 8 Done Sent Done false
26444 7871 Acked Read 2 Done Sent Done false
26445 7872 Acked Write 2 Done Sent Done false
26446 7873 Acked Read 4 Done Sent Done false
26447 7874 Acked Write 8 Done Sent Done false
26448 7875 Acked Write 1 Done Sent Done false
26449 7876 Acked Read 6 Done Sent Done false
26450 7877 Acked Read 7 Done Sent Done false
26451 7878 Acked Write 8 Done Sent Done false
26452 7879 Acked Write 4 Done Sent Done false
26453 7880 Acked Flush 0 Done Sent Done false
26454 7881 Acked Read 2 Done Sent Done false
26455 7882 Acked Read 2 Done Sent Done false
26456 7883 Acked Read 9 Done Sent Done false
26457 7884 Acked Write 4 Done Sent Done false
26458 7885 Acked Read 6 Done Sent Done false
26459 7886 Acked Write 4 Done Sent Done false
26460 7887 Acked Write 7 Done Sent Done false
26461 7888 Acked Write 8 Done Sent Done false
26462 7889 Acked Write 10 Done Sent Done false
26463 7890 Acked Write 8 Done Sent Done false
26464 7891 Acked Write 1 Done Sent Done false
26465 7892 Acked Read 2 Done Sent Done false
26466 7893 Acked Write 6 Done Sent Done false
26467 7894 Acked Read 2 Done Sent Done false
26468 7895 Acked Write 1 Done Sent Done false
26469 7896 Acked Flush 0 Done Sent Done false
26470 7897 Acked Flush 0 Done Sent Done false
26471 7898 Acked Write 9 Done Sent Done false
26472 7899 Acked Write 4 Done Sent Done false
26473 7900 Acked Write 10 Done Sent Done false
26474 7901 Acked Read 2 Done Sent Done false
26475 7902 Acked Flush 0 Done Sent Done false
26476 7903 Acked Write 7 Done Sent Done false
26477 7904 Acked Read 10 Done Sent Done false
26478 7905 Acked Read 7 Done Sent Done false
26479 7906 Acked Write 10 Done Sent Done false
26480 7907 Acked Read 3 Done Sent Done false
26481 7908 Acked Write 2 Done Sent Done false
26482 7909 Acked Read 5 Done Sent Done false
26483 7910 Acked Read 4 Done Sent Done false
26484 7911 Acked Read 9 Done Sent Done false
26485 7912 Acked Write 10 Done Sent Done false
26486 7913 Acked Read 4 Done Sent Done false
26487 7914 Acked Write 4 Done Sent Done false
26488 7915 Acked Read 5 Done Sent Done false
26489 7916 Acked Read 4 Done Sent Done false
26490 7917 Acked Read 5 Done Sent Done false
26491 7918 Acked Flush 0 Done Sent Done false
26492 7919 Acked Read 4 Done Sent Done false
26493 7920 Acked Write 2 Done Sent Done false
26494 7921 Acked Read 10 Done Sent Done false
26495 7922 Acked Flush 0 Done Sent Done false
26496 7923 Acked Read 6 Done Sent Done false
26497 7924 Acked Write 1 Done Sent Done false
26498 7925 Acked Read 10 Done Sent Done false
26499 7926 Acked Read 3 Done Sent Done false
26500 7927 Acked Read 2 Done Sent Done false
26501 7928 Acked Write 10 Done Sent Done false
26502 7929 Acked Read 8 Done Sent Done false
26503 7930 Acked Write 9 Done Sent Done false
26504 7931 Acked Write 7 Done Sent Done false
26505 7932 Acked Read 8 Done Sent Done false
26506 7933 Acked Write 2 Done Sent Done false
26507 7934 Acked Read 5 Done Sent Done false
26508 7935 Acked Read 8 Done Sent Done false
26509 7936 Acked Read 1 Done Sent Done false
26510 7937 Acked Read 8 Done Sent Done false
26511 7938 Acked Read 2 Done Sent Done false
26512 7939 Acked Read 6 Done Sent Done false
26513 7940 Acked Write 10 Done Sent Done false
26514 7941 Acked Flush 0 Done Sent Done false
26515 STATES DS:0 DS:1 DS:2 TOTAL
26516 Sent 0 301 0 301
26517 Done 301 0 301 602
26518 Skipped 0 0 0 0
26519 Error 0 0 0 0
26520 Last Flush: Some(JobId(7941)) Some(JobId(7640)) Some(JobId(7941))
26521 Downstairs last five completed: 7640 7639 7638 7637 7636
26522 Upstairs last five completed: 7941 7940 7939 7938 7937
26523 CLIENT: Up:0 ds:301 act:2
265242025-02-25T20:52:23.045ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
265252025-02-25T20:52:24.007ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
26526 [8820] got start action
26527 [8820] I am going to restart
26528 Starting downstairs at port 8820
26529 Make output file at "/tmp/dsc/downstairs-8820.txt"
26530 [8820][1] reports Starting
26531 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
26532 [8820][1] reports Running
265332025-02-25T20:52:24.610ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
265342025-02-25T20:52:25.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
26535 ----------------------------------------------------------------
26536 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26537 States: Active Offline Active
26538 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26539 7641 Acked Write 7 Done Sent Done false
26540 7642 Acked Read 7 Done Sent Done false
26541 7643 Acked Read 8 Done Sent Done false
26542 7644 Acked Write 5 Done Sent Done false
26543 7645 Acked Write 3 Done Sent Done false
26544 7646 Acked Read 10 Done Sent Done false
26545 7647 Acked Write 3 Done Sent Done false
26546 7648 Acked Read 6 Done Sent Done false
26547 7649 Acked Read 8 Done Sent Done false
26548 7650 Acked Read 7 Done Sent Done false
26549 7651 Acked Read 10 Done Sent Done false
26550 7652 Acked Flush 0 Done Sent Done false
26551 7653 Acked Read 8 Done Sent Done false
26552 7654 Acked Write 1 Done Sent Done false
26553 7655 Acked Write 4 Done Sent Done false
26554 7656 Acked Write 3 Done Sent Done false
26555 7657 Acked Write 1 Done Sent Done false
26556 7658 Acked Read 9 Done Sent Done false
26557 7659 Acked Write 6 Done Sent Done false
26558 7660 Acked Read 4 Done Sent Done false
26559 7661 Acked Read 9 Done Sent Done false
26560 7662 Acked Read 8 Done Sent Done false
26561 7663 Acked Write 3 Done Sent Done false
26562 7664 Acked Read 7 Done Sent Done false
26563 7665 Acked Write 6 Done Sent Done false
26564 7666 Acked Write 10 Done Sent Done false
26565 7667 Acked Write 4 Done Sent Done false
26566 7668 Acked Write 3 Done Sent Done false
26567 7669 Acked Read 3 Done Sent Done false
26568 7670 Acked Write 1 Done Sent Done false
26569 7671 Acked Write 8 Done Sent Done false
26570 7672 Acked Read 1 Done Sent Done false
26571 7673 Acked Read 9 Done Sent Done false
26572 7674 Acked Read 7 Done Sent Done false
26573 7675 Acked Write 9 Done Sent Done false
26574 7676 Acked Read 8 Done Sent Done false
26575 7677 Acked Read 3 Done Sent Done false
26576 7678 Acked Write 3 Done Sent Done false
26577 7679 Acked Write 3 Done Sent Done false
26578 7680 Acked Read 10 Done Sent Done false
26579 7681 Acked Write 4 Done Sent Done false
26580 7682 Acked Write 2 Done Sent Done false
26581 7683 Acked Read 8 Done Sent Done false
26582 7684 Acked Read 2 Done Sent Done false
26583 7685 Acked Write 8 Done Sent Done false
26584 7686 Acked Write 8 Done Sent Done false
26585 7687 Acked Read 6 Done Sent Done false
26586 7688 Acked Read 4 Done Sent Done false
26587 7689 Acked Write 3 Done Sent Done false
26588 7690 Acked Read 1 Done Sent Done false
26589 7691 Acked Read 4 Done Sent Done false
26590 7692 Acked Write 4 Done Sent Done false
26591 7693 Acked Write 5 Done Sent Done false
26592 7694 Acked Write 6 Done Sent Done false
26593 7695 Acked Write 9 Done Sent Done false
26594 7696 Acked Read 3 Done Sent Done false
26595 7697 Acked Read 1 Done Sent Done false
26596 7698 Acked Flush 0 Done Sent Done false
26597 7699 Acked Write 7 Done Sent Done false
26598 7700 Acked Read 4 Done Sent Done false
26599 7701 Acked Write 10 Done Sent Done false
26600 7702 Acked Write 3 Done Sent Done false
26601 7703 Acked Read 8 Done Sent Done false
26602 7704 Acked Write 9 Done Sent Done false
26603 7705 Acked Flush 0 Done Sent Done false
26604 7706 Acked Read 9 Done Sent Done false
26605 7707 Acked Flush 0 Done Sent Done false
26606 7708 Acked Read 6 Done Sent Done false
26607 7709 Acked Flush 0 Done Sent Done false
26608 7710 Acked Write 9 Done Sent Done false
26609 7711 Acked Read 10 Done Sent Done false
26610 7712 Acked Write 1 Done Sent Done false
26611 7713 Acked Read 2 Done Sent Done false
26612 7714 Acked Write 4 Done Sent Done false
26613 7715 Acked Read 2 Done Sent Done false
26614 7716 Acked Read 10 Done Sent Done false
26615 7717 Acked Read 1 Done Sent Done false
26616 7718 Acked Read 10 Done Sent Done false
26617 7719 Acked Read 8 Done Sent Done false
26618 7720 Acked Write 9 Done Sent Done false
26619 7721 Acked Write 5 Done Sent Done false
26620 7722 Acked Read 4 Done Sent Done false
26621 7723 Acked Read 2 Done Sent Done false
26622 7724 Acked Read 5 Done Sent Done false
26623 7725 Acked Read 9 Done Sent Done false
26624 7726 Acked Read 8 Done Sent Done false
26625 7727 Acked Write 6 Done Sent Done false
26626 7728 Acked Write 1 Done Sent Done false
26627 7729 Acked Write 2 Done Sent Done false
26628 7730 Acked Read 10 Done Sent Done false
26629 7731 Acked Write 6 Done Sent Done false
26630 7732 Acked Read 5 Done Sent Done false
26631 7733 Acked Write 10 Done Sent Done false
26632 7734 Acked Flush 0 Done Sent Done false
26633 7735 Acked Read 10 Done Sent Done false
26634 7736 Acked Write 8 Done Sent Done false
26635 7737 Acked Write 8 Done Sent Done false
26636 7738 Acked Read 4 Done Sent Done false
26637 7739 Acked Read 3 Done Sent Done false
26638 7740 Acked Read 4 Done Sent Done false
26639 7741 Acked Flush 0 Done Sent Done false
26640 7742 Acked Write 9 Done Sent Done false
26641 7743 Acked Write 9 Done Sent Done false
26642 7744 Acked Write 8 Done Sent Done false
26643 7745 Acked Write 7 Done Sent Done false
26644 7746 Acked Write 2 Done Sent Done false
26645 7747 Acked Write 7 Done Sent Done false
26646 7748 Acked Read 9 Done Sent Done false
26647 7749 Acked Write 10 Done Sent Done false
26648 7750 Acked Read 5 Done Sent Done false
26649 7751 Acked Write 7 Done Sent Done false
26650 7752 Acked Write 10 Done Sent Done false
26651 7753 Acked Read 10 Done Sent Done false
26652 7754 Acked Write 5 Done Sent Done false
26653 7755 Acked Read 8 Done Sent Done false
26654 7756 Acked Read 6 Done Sent Done false
26655 7757 Acked Read 8 Done Sent Done false
26656 7758 Acked Read 4 Done Sent Done false
26657 7759 Acked Read 9 Done Sent Done false
26658 7760 Acked Write 10 Done Sent Done false
26659 7761 Acked Write 4 Done Sent Done false
26660 7762 Acked Read 7 Done Sent Done false
26661 7763 Acked Read 8 Done Sent Done false
26662 7764 Acked Flush 0 Done Sent Done false
26663 7765 Acked Read 1 Done Sent Done false
26664 7766 Acked Write 4 Done Sent Done false
26665 7767 Acked Read 7 Done Sent Done false
26666 7768 Acked Read 10 Done Sent Done false
26667 7769 Acked Write 4 Done Sent Done false
26668 7770 Acked Write 6 Done Sent Done false
26669 7771 Acked Write 9 Done Sent Done false
26670 7772 Acked Read 2 Done Sent Done false
26671 7773 Acked Read 8 Done Sent Done false
26672 7774 Acked Read 5 Done Sent Done false
26673 7775 Acked Read 5 Done Sent Done false
26674 7776 Acked Write 5 Done Sent Done false
26675 7777 Acked Read 6 Done Sent Done false
26676 7778 Acked Read 2 Done Sent Done false
26677 7779 Acked Write 8 Done Sent Done false
26678 7780 Acked Read 9 Done Sent Done false
26679 7781 Acked Write 3 Done Sent Done false
26680 7782 Acked Flush 0 Done Sent Done false
26681 7783 Acked Read 9 Done Sent Done false
26682 7784 Acked Write 8 Done Sent Done false
26683 7785 Acked Write 9 Done Sent Done false
26684 7786 Acked Read 4 Done Sent Done false
26685 7787 Acked Write 10 Done Sent Done false
26686 7788 Acked Read 7 Done Sent Done false
26687 7789 Acked Write 2 Done Sent Done false
26688 7790 Acked Read 2 Done Sent Done false
26689 7791 Acked Flush 0 Done Sent Done false
26690 7792 Acked Read 10 Done Sent Done false
26691 7793 Acked Flush 0 Done Sent Done false
26692 7794 Acked Write 1 Done Sent Done false
26693 7795 Acked Write 5 Done Sent Done false
26694 7796 Acked Read 1 Done Sent Done false
26695 7797 Acked Read 3 Done Sent Done false
26696 7798 Acked Write 5 Done Sent Done false
26697 7799 Acked Write 3 Done Sent Done false
26698 7800 Acked Write 8 Done Sent Done false
26699 7801 Acked Write 2 Done Sent Done false
26700 7802 Acked Read 5 Done Sent Done false
26701 7803 Acked Read 10 Done Sent Done false
26702 7804 Acked Read 10 Done Sent Done false
26703 7805 Acked Flush 0 Done Sent Done false
26704 7806 Acked Read 1 Done Sent Done false
26705 7807 Acked Read 7 Done Sent Done false
26706 7808 Acked Write 1 Done Sent Done false
26707 7809 Acked Write 9 Done Sent Done false
26708 7810 Acked Read 4 Done Sent Done false
26709 7811 Acked Read 10 Done Sent Done false
26710 7812 Acked Write 3 Done Sent Done false
26711 7813 Acked Read 4 Done Sent Done false
26712 7814 Acked Read 3 Done Sent Done false
26713 7815 Acked Write 6 Done Sent Done false
26714 7816 Acked Write 7 Done Sent Done false
26715 7817 Acked Write 3 Done Sent Done false
26716 7818 Acked Read 6 Done Sent Done false
26717 7819 Acked Write 2 Done Sent Done false
26718 7820 Acked Read 3 Done Sent Done false
26719 7821 Acked Read 7 Done Sent Done false
26720 7822 Acked Write 8 Done Sent Done false
26721 7823 Acked Write 9 Done Sent Done false
26722 7824 Acked Read 6 Done Sent Done false
26723 7825 Acked Flush 0 Done Sent Done false
26724 7826 Acked Write 8 Done Sent Done false
26725 7827 Acked Write 6 Done Sent Done false
26726 7828 Acked Read 4 Done Sent Done false
26727 7829 Acked Read 4 Done Sent Done false
26728 7830 Acked Flush 0 Done Sent Done false
26729 7831 Acked Write 4 Done Sent Done false
26730 7832 Acked Write 4 Done Sent Done false
26731 7833 Acked Write 10 Done Sent Done false
26732 7834 Acked Read 7 Done Sent Done false
26733 7835 Acked Read 5 Done Sent Done false
26734 7836 Acked Flush 0 Done Sent Done false
26735 7837 Acked Write 8 Done Sent Done false
26736 7838 Acked Read 2 Done Sent Done false
26737 7839 Acked Read 6 Done Sent Done false
26738 7840 Acked Read 9 Done Sent Done false
26739 7841 Acked Write 1 Done Sent Done false
26740 7842 Acked Read 4 Done Sent Done false
26741 7843 Acked Write 9 Done Sent Done false
26742 7844 Acked Flush 0 Done Sent Done false
26743 7845 Acked Write 2 Done Sent Done false
26744 7846 Acked Read 4 Done Sent Done false
26745 7847 Acked Write 8 Done Sent Done false
26746 7848 Acked Write 3 Done Sent Done false
26747 7849 Acked Write 1 Done Sent Done false
26748 7850 Acked Read 6 Done Sent Done false
26749 7851 Acked Write 7 Done Sent Done false
26750 7852 Acked Read 4 Done Sent Done false
26751 7853 Acked Read 1 Done Sent Done false
26752 7854 Acked Write 9 Done Sent Done false
26753 7855 Acked Read 4 Done Sent Done false
26754 7856 Acked Write 5 Done Sent Done false
26755 7857 Acked Read 8 Done Sent Done false
26756 7858 Acked Read 5 Done Sent Done false
26757 7859 Acked Write 3 Done Sent Done false
26758 7860 Acked Read 7 Done Sent Done false
26759 7861 Acked Read 2 Done Sent Done false
26760 7862 Acked Read 9 Done Sent Done false
26761 7863 Acked Write 6 Done Sent Done false
26762 7864 Acked Read 1 Done Sent Done false
26763 7865 Acked Write 10 Done Sent Done false
26764 7866 Acked Write 6 Done Sent Done false
26765 7867 Acked Read 10 Done Sent Done false
26766 7868 Acked Read 8 Done Sent Done false
26767 7869 Acked Write 3 Done Sent Done false
26768 7870 Acked Write 8 Done Sent Done false
26769 7871 Acked Read 2 Done Sent Done false
26770 7872 Acked Write 2 Done Sent Done false
26771 7873 Acked Read 4 Done Sent Done false
26772 7874 Acked Write 8 Done Sent Done false
26773 7875 Acked Write 1 Done Sent Done false
26774 7876 Acked Read 6 Done Sent Done false
26775 7877 Acked Read 7 Done Sent Done false
26776 7878 Acked Write 8 Done Sent Done false
26777 7879 Acked Write 4 Done Sent Done false
26778 7880 Acked Flush 0 Done Sent Done false
26779 7881 Acked Read 2 Done Sent Done false
26780 7882 Acked Read 2 Done Sent Done false
26781 7883 Acked Read 9 Done Sent Done false
26782 7884 Acked Write 4 Done Sent Done false
26783 7885 Acked Read 6 Done Sent Done false
26784 7886 Acked Write 4 Done Sent Done false
26785 7887 Acked Write 7 Done Sent Done false
26786 7888 Acked Write 8 Done Sent Done false
26787 7889 Acked Write 10 Done Sent Done false
26788 7890 Acked Write 8 Done Sent Done false
26789 7891 Acked Write 1 Done Sent Done false
26790 7892 Acked Read 2 Done Sent Done false
26791 7893 Acked Write 6 Done Sent Done false
26792 7894 Acked Read 2 Done Sent Done false
26793 7895 Acked Write 1 Done Sent Done false
26794 7896 Acked Flush 0 Done Sent Done false
26795 7897 Acked Flush 0 Done Sent Done false
26796 7898 Acked Write 9 Done Sent Done false
26797 7899 Acked Write 4 Done Sent Done false
26798 7900 Acked Write 10 Done Sent Done false
26799 7901 Acked Read 2 Done Sent Done false
26800 7902 Acked Flush 0 Done Sent Done false
26801 7903 Acked Write 7 Done Sent Done false
26802 7904 Acked Read 10 Done Sent Done false
26803 7905 Acked Read 7 Done Sent Done false
26804 7906 Acked Write 10 Done Sent Done false
26805 7907 Acked Read 3 Done Sent Done false
26806 7908 Acked Write 2 Done Sent Done false
26807 7909 Acked Read 5 Done Sent Done false
26808 7910 Acked Read 4 Done Sent Done false
26809 7911 Acked Read 9 Done Sent Done false
26810 7912 Acked Write 10 Done Sent Done false
26811 7913 Acked Read 4 Done Sent Done false
26812 7914 Acked Write 4 Done Sent Done false
26813 7915 Acked Read 5 Done Sent Done false
26814 7916 Acked Read 4 Done Sent Done false
26815 7917 Acked Read 5 Done Sent Done false
26816 7918 Acked Flush 0 Done Sent Done false
26817 7919 Acked Read 4 Done Sent Done false
26818 7920 Acked Write 2 Done Sent Done false
26819 7921 Acked Read 10 Done Sent Done false
26820 7922 Acked Flush 0 Done Sent Done false
26821 7923 Acked Read 6 Done Sent Done false
26822 7924 Acked Write 1 Done Sent Done false
26823 7925 Acked Read 10 Done Sent Done false
26824 7926 Acked Read 3 Done Sent Done false
26825 7927 Acked Read 2 Done Sent Done false
26826 7928 Acked Write 10 Done Sent Done false
26827 7929 Acked Read 8 Done Sent Done false
26828 7930 Acked Write 9 Done Sent Done false
26829 7931 Acked Write 7 Done Sent Done false
26830 7932 Acked Read 8 Done Sent Done false
26831 7933 Acked Write 2 Done Sent Done false
26832 7934 Acked Read 5 Done Sent Done false
26833 7935 Acked Read 8 Done Sent Done false
26834 7936 Acked Read 1 Done Sent Done false
26835 7937 Acked Read 8 Done Sent Done false
26836 7938 Acked Read 2 Done Sent Done false
26837 7939 Acked Read 6 Done Sent Done false
26838 7940 Acked Write 10 Done Sent Done false
26839 7941 Acked Flush 0 Done Sent Done false
26840 STATES DS:0 DS:1 DS:2 TOTAL
26841 Sent 0 301 0 301
26842 Done 301 0 301 602
26843 Skipped 0 0 0 0
26844 Error 0 0 0 0
26845 Last Flush: Some(JobId(7941)) Some(JobId(7640)) Some(JobId(7941))
26846 Downstairs last five completed: 7640 7639 7638 7637 7636
26847 Upstairs last five completed: 7941 7940 7939 7938 7937
26848 CLIENT: Up:0 ds:301 act:2
268492025-02-25T20:52:27.047ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268502025-02-25T20:52:27.614ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268512025-02-25T20:52:28.291ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268522025-02-25T20:52:28.292ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49597) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268532025-02-25T20:52:28.293ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268542025-02-25T20:52:28.293ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268552025-02-25T20:52:28.293ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268562025-02-25T20:52:28.294ZINFOcrucible: send last flush ID to this DS: Some(JobId(7640)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268572025-02-25T20:52:28.294ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268582025-02-25T20:52:28.294ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7640)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268592025-02-25T20:52:28.294ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7640)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
26860 ----------------------------------------------------------------
26861 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26862 Downstairs last five completed: 7941 7940 7939 7938 7937
26863 Upstairs last five completed: 7941 7940 7939 7938 7937
26864 CLIENT: Up:0 ds:0 act:3
26865 Replay: All jobs finished
26866 Main task has work to do, go find it
26867 got dsc Stop(2)
26868 stop 2
26869 Feb 25 20:52:30.549 INFO[8830] Got stop action so:false kr:false
26870 request completed, latency_us: 384, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c217fc03-09a7-4bdb-90d3-9dc083839a4c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
26871 Feb 25 20:52:30.550 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:52:30.550274959Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
26872 GET, req_id: 75cd0f11-0134-4610-b175-e89175aea16d, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
268732025-02-25T20:52:30.550ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268742025-02-25T20:52:30.550ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
268752025-02-25T20:52:30.550ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
26876 [8830] Exited with: ExitStatus(unix_wait_status(9))
26877 [8830][2] reports Exit
26878 Feb 25 20:52:34.552 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 155547c8-61de-4cf2-a056-e30dbd3c87bb, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
26879 001/300 Write block 3265 len 4096 data: 5
26880 002/300 Read block 2287 len 16384
26881 003/300 Read block 3374 len 12288
26882 004/300 Write block 1906 len 8192 data: 6 4
26883 005/300 Read block 1561 len 16384
26884 006/300 Flush
26885 007/300 Read block 2833 len 12288
26886 008/300 Write block 3515 len 28672 data: 5 5 5 5 4 4 3
26887 009/300 Read block 4919 len 28672
26888 010/300 Read block 4403 len 12288
26889 011/300 Flush
26890 012/300 Read block 3840 len 4096
26891 013/300 Read block 2155 len 16384
26892 014/300 Read block 2592 len 16384
26893 015/300 Read block 1127 len 16384
26894 016/300 Write block 3891 len 40960 data: 6 6 5 5 5 5 5 5 5 4
26895 017/300 Read block 3233 len 20480
26896 018/300 Write block 1424 len 4096 data: 5
26897 019/300 Write block 4168 len 36864 data: 9 10 9 7 7 10 10 7 6
26898 020/300 Write block 2218 len 4096 data: 4
26899 021/300 Read block 4794 len 28672
26900 022/300 Read block 1028 len 12288
26901 023/300 Flush
26902 024/300 Write block 3232 len 4096 data: 3
26903 025/300 Read block 2790 len 8192
26904 026/300 Read block 4019 len 12288
26905 027/300 Read block 4905 len 20480
26906 028/300 Read block 531 len 36864
26907 029/300 Write block 1207 len 32768 data: 3 5 5 6 6 6 5 5
26908 030/300 Write block 3871 len 4096 data: 7
26909 031/300 Read block 3352 len 36864
26910 032/300 Flush
26911 033/300 Write block 895 len 28672 data: 3 4 5 5 5 5 5
26912 034/300 Write block 1240 len 12288 data: 4 3 4
26913 035/300 Read block 3990 len 16384
26914 036/300 Write block 637 len 32768 data: 3 3 2 3 3 3 3 3
26915 037/300 Flush
26916 038/300 Read block 2115 len 24576
26917 039/300 Write block 264 len 20480 data: 4 4 5 5 7
26918 040/300 Read block 489 len 40960
26919 041/300 Read block 2809 len 36864
26920 042/300 Read block 2718 len 36864
26921 043/300 Write block 3486 len 32768 data: 5 5 6 6 6 5 6 7
26922 044/300 Read block 1921 len 20480
26923 045/300 Write block 1155 len 32768 data: 3 4 5 5 5 4 3 3
26924 046/300 Read block 190 len 12288
26925 047/300 Read block 278 len 8192
26926 048/300 Read block 4823 len 40960
26927 049/300 Flush
26928 050/300 Read block 2961 len 24576
26929 051/300 Read block 3739 len 20480
26930 052/300 Read block 4367 len 40960
26931 053/300 Write block 137 len 24576 data: 4 4 4 4 4 4
26932 054/300 Read block 3259 len 36864
26933 055/300 Write block 937 len 4096 data: 4
26934 056/300 Read block 563 len 16384
26935 057/300 Read block 4360 len 40960
26936 058/300 Write block 684 len 4096 data: 4
26937 059/300 Read block 3421 len 8192
26938 060/300 Write block 4296 len 36864 data: 6 5 5 5 7 6 6 5 4
26939 061/300 Read block 4196 len 8192
26940 062/300 Write block 1173 len 40960 data: 3 4 4 4 3 3 3 2 3 5
26941 063/300 Read block 1831 len 32768
26942 064/300 Read block 3602 len 28672
26943 065/300 Write block 3617 len 32768 data: 7 6 7 5 5 5 5 4
26944 066/300 Read block 3345 len 16384
26945 067/300 Write block 2667 len 8192 data: 5 6
26946 068/300 Write block 2133 len 24576 data: 3 3 3 3 3 3
26947 069/300 Flush
26948 070/300 Write block 3644 len 24576 data: 8 6 8 5 5 6
26949 071/300 Read block 2157 len 4096
26950 072/300 Write block 4687 len 32768 data: 5 5 6 8 8 7 6 7
26951 073/300 Read block 1175 len 16384
26952 074/300 Write block 2843 len 40960 data: 5 6 7 5 5 5 4 4 4 4
26953 075/300 Read block 2577 len 32768
26954 076/300 Read block 1079 len 40960
26955 077/300 Read block 309 len 28672
26956 078/300 Read block 3646 len 32768
26957 079/300 Write block 4043 len 20480 data: 3 3 4 5 5
26958 080/300 Flush
26959 081/300 Write block 2160 len 40960 data: 11 9 8 7 7 7 7 7 6 5
26960 082/300 Flush
26961 083/300 Flush
26962 084/300 Write block 744 len 40960 data: 2 4 4 4 3 3 3 2 2 2
26963 085/300 Read block 2874 len 36864
26964 086/300 Read block 172 len 12288
26965 087/300 Read block 3970 len 28672
26966 088/300 Write block 673 len 20480 data: 8 10 12 9 6
26967 089/300 Read block 1254 len 36864
26968 090/300 Read block 4381 len 20480
26969 091/300 Write block 4906 len 28672 data: 5 4 4 4 3 3 3
26970 092/300 Write block 3339 len 20480 data: 4 4 3 3 4
26971 093/300 Read block 3842 len 28672
26972 094/300 Read block 661 len 32768
26973 095/300 Read block 1950 len 40960
26974 096/300 Read block 4534 len 16384
26975 097/300 Write block 1069 len 16384 data: 4 4 5 6
26976 098/300 Write block 4017 len 36864 data: 4 4 3 3 4 5 6 5 5
26977 099/300 Read block 3668 len 20480
26978 100/300 Read block 3623 len 4096
26979 101/300 Read block 3625 len 20480
26980 102/300 Write block 941 len 8192 data: 2 4
26981 103/300 Read block 4488 len 4096
26982 104/300 Flush
26983 105/300 Read block 608 len 40960
26984 106/300 Read block 3369 len 8192
26985 107/300 Write block 2638 len 8192 data: 5 4
26986 108/300 Write block 1022 len 20480 data: 7 7 9 9 7
26987 109/300 Read block 3843 len 4096
26988 110/300 Write block 4614 len 40960 data: 5 5 5 6 5 4 4 3 3 2
26989 111/300 Write block 4968 len 36864 data: 6 5 5 5 4 4 5 5 4
26990 112/300 Read block 4379 len 28672
26991 113/300 Write block 4179 len 40960 data: 4 5 4 4 4 3 4 3 3 2
26992 114/300 Read block 3881 len 20480
26993 115/300 Read block 4657 len 20480
26994 116/300 Read block 2441 len 20480
26995 117/300 Write block 150 len 36864 data: 5 5 5 5 4 4 3 3 3
26996 118/300 Write block 1078 len 24576 data: 3 3 2 2 2 2
26997 119/300 Read block 2019 len 8192
26998 120/300 Flush
26999 121/300 Read block 496 len 28672
27000 122/300 Read block 2069 len 8192
27001 123/300 Write block 3263 len 24576 data: 3 4 6 6 6 7
27002 124/300 Write block 2554 len 8192 data: 4 5
27003 125/300 Write block 2552 len 36864 data: 6 6 5 6 5 5 4 3 2
27004 126/300 Read block 4135 len 8192
27005 127/300 Read block 3803 len 4096
27006 128/300 Flush
27007 129/300 Write block 3137 len 20480 data: 7 5 6 7 6
27008 130/300 Write block 3349 len 20480 data: 8 8 5 7 6
27009 131/300 Read block 2774 len 36864
27010 132/300 Read block 1459 len 36864
27011 133/300 Read block 3707 len 16384
27012 134/300 Flush
27013 135/300 Write block 2552 len 12288 data: 7 7 6
27014 136/300 Write block 3303 len 4096 data: 10
27015 137/300 Write block 1657 len 40960 data: 8 8 6 6 5 6 5 4 3 3
27016 138/300 Read block 60 len 8192
27017 139/300 Read block 4228 len 16384
27018 140/300 Read block 1217 len 16384
27019 141/300 Write block 2282 len 32768 data: 4 5 5 6 5 5 4 4
27020 142/300 Write block 1761 len 28672 data: 4 4 4 4 4 3 3
27021 143/300 Write block 4551 len 16384 data: 3 2 2 3
27022 144/300 Read block 4597 len 16384
27023 145/300 Read block 2390 len 8192
27024 146/300 Write block 4060 len 16384 data: 4 3 3 4
27025 147/300 Write block 1963 len 12288 data: 4 5 5
27026 148/300 Read block 1038 len 36864
27027 149/300 Write block 59 len 36864 data: 4 5 6 5 5 5 6 4 3
27028 150/300 Read block 2861 len 8192
27029 151/300 Read block 144 len 4096
27030 152/300 Read block 610 len 24576
27031 153/300 Read block 436 len 36864
27032 154/300 Read block 1678 len 24576
27033 155/300 Write block 1899 len 36864 data: 5 5 5 6 7 6 6 7 5
27034 156/300 Write block 956 len 36864 data: 5 7 9 7 9 7 6 5 5
27035 157/300 Write block 2986 len 20480 data: 9 10 12 8 8
27036 158/300 Read block 3447 len 32768
27037 159/300 Write block 3620 len 8192 data: 6 6
27038 160/300 Read block 1273 len 8192
27039 161/300 Flush
27040 162/300 Read block 3565 len 16384
27041 163/300 Write block 2131 len 24576 data: 3 3 4 4 4 4
27042 164/300 Read block 423 len 32768
27043 165/300 Write block 1668 len 28672 data: 4 6 6 6 6 6 5
27044 166/300 Write block 2283 len 20480 data: 6 6 7 6 6
27045 167/300 Write block 4244 len 12288 data: 4 5 6
27046 168/300 Write block 4815 len 28672 data: 4 3 5 5 5 6 7
27047 169/300 Read block 1555 len 4096
27048 170/300 Write block 3373 len 4096 data: 5
27049 171/300 Write block 4638 len 36864 data: 4 5 5 3 3 4 3 3 3
27050 172/300 Flush
27051 173/300 Write block 1302 len 12288 data: 3 4 5
27052 174/300 Write block 1849 len 32768 data: 6 6 7 5 5 4 6 4
27053 175/300 Read block 817 len 40960
27054 176/300 Write block 3701 len 20480 data: 7 7 5 6 5
27055 177/300 Read block 3118 len 16384
27056 178/300 Read block 401 len 8192
27057 179/300 Read block 634 len 24576
27058 180/300 Read block 1488 len 20480
27059 181/300 Flush
27060 182/300 Write block 4289 len 36864 data: 3 3 4 4 4 7 5 7 6
27061 183/300 Read block 4141 len 40960
27062 184/300 Flush
27063 185/300 Read block 2904 len 4096
27064 186/300 Read block 3227 len 8192
27065 187/300 Flush
27066 188/300 Write block 3557 len 16384 data: 6 5 5 5
27067 189/300 Write block 3173 len 32768 data: 5 6 6 5 6 5 6 7
27068 190/300 Write block 3964 len 28672 data: 4 3 3 3 5 5 5
27069 191/300 Read block 1574 len 24576
27070 192/300 Read block 145 len 16384
27071 193/300 Flush
27072 194/300 Read block 1036 len 12288
27073 195/300 Read block 4535 len 36864
27074 196/300 Read block 4558 len 12288
27075 197/300 Read block 1103 len 32768
27076 198/300 Write block 1701 len 36864 data: 4 4 4 3 3 3 3 4 4
27077 199/300 Flush
27078 200/300 Write block 1755 len 16384 data: 2 2 2 2
27079 201/300 Read block 2507 len 36864
27080 202/300 Read block 4924 len 16384
27081 203/300 Flush
27082 204/300 Read block 2921 len 16384
27083 205/300 Read block 3105 len 16384
27084 206/300 Write block 4665 len 16384 data: 6 4 4 4
27085 207/300 Write block 3710 len 20480 data: 6 8 8 7 8
27086 208/300 Read block 2978 len 4096
27087 209/300 Read block 1618 len 28672
27088 210/300 Read block 2997 len 28672
27089 211/300 Write block 522 len 16384 data: 2 2 3 4
27090 212/300 Write block 4186 len 40960 data: 4 4 3 2 3 4 4 4 4 5
27091 213/300 Flush
27092 214/300 Write block 4323 len 40960 data: 4 4 4 5 5 4 4 3 4 5
27093 215/300 Read block 2606 len 8192
27094 216/300 Write block 4771 len 40960 data: 7 5 6 6 6 6 7 3 3 3
27095 217/300 Read block 4390 len 28672
27096 218/300 Read block 2447 len 24576
27097 219/300 Flush
27098 220/300 Write block 3475 len 32768 data: 3 5 6 5 5 5 5 5
27099 221/300 Write block 4366 len 36864 data: 5 5 8 6 5 4 4 4 4
27100 222/300 Write block 3558 len 4096 data: 6
27101 223/300 Read block 4538 len 12288
27102 224/300 Read block 2624 len 28672
27103 225/300 Read block 178 len 16384
27104 226/300 Write block 4486 len 24576 data: 6 5 4 3 4 3
27105 227/300 Read block 742 len 24576
27106 228/300 Read block 497 len 20480
27107 229/300 Write block 2212 len 36864 data: 3 3 2 4 4 5 5 5 6
27108 230/300 Read block 1844 len 12288
27109 231/300 Read block 1679 len 8192
27110 232/300 Write block 81 len 28672 data: 7 5 5 3 3 2 5
27111 233/300 Flush
27112 234/300 Read block 4312 len 32768
27113 235/300 Write block 3241 len 36864 data: 6 9 8 8 6 7 5 4 4
27114 236/300 Read block 609 len 12288
27115 237/300 Read block 4775 len 28672
27116 238/300 Write block 1444 len 36864 data: 3 3 3 3 3 3 3 3 3
27117 239/300 Read block 1599 len 12288
27118 240/300 Write block 2815 len 16384 data: 4 3 3 2
27119 241/300 Read block 2047 len 8192
27120 242/300 Read block 2649 len 12288
27121 243/300 Write block 2471 len 28672 data: 9 8 7 6 6 6 5
27122 244/300 Write block 1841 len 16384 data: 2 2 3 3
27123 245/300 Flush
27124 246/300 Read block 911 len 24576
27125 247/300 Read block 4088 len 16384
27126 248/300 Read block 4579 len 36864
27127 249/300 Write block 1784 len 16384 data: 6 6 8 7
27128 250/300 Read block 2835 len 12288
27129 251/300 Write block 386 len 36864 data: 5 5 4 5 5 5 4 4 4
27130 252/300 Read block 3679 len 24576
27131 253/300 Write block 689 len 16384 data: 2 4 6 5
27132 254/300 Write block 813 len 40960 data: 6 6 7 6 5 5 5 6 4 5
27133 255/300 Write block 2161 len 20480 data: 10 9 8 8 8
27134 256/300 Read block 1787 len 28672
27135 257/300 Write block 796 len 8192 data: 4 4
27136 258/300 Write block 276 len 16384 data: 6 6 5 6
27137 259/300 Write block 209 len 36864 data: 4 3 4 3 3 5 7 7 5
27138 260/300 Read block 228 len 20480
27139 261/300 Read block 527 len 24576
27140 262/300 Read block 59 len 20480
27141 263/300 Read block 4711 len 4096
27142 264/300 Write block 1470 len 16384 data: 4 5 5 5
27143 265/300 Read block 4094 len 8192
27144 266/300 Read block 3106 len 36864
27145 267/300 Flush
27146 268/300 Write block 636 len 40960 data: 3 4 4 3 4 4 4 4 4 2
27147 269/300 Read block 498 len 28672
27148 270/300 Read block 1573 len 40960
27149 271/300 Read block 2675 len 4096
27150 272/300 Flush
27151 273/300 Write block 3460 len 40960 data: 2 3 3 3 3 4 4 5 5 6
27152 274/300 Read block 1268 len 40960
27153 275/300 Write block 3852 len 16384 data: 7 8 9 8
27154 276/300 Read block 1777 len 4096
27155 277/300 Write block 2605 len 12288 data: 4 8 8
27156 278/300 Read block 1320 len 36864
27157 279/300 Read block 3695 len 20480
27158 280/300 Flush
27159 281/300 Read block 675 len 4096
27160 282/300 Flush
27161 283/300 Write block 1952 len 24576 data: 5 6 7 6 6 7
27162 284/300 Write block 1008 len 20480 data: 6 6 6 6 7
27163 285/300 Read block 280 len 40960
27164 286/300 Write block 2845 len 16384 data: 8 6 6 6
27165 287/300 Flush
27166 288/300 Write block 3458 len 16384 data: 3 3 3 4
27167 289/300 Write block 941 len 40960 data: 3 5 4 4 5 5 5 6 4 3
27168 290/300 Write block 742 len 4096 data: 3
27169 291/300 Read block 570 len 32768
27170 292/300 Write block 1374 len 16384 data: 4 4 3 3
27171 293/300 Write block 1953 len 32768 data: 7 8 7 7 8 7 7 6
27172 294/300 Write block 855 len 16384 data: 6 7 5 6
27173 295/300 Read block 554 len 8192
27174 296/300 Write block 120 len 28672 data: 4 4 3 4 6 7 7
27175 297/300 Read block 403 len 12288
27176 298/300 Flush
27177 299/300 Write block 3012 len 8192 data: 8 7
27178 300/300 Flush
27179 301/300 Flush
27180 Main task has work to do, go find it
27181 got dsc Start(2)
27182 start 2
27183 Feb 25 20:52:34.801 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/2, method: POST, req_id: bc0ba69a-f7e4-4cb6-84c7-ebff386af567, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
27184 Replay: started 2, returned:Ok(())
27185 ----------------------------------------------------------------
27186 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27187 States: Active Active Offline
27188 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27189 7942 Acked Write 1 Done Done Sent false
27190 7943 Acked Read 4 Done Done Sent false
27191 7944 Acked Read 3 Done Done Sent false
27192 7945 Acked Write 2 Done Done Sent false
27193 7946 Acked Read 4 Done Done Sent false
27194 7947 Acked Flush 0 Done Done Sent false
27195 7948 Acked Read 3 Done Done Sent false
27196 7949 Acked Write 7 Done Done Sent false
27197 7950 Acked Read 7 Done Done Sent false
27198 7951 Acked Read 3 Done Done Sent false
27199 7952 Acked Flush 0 Done Done Sent false
27200 7953 Acked Read 1 Done Done Sent false
27201 7954 Acked Read 4 Done Done Sent false
27202 7955 Acked Read 4 Done Done Sent false
27203 7956 Acked Read 4 Done Done Sent false
27204 7957 Acked Write 10 Done Done Sent false
27205 7958 Acked Read 5 Done Done Sent false
27206 7959 Acked Write 1 Done Done Sent false
27207 7960 Acked Write 9 Done Done Sent false
27208 7961 Acked Write 1 Done Done Sent false
27209 7962 Acked Read 7 Done Done Sent false
27210 7963 Acked Read 3 Done Done Sent false
27211 7964 Acked Flush 0 Done Done Sent false
27212 7965 Acked Write 1 Done Done Sent false
27213 7966 Acked Read 2 Done Done Sent false
27214 7967 Acked Read 3 Done Done Sent false
27215 7968 Acked Read 5 Done Done Sent false
27216 7969 Acked Read 9 Done Done Sent false
27217 7970 Acked Write 8 Done Done Sent false
27218 7971 Acked Write 1 Done Done Sent false
27219 7972 Acked Read 9 Done Done Sent false
27220 7973 Acked Flush 0 Done Done Sent false
27221 7974 Acked Write 7 Done Done Sent false
27222 7975 Acked Write 3 Done Done Sent false
27223 7976 Acked Read 4 Done Done Sent false
27224 7977 Acked Write 8 Done Done Sent false
27225 7978 Acked Flush 0 Done Done Sent false
27226 7979 Acked Read 6 Done Done Sent false
27227 7980 Acked Write 5 Done Done Sent false
27228 7981 Acked Read 10 Done Done Sent false
27229 7982 Acked Read 9 Done Done Sent false
27230 7983 Acked Read 9 Done Done Sent false
27231 7984 Acked Write 8 Done Done Sent false
27232 7985 Acked Read 5 Done Done Sent false
27233 7986 Acked Write 8 Done Done Sent false
27234 7987 Acked Read 3 Done Done Sent false
27235 7988 Acked Read 2 Done Done Sent false
27236 7989 Acked Read 10 Done Done Sent false
27237 7990 Acked Flush 0 Done Done Sent false
27238 7991 Acked Read 6 Done Done Sent false
27239 7992 Acked Read 5 Done Done Sent false
27240 7993 Acked Read 10 Done Done Sent false
27241 7994 Acked Write 6 Done Done Sent false
27242 7995 Acked Read 9 Done Done Sent false
27243 7996 Acked Write 1 Done Done Sent false
27244 7997 Acked Read 4 Done Done Sent false
27245 7998 Acked Read 10 Done Done Sent false
27246 7999 Acked Write 1 Done Done Sent false
27247 8000 Acked Read 2 Done Done Sent false
27248 8001 Acked Write 9 Done Done Sent false
27249 8002 Acked Read 2 Done Done Sent false
27250 8003 Acked Write 10 Done Done Sent false
27251 8004 Acked Read 8 Done Done Sent false
27252 8005 Acked Read 7 Done Done Sent false
27253 8006 Acked Write 8 Done Done Sent false
27254 8007 Acked Read 4 Done Done Sent false
27255 8008 Acked Write 2 Done Done Sent false
27256 8009 Acked Write 6 Done Done Sent false
27257 8010 Acked Flush 0 Done Done Sent false
27258 8011 Acked Write 6 Done Done Sent false
27259 8012 Acked Read 1 Done Done Sent false
27260 8013 Acked Write 8 Done Done Sent false
27261 8014 Acked Read 4 Done Done Sent false
27262 8015 Acked Write 10 Done Done Sent false
27263 8016 Acked Read 8 Done Done Sent false
27264 8017 Acked Read 10 Done Done Sent false
27265 8018 Acked Read 7 Done Done Sent false
27266 8019 Acked Read 8 Done Done Sent false
27267 8020 Acked Write 5 Done Done Sent false
27268 8021 Acked Flush 0 Done Done Sent false
27269 8022 Acked Write 10 Done Done Sent false
27270 8023 Acked Flush 0 Done Done Sent false
27271 8024 Acked Flush 0 Done Done Sent false
27272 8025 Acked Write 10 Done Done Sent false
27273 8026 Acked Read 9 Done Done Sent false
27274 8027 Acked Read 3 Done Done Sent false
27275 8028 Acked Read 7 Done Done Sent false
27276 8029 Acked Write 5 Done Done Sent false
27277 8030 Acked Read 9 Done Done Sent false
27278 8031 Acked Read 5 Done Done Sent false
27279 8032 Acked Write 7 Done Done Sent false
27280 8033 Acked Write 5 Done Done Sent false
27281 8034 Acked Read 7 Done Done Sent false
27282 8035 Acked Read 8 Done Done Sent false
27283 8036 Acked Read 10 Done Done Sent false
27284 8037 Acked Read 4 Done Done Sent false
27285 8038 Acked Write 4 Done Done Sent false
27286 8039 Acked Write 9 Done Done Sent false
27287 8040 Acked Read 5 Done Done Sent false
27288 8041 Acked Read 1 Done Done Sent false
27289 8042 Acked Read 5 Done Done Sent false
27290 8043 Acked Write 2 Done Done Sent false
27291 8044 Acked Read 1 Done Done Sent false
27292 8045 Acked Flush 0 Done Done Sent false
27293 8046 Acked Read 10 Done Done Sent false
27294 8047 Acked Read 2 Done Done Sent false
27295 8048 Acked Write 2 Done Done Sent false
27296 8049 Acked Write 5 Done Done Sent false
27297 8050 Acked Read 1 Done Done Sent false
27298 8051 Acked Write 10 Done Done Sent false
27299 8052 Acked Write 9 Done Done Sent false
27300 8053 Acked Read 7 Done Done Sent false
27301 8054 Acked Write 10 Done Done Sent false
27302 8055 Acked Read 5 Done Done Sent false
27303 8056 Acked Read 5 Done Done Sent false
27304 8057 Acked Read 5 Done Done Sent false
27305 8058 Acked Write 9 Done Done Sent false
27306 8059 Acked Write 6 Done Done Sent false
27307 8060 Acked Read 2 Done Done Sent false
27308 8061 Acked Flush 0 Done Done Sent false
27309 8062 Acked Read 7 Done Done Sent false
27310 8063 Acked Read 2 Done Done Sent false
27311 8064 Acked Write 6 Done Done Sent false
27312 8065 Acked Write 2 Done Done Sent false
27313 8066 Acked Write 9 Done Done Sent false
27314 8067 Acked Read 2 Done Done Sent false
27315 8068 Acked Read 1 Done Done Sent false
27316 8069 Acked Flush 0 Done Done Sent false
27317 8070 Acked Write 5 Done Done Sent false
27318 8071 Acked Write 5 Done Done Sent false
27319 8072 Acked Read 9 Done Done Sent false
27320 8073 Acked Read 9 Done Done Sent false
27321 8074 Acked Read 4 Done Done Sent false
27322 8075 Acked Flush 0 Done Done Sent false
27323 8076 Acked Write 3 Done Done Sent false
27324 8077 Acked Write 1 Done Done Sent false
27325 8078 Acked Write 10 Done Done Sent false
27326 8079 Acked Read 2 Done Done Sent false
27327 8080 Acked Read 4 Done Done Sent false
27328 8081 Acked Read 4 Done Done Sent false
27329 8082 Acked Write 8 Done Done Sent false
27330 8083 Acked Write 7 Done Done Sent false
27331 8084 Acked Write 4 Done Done Sent false
27332 8085 Acked Read 4 Done Done Sent false
27333 8086 Acked Read 2 Done Done Sent false
27334 8087 Acked Write 4 Done Done Sent false
27335 8088 Acked Write 3 Done Done Sent false
27336 8089 Acked Read 9 Done Done Sent false
27337 8090 Acked Write 9 Done Done Sent false
27338 8091 Acked Read 2 Done Done Sent false
27339 8092 Acked Read 1 Done Done Sent false
27340 8093 Acked Read 6 Done Done Sent false
27341 8094 Acked Read 9 Done Done Sent false
27342 8095 Acked Read 6 Done Done Sent false
27343 8096 Acked Write 9 Done Done Sent false
27344 8097 Acked Write 9 Done Done Sent false
27345 8098 Acked Write 5 Done Done Sent false
27346 8099 Acked Read 8 Done Done Sent false
27347 8100 Acked Write 2 Done Done Sent false
27348 8101 Acked Read 2 Done Done Sent false
27349 8102 Acked Flush 0 Done Done Sent false
27350 8103 Acked Read 4 Done Done Sent false
27351 8104 Acked Write 6 Done Done Sent false
27352 8105 Acked Read 8 Done Done Sent false
27353 8106 Acked Write 7 Done Done Sent false
27354 8107 Acked Write 5 Done Done Sent false
27355 8108 Acked Write 3 Done Done Sent false
27356 8109 Acked Write 7 Done Done Sent false
27357 8110 Acked Read 1 Done Done Sent false
27358 8111 Acked Write 1 Done Done Sent false
27359 8112 Acked Write 9 Done Done Sent false
27360 8113 Acked Flush 0 Done Done Sent false
27361 8114 Acked Write 3 Done Done Sent false
27362 8115 Acked Write 8 Done Done Sent false
27363 8116 Acked Read 10 Done Done Sent false
27364 8117 Acked Write 5 Done Done Sent false
27365 8118 Acked Read 4 Done Done Sent false
27366 8119 Acked Read 2 Done Done Sent false
27367 8120 Acked Read 6 Done Done Sent false
27368 8121 Acked Read 5 Done Done Sent false
27369 8122 Acked Flush 0 Done Done Sent false
27370 8123 Acked Write 9 Done Done Sent false
27371 8124 Acked Read 10 Done Done Sent false
27372 8125 Acked Flush 0 Done Done Sent false
27373 8126 Acked Read 1 Done Done Sent false
27374 8127 Acked Read 2 Done Done Sent false
27375 8128 Acked Flush 0 Done Done Sent false
27376 8129 Acked Write 4 Done Done Sent false
27377 8130 Acked Write 8 Done Done Sent false
27378 8131 Acked Write 7 Done Done Sent false
27379 8132 Acked Read 6 Done Done Sent false
27380 8133 Acked Read 4 Done Done Sent false
27381 8134 Acked Flush 0 Done Done Sent false
27382 8135 Acked Read 3 Done Done Sent false
27383 8136 Acked Read 9 Done Done Sent false
27384 8137 Acked Read 3 Done Done Sent false
27385 8138 Acked Read 8 Done Done Sent false
27386 8139 Acked Write 9 Done Done Sent false
27387 8140 Acked Flush 0 Done Done Sent false
27388 8141 Acked Write 4 Done Done Sent false
27389 8142 Acked Read 9 Done Done Sent false
27390 8143 Acked Read 4 Done Done Sent false
27391 8144 Acked Flush 0 Done Done Sent false
27392 8145 Acked Read 4 Done Done Sent false
27393 8146 Acked Read 4 Done Done Sent false
27394 8147 Acked Write 4 Done Done Sent false
27395 8148 Acked Write 5 Done Done Sent false
27396 8149 Acked Read 1 Done Done Sent false
27397 8150 Acked Read 7 Done Done Sent false
27398 8151 Acked Read 7 Done Done Sent false
27399 8152 Acked Write 4 Done Done Sent false
27400 8153 Acked Write 10 Done Done Sent false
27401 8154 Acked Flush 0 Done Done Sent false
27402 8155 Acked Write 10 Done Done Sent false
27403 8156 Acked Read 2 Done Done Sent false
27404 8157 Acked Write 10 Done Done Sent false
27405 8158 Acked Read 7 Done Done Sent false
27406 8159 Acked Read 6 Done Done Sent false
27407 8160 Acked Flush 0 Done Done Sent false
27408 8161 Acked Write 8 Done Done Sent false
27409 8162 Acked Write 9 Done Done Sent false
27410 8163 Acked Write 1 Done Done Sent false
27411 8164 Acked Read 3 Done Done Sent false
27412 8165 Acked Read 7 Done Done Sent false
27413 8166 Acked Read 4 Done Done Sent false
27414 8167 Acked Write 6 Done Done Sent false
27415 8168 Acked Read 6 Done Done Sent false
27416 8169 Acked Read 5 Done Done Sent false
27417 8170 Acked Write 9 Done Done Sent false
27418 8171 Acked Read 3 Done Done Sent false
27419 8172 Acked Read 2 Done Done Sent false
27420 8173 Acked Write 7 Done Done Sent false
27421 8174 Acked Flush 0 Done Done Sent false
27422 8175 Acked Read 8 Done Done Sent false
27423 8176 Acked Write 9 Done Done Sent false
27424 8177 Acked Read 3 Done Done Sent false
27425 8178 Acked Read 7 Done Done Sent false
27426 8179 Acked Write 9 Done Done Sent false
27427 8180 Acked Read 3 Done Done Sent false
27428 8181 Acked Write 4 Done Done Sent false
27429 8182 Acked Read 2 Done Done Sent false
27430 8183 Acked Read 3 Done Done Sent false
27431 8184 Acked Write 7 Done Done Sent false
27432 8185 Acked Write 4 Done Done Sent false
27433 8186 Acked Flush 0 Done Done Sent false
27434 8187 Acked Read 6 Done Done Sent false
27435 8188 Acked Read 4 Done Done Sent false
27436 8189 Acked Read 9 Done Done Sent false
27437 8190 Acked Write 4 Done Done Sent false
27438 8191 Acked Read 3 Done Done Sent false
27439 8192 Acked Write 9 Done Done Sent false
27440 8193 Acked Read 6 Done Done Sent false
27441 8194 Acked Write 4 Done Done Sent false
27442 8195 Acked Write 10 Done Done Sent false
27443 8196 Acked Write 5 Done Done Sent false
27444 8197 Acked Read 7 Done Done Sent false
27445 8198 Acked Write 2 Done Done Sent false
27446 8199 Acked Write 4 Done Done Sent false
27447 8200 Acked Write 9 Done Done Sent false
27448 8201 Acked Read 5 Done Done Sent false
27449 8202 Acked Read 6 Done Done Sent false
27450 8203 Acked Read 5 Done Done Sent false
27451 8204 Acked Read 1 Done Done Sent false
27452 8205 Acked Write 4 Done Done Sent false
27453 8206 Acked Read 2 Done Done Sent false
27454 8207 Acked Read 9 Done Done Sent false
27455 8208 Acked Flush 0 Done Done Sent false
27456 8209 Acked Write 10 Done Done Sent false
27457 8210 Acked Read 7 Done Done Sent false
27458 8211 Acked Read 10 Done Done Sent false
27459 8212 Acked Read 1 Done Done Sent false
27460 8213 Acked Flush 0 Done Done Sent false
27461 8214 Acked Write 10 Done Done Sent false
27462 8215 Acked Read 10 Done Done Sent false
27463 8216 Acked Write 4 Done Done Sent false
27464 8217 Acked Read 1 Done Done Sent false
27465 8218 Acked Write 3 Done Done Sent false
27466 8219 Acked Read 9 Done Done Sent false
27467 8220 Acked Read 5 Done Done Sent false
27468 8221 Acked Flush 0 Done Done Sent false
27469 8222 Acked Read 1 Done Done Sent false
27470 8223 Acked Flush 0 Done Done Sent false
27471 8224 Acked Write 6 Done Done Sent false
27472 8225 Acked Write 5 Done Done Sent false
27473 8226 Acked Read 10 Done Done Sent false
27474 8227 Acked Write 4 Done Done Sent false
27475 8228 Acked Flush 0 Done Done Sent false
27476 8229 Acked Write 4 Done Done Sent false
27477 8230 Acked Write 10 Done Done Sent false
27478 8231 Acked Write 1 Done Done Sent false
27479 8232 Acked Read 8 Done Done Sent false
27480 8233 Acked Write 4 Done Done Sent false
27481 8234 Acked Write 8 Done Done Sent false
27482 8235 Acked Write 4 Done Done Sent false
27483 8236 Acked Read 2 Done Done Sent false
27484 8237 Acked Write 7 Done Done Sent false
27485 8238 Acked Read 3 Done Done Sent false
27486 8239 Acked Flush 0 Done Done Sent false
27487 8240 Acked Write 2 Done Done Sent false
27488 8241 Acked Flush 0 Done Done Sent false
27489 8242 Acked Flush 0 Done Done Sent false
27490 STATES DS:0 DS:1 DS:2 TOTAL
27491 Sent 0 0 301 301
27492 Done 301 301 0 602
27493 Skipped 0 0 0 0
27494 Error 0 0 0 0
27495 Last Flush: Some(JobId(8242)) Some(JobId(8242)) Some(JobId(7941))
27496 Downstairs last five completed: 7941 7940 7939 7938 7937
27497 Upstairs last five completed: 8242 8241 8240 8239 8238
27498 CLIENT: Up:0 ds:301 act:2
274992025-02-25T20:52:35.630ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
275002025-02-25T20:52:36.633ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
275012025-02-25T20:52:37.633ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
27502 ----------------------------------------------------------------
27503 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27504 States: Active Active Offline
27505 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27506 7942 Acked Write 1 Done Done Sent false
27507 7943 Acked Read 4 Done Done Sent false
27508 7944 Acked Read 3 Done Done Sent false
27509 7945 Acked Write 2 Done Done Sent false
27510 7946 Acked Read 4 Done Done Sent false
27511 7947 Acked Flush 0 Done Done Sent false
27512 7948 Acked Read 3 Done Done Sent false
27513 7949 Acked Write 7 Done Done Sent false
27514 7950 Acked Read 7 Done Done Sent false
27515 7951 Acked Read 3 Done Done Sent false
27516 7952 Acked Flush 0 Done Done Sent false
27517 7953 Acked Read 1 Done Done Sent false
27518 7954 Acked Read 4 Done Done Sent false
27519 7955 Acked Read 4 Done Done Sent false
27520 7956 Acked Read 4 Done Done Sent false
27521 7957 Acked Write 10 Done Done Sent false
27522 7958 Acked Read 5 Done Done Sent false
27523 7959 Acked Write 1 Done Done Sent false
27524 7960 Acked Write 9 Done Done Sent false
27525 7961 Acked Write 1 Done Done Sent false
27526 7962 Acked Read 7 Done Done Sent false
27527 7963 Acked Read 3 Done Done Sent false
27528 7964 Acked Flush 0 Done Done Sent false
27529 7965 Acked Write 1 Done Done Sent false
27530 7966 Acked Read 2 Done Done Sent false
27531 7967 Acked Read 3 Done Done Sent false
27532 7968 Acked Read 5 Done Done Sent false
27533 7969 Acked Read 9 Done Done Sent false
27534 7970 Acked Write 8 Done Done Sent false
27535 7971 Acked Write 1 Done Done Sent false
27536 7972 Acked Read 9 Done Done Sent false
27537 7973 Acked Flush 0 Done Done Sent false
27538 7974 Acked Write 7 Done Done Sent false
27539 7975 Acked Write 3 Done Done Sent false
27540 7976 Acked Read 4 Done Done Sent false
27541 7977 Acked Write 8 Done Done Sent false
27542 7978 Acked Flush 0 Done Done Sent false
27543 7979 Acked Read 6 Done Done Sent false
27544 7980 Acked Write 5 Done Done Sent false
27545 7981 Acked Read 10 Done Done Sent false
27546 7982 Acked Read 9 Done Done Sent false
27547 7983 Acked Read 9 Done Done Sent false
27548 7984 Acked Write 8 Done Done Sent false
27549 7985 Acked Read 5 Done Done Sent false
27550 7986 Acked Write 8 Done Done Sent false
27551 7987 Acked Read 3 Done Done Sent false
27552 7988 Acked Read 2 Done Done Sent false
27553 7989 Acked Read 10 Done Done Sent false
27554 7990 Acked Flush 0 Done Done Sent false
27555 7991 Acked Read 6 Done Done Sent false
27556 7992 Acked Read 5 Done Done Sent false
27557 7993 Acked Read 10 Done Done Sent false
27558 7994 Acked Write 6 Done Done Sent false
27559 7995 Acked Read 9 Done Done Sent false
27560 7996 Acked Write 1 Done Done Sent false
27561 7997 Acked Read 4 Done Done Sent false
27562 7998 Acked Read 10 Done Done Sent false
27563 7999 Acked Write 1 Done Done Sent false
27564 8000 Acked Read 2 Done Done Sent false
27565 8001 Acked Write 9 Done Done Sent false
27566 8002 Acked Read 2 Done Done Sent false
27567 8003 Acked Write 10 Done Done Sent false
27568 8004 Acked Read 8 Done Done Sent false
27569 8005 Acked Read 7 Done Done Sent false
27570 8006 Acked Write 8 Done Done Sent false
27571 8007 Acked Read 4 Done Done Sent false
27572 8008 Acked Write 2 Done Done Sent false
27573 8009 Acked Write 6 Done Done Sent false
27574 8010 Acked Flush 0 Done Done Sent false
27575 8011 Acked Write 6 Done Done Sent false
27576 8012 Acked Read 1 Done Done Sent false
27577 8013 Acked Write 8 Done Done Sent false
27578 8014 Acked Read 4 Done Done Sent false
27579 8015 Acked Write 10 Done Done Sent false
27580 8016 Acked Read 8 Done Done Sent false
27581 8017 Acked Read 10 Done Done Sent false
27582 8018 Acked Read 7 Done Done Sent false
27583 8019 Acked Read 8 Done Done Sent false
27584 8020 Acked Write 5 Done Done Sent false
27585 8021 Acked Flush 0 Done Done Sent false
27586 8022 Acked Write 10 Done Done Sent false
27587 8023 Acked Flush 0 Done Done Sent false
27588 8024 Acked Flush 0 Done Done Sent false
27589 8025 Acked Write 10 Done Done Sent false
27590 8026 Acked Read 9 Done Done Sent false
27591 8027 Acked Read 3 Done Done Sent false
27592 8028 Acked Read 7 Done Done Sent false
27593 8029 Acked Write 5 Done Done Sent false
27594 8030 Acked Read 9 Done Done Sent false
27595 8031 Acked Read 5 Done Done Sent false
27596 8032 Acked Write 7 Done Done Sent false
27597 8033 Acked Write 5 Done Done Sent false
27598 8034 Acked Read 7 Done Done Sent false
27599 8035 Acked Read 8 Done Done Sent false
27600 8036 Acked Read 10 Done Done Sent false
27601 8037 Acked Read 4 Done Done Sent false
27602 8038 Acked Write 4 Done Done Sent false
27603 8039 Acked Write 9 Done Done Sent false
27604 8040 Acked Read 5 Done Done Sent false
27605 8041 Acked Read 1 Done Done Sent false
27606 8042 Acked Read 5 Done Done Sent false
27607 8043 Acked Write 2 Done Done Sent false
27608 8044 Acked Read 1 Done Done Sent false
27609 8045 Acked Flush 0 Done Done Sent false
27610 8046 Acked Read 10 Done Done Sent false
27611 8047 Acked Read 2 Done Done Sent false
27612 8048 Acked Write 2 Done Done Sent false
27613 8049 Acked Write 5 Done Done Sent false
27614 8050 Acked Read 1 Done Done Sent false
27615 8051 Acked Write 10 Done Done Sent false
27616 8052 Acked Write 9 Done Done Sent false
27617 8053 Acked Read 7 Done Done Sent false
27618 8054 Acked Write 10 Done Done Sent false
27619 8055 Acked Read 5 Done Done Sent false
27620 8056 Acked Read 5 Done Done Sent false
27621 8057 Acked Read 5 Done Done Sent false
27622 8058 Acked Write 9 Done Done Sent false
27623 8059 Acked Write 6 Done Done Sent false
27624 8060 Acked Read 2 Done Done Sent false
27625 8061 Acked Flush 0 Done Done Sent false
27626 8062 Acked Read 7 Done Done Sent false
27627 8063 Acked Read 2 Done Done Sent false
27628 8064 Acked Write 6 Done Done Sent false
27629 8065 Acked Write 2 Done Done Sent false
27630 8066 Acked Write 9 Done Done Sent false
27631 8067 Acked Read 2 Done Done Sent false
27632 8068 Acked Read 1 Done Done Sent false
27633 8069 Acked Flush 0 Done Done Sent false
27634 8070 Acked Write 5 Done Done Sent false
27635 8071 Acked Write 5 Done Done Sent false
27636 8072 Acked Read 9 Done Done Sent false
27637 8073 Acked Read 9 Done Done Sent false
27638 8074 Acked Read 4 Done Done Sent false
27639 8075 Acked Flush 0 Done Done Sent false
27640 8076 Acked Write 3 Done Done Sent false
27641 8077 Acked Write 1 Done Done Sent false
27642 8078 Acked Write 10 Done Done Sent false
27643 8079 Acked Read 2 Done Done Sent false
27644 8080 Acked Read 4 Done Done Sent false
27645 8081 Acked Read 4 Done Done Sent false
27646 8082 Acked Write 8 Done Done Sent false
27647 8083 Acked Write 7 Done Done Sent false
27648 8084 Acked Write 4 Done Done Sent false
27649 8085 Acked Read 4 Done Done Sent false
27650 8086 Acked Read 2 Done Done Sent false
27651 8087 Acked Write 4 Done Done Sent false
27652 8088 Acked Write 3 Done Done Sent false
27653 8089 Acked Read 9 Done Done Sent false
27654 8090 Acked Write 9 Done Done Sent false
27655 8091 Acked Read 2 Done Done Sent false
27656 8092 Acked Read 1 Done Done Sent false
27657 8093 Acked Read 6 Done Done Sent false
27658 8094 Acked Read 9 Done Done Sent false
27659 8095 Acked Read 6 Done Done Sent false
27660 8096 Acked Write 9 Done Done Sent false
27661 8097 Acked Write 9 Done Done Sent false
27662 8098 Acked Write 5 Done Done Sent false
27663 8099 Acked Read 8 Done Done Sent false
27664 8100 Acked Write 2 Done Done Sent false
27665 8101 Acked Read 2 Done Done Sent false
27666 8102 Acked Flush 0 Done Done Sent false
27667 8103 Acked Read 4 Done Done Sent false
27668 8104 Acked Write 6 Done Done Sent false
27669 8105 Acked Read 8 Done Done Sent false
27670 8106 Acked Write 7 Done Done Sent false
27671 8107 Acked Write 5 Done Done Sent false
27672 8108 Acked Write 3 Done Done Sent false
27673 8109 Acked Write 7 Done Done Sent false
27674 8110 Acked Read 1 Done Done Sent false
27675 8111 Acked Write 1 Done Done Sent false
27676 8112 Acked Write 9 Done Done Sent false
27677 8113 Acked Flush 0 Done Done Sent false
27678 8114 Acked Write 3 Done Done Sent false
27679 8115 Acked Write 8 Done Done Sent false
27680 8116 Acked Read 10 Done Done Sent false
27681 8117 Acked Write 5 Done Done Sent false
27682 8118 Acked Read 4 Done Done Sent false
27683 8119 Acked Read 2 Done Done Sent false
27684 8120 Acked Read 6 Done Done Sent false
27685 8121 Acked Read 5 Done Done Sent false
27686 8122 Acked Flush 0 Done Done Sent false
27687 8123 Acked Write 9 Done Done Sent false
27688 8124 Acked Read 10 Done Done Sent false
27689 8125 Acked Flush 0 Done Done Sent false
27690 8126 Acked Read 1 Done Done Sent false
27691 8127 Acked Read 2 Done Done Sent false
27692 8128 Acked Flush 0 Done Done Sent false
27693 8129 Acked Write 4 Done Done Sent false
27694 8130 Acked Write 8 Done Done Sent false
27695 8131 Acked Write 7 Done Done Sent false
27696 8132 Acked Read 6 Done Done Sent false
27697 8133 Acked Read 4 Done Done Sent false
27698 8134 Acked Flush 0 Done Done Sent false
27699 8135 Acked Read 3 Done Done Sent false
27700 8136 Acked Read 9 Done Done Sent false
27701 8137 Acked Read 3 Done Done Sent false
27702 8138 Acked Read 8 Done Done Sent false
27703 8139 Acked Write 9 Done Done Sent false
27704 8140 Acked Flush 0 Done Done Sent false
27705 8141 Acked Write 4 Done Done Sent false
27706 8142 Acked Read 9 Done Done Sent false
27707 8143 Acked Read 4 Done Done Sent false
27708 8144 Acked Flush 0 Done Done Sent false
27709 8145 Acked Read 4 Done Done Sent false
27710 8146 Acked Read 4 Done Done Sent false
27711 8147 Acked Write 4 Done Done Sent false
27712 8148 Acked Write 5 Done Done Sent false
27713 8149 Acked Read 1 Done Done Sent false
27714 8150 Acked Read 7 Done Done Sent false
27715 8151 Acked Read 7 Done Done Sent false
27716 8152 Acked Write 4 Done Done Sent false
27717 8153 Acked Write 10 Done Done Sent false
27718 8154 Acked Flush 0 Done Done Sent false
27719 8155 Acked Write 10 Done Done Sent false
27720 8156 Acked Read 2 Done Done Sent false
27721 8157 Acked Write 10 Done Done Sent false
27722 8158 Acked Read 7 Done Done Sent false
27723 8159 Acked Read 6 Done Done Sent false
27724 8160 Acked Flush 0 Done Done Sent false
27725 8161 Acked Write 8 Done Done Sent false
27726 8162 Acked Write 9 Done Done Sent false
27727 8163 Acked Write 1 Done Done Sent false
27728 8164 Acked Read 3 Done Done Sent false
27729 8165 Acked Read 7 Done Done Sent false
27730 8166 Acked Read 4 Done Done Sent false
27731 8167 Acked Write 6 Done Done Sent false
27732 8168 Acked Read 6 Done Done Sent false
27733 8169 Acked Read 5 Done Done Sent false
27734 8170 Acked Write 9 Done Done Sent false
27735 8171 Acked Read 3 Done Done Sent false
27736 8172 Acked Read 2 Done Done Sent false
27737 8173 Acked Write 7 Done Done Sent false
27738 8174 Acked Flush 0 Done Done Sent false
27739 8175 Acked Read 8 Done Done Sent false
27740 8176 Acked Write 9 Done Done Sent false
27741 8177 Acked Read 3 Done Done Sent false
27742 8178 Acked Read 7 Done Done Sent false
27743 8179 Acked Write 9 Done Done Sent false
27744 8180 Acked Read 3 Done Done Sent false
27745 8181 Acked Write 4 Done Done Sent false
27746 8182 Acked Read 2 Done Done Sent false
27747 8183 Acked Read 3 Done Done Sent false
27748 8184 Acked Write 7 Done Done Sent false
27749 8185 Acked Write 4 Done Done Sent false
27750 8186 Acked Flush 0 Done Done Sent false
27751 8187 Acked Read 6 Done Done Sent false
27752 8188 Acked Read 4 Done Done Sent false
27753 8189 Acked Read 9 Done Done Sent false
27754 8190 Acked Write 4 Done Done Sent false
27755 8191 Acked Read 3 Done Done Sent false
27756 8192 Acked Write 9 Done Done Sent false
27757 8193 Acked Read 6 Done Done Sent false
27758 8194 Acked Write 4 Done Done Sent false
27759 8195 Acked Write 10 Done Done Sent false
27760 8196 Acked Write 5 Done Done Sent false
27761 8197 Acked Read 7 Done Done Sent false
27762 8198 Acked Write 2 Done Done Sent false
27763 8199 Acked Write 4 Done Done Sent false
27764 8200 Acked Write 9 Done Done Sent false
27765 8201 Acked Read 5 Done Done Sent false
27766 8202 Acked Read 6 Done Done Sent false
27767 8203 Acked Read 5 Done Done Sent false
27768 8204 Acked Read 1 Done Done Sent false
27769 8205 Acked Write 4 Done Done Sent false
27770 8206 Acked Read 2 Done Done Sent false
27771 8207 Acked Read 9 Done Done Sent false
27772 8208 Acked Flush 0 Done Done Sent false
27773 8209 Acked Write 10 Done Done Sent false
27774 8210 Acked Read 7 Done Done Sent false
27775 8211 Acked Read 10 Done Done Sent false
27776 8212 Acked Read 1 Done Done Sent false
27777 8213 Acked Flush 0 Done Done Sent false
27778 8214 Acked Write 10 Done Done Sent false
27779 8215 Acked Read 10 Done Done Sent false
27780 8216 Acked Write 4 Done Done Sent false
27781 8217 Acked Read 1 Done Done Sent false
27782 8218 Acked Write 3 Done Done Sent false
27783 8219 Acked Read 9 Done Done Sent false
27784 8220 Acked Read 5 Done Done Sent false
27785 8221 Acked Flush 0 Done Done Sent false
27786 8222 Acked Read 1 Done Done Sent false
27787 8223 Acked Flush 0 Done Done Sent false
27788 8224 Acked Write 6 Done Done Sent false
27789 8225 Acked Write 5 Done Done Sent false
27790 8226 Acked Read 10 Done Done Sent false
27791 8227 Acked Write 4 Done Done Sent false
27792 8228 Acked Flush 0 Done Done Sent false
27793 8229 Acked Write 4 Done Done Sent false
27794 8230 Acked Write 10 Done Done Sent false
27795 8231 Acked Write 1 Done Done Sent false
27796 8232 Acked Read 8 Done Done Sent false
27797 8233 Acked Write 4 Done Done Sent false
27798 8234 Acked Write 8 Done Done Sent false
27799 8235 Acked Write 4 Done Done Sent false
27800 8236 Acked Read 2 Done Done Sent false
27801 8237 Acked Write 7 Done Done Sent false
27802 8238 Acked Read 3 Done Done Sent false
27803 8239 Acked Flush 0 Done Done Sent false
27804 8240 Acked Write 2 Done Done Sent false
27805 8241 Acked Flush 0 Done Done Sent false
27806 8242 Acked Flush 0 Done Done Sent false
27807 STATES DS:0 DS:1 DS:2 TOTAL
27808 Sent 0 0 301 301
27809 Done 301 301 0 602
27810 Skipped 0 0 0 0
27811 Error 0 0 0 0
27812 Last Flush: Some(JobId(8242)) Some(JobId(8242)) Some(JobId(7941))
27813 Downstairs last five completed: 7941 7940 7939 7938 7937
27814 Upstairs last five completed: 8242 8241 8240 8239 8238
27815 CLIENT: Up:0 ds:301 act:2
278162025-02-25T20:52:39.637ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278172025-02-25T20:52:40.552ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278182025-02-25T20:52:40.552ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278192025-02-25T20:52:40.552ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278202025-02-25T20:52:40.552ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278212025-02-25T20:52:40.552ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
27822 [8830] got start action
27823 [8830] I am going to restart
27824 Starting downstairs at port 8830
27825 Make output file at "/tmp/dsc/downstairs-8830.txt"
27826 [8830][2] reports Starting
27827 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
27828 [8830][2] reports Running
278292025-02-25T20:52:41.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278302025-02-25T20:52:41.639ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
278312025-02-25T20:52:42.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
27832 ----------------------------------------------------------------
27833 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27834 States: Active Active Offline
27835 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27836 7942 Acked Write 1 Done Done Sent false
27837 7943 Acked Read 4 Done Done Sent false
27838 7944 Acked Read 3 Done Done Sent false
27839 7945 Acked Write 2 Done Done Sent false
27840 7946 Acked Read 4 Done Done Sent false
27841 7947 Acked Flush 0 Done Done Sent false
27842 7948 Acked Read 3 Done Done Sent false
27843 7949 Acked Write 7 Done Done Sent false
27844 7950 Acked Read 7 Done Done Sent false
27845 7951 Acked Read 3 Done Done Sent false
27846 7952 Acked Flush 0 Done Done Sent false
27847 7953 Acked Read 1 Done Done Sent false
27848 7954 Acked Read 4 Done Done Sent false
27849 7955 Acked Read 4 Done Done Sent false
27850 7956 Acked Read 4 Done Done Sent false
27851 7957 Acked Write 10 Done Done Sent false
27852 7958 Acked Read 5 Done Done Sent false
27853 7959 Acked Write 1 Done Done Sent false
27854 7960 Acked Write 9 Done Done Sent false
27855 7961 Acked Write 1 Done Done Sent false
27856 7962 Acked Read 7 Done Done Sent false
27857 7963 Acked Read 3 Done Done Sent false
27858 7964 Acked Flush 0 Done Done Sent false
27859 7965 Acked Write 1 Done Done Sent false
27860 7966 Acked Read 2 Done Done Sent false
27861 7967 Acked Read 3 Done Done Sent false
27862 7968 Acked Read 5 Done Done Sent false
27863 7969 Acked Read 9 Done Done Sent false
27864 7970 Acked Write 8 Done Done Sent false
27865 7971 Acked Write 1 Done Done Sent false
27866 7972 Acked Read 9 Done Done Sent false
27867 7973 Acked Flush 0 Done Done Sent false
27868 7974 Acked Write 7 Done Done Sent false
27869 7975 Acked Write 3 Done Done Sent false
27870 7976 Acked Read 4 Done Done Sent false
27871 7977 Acked Write 8 Done Done Sent false
27872 7978 Acked Flush 0 Done Done Sent false
27873 7979 Acked Read 6 Done Done Sent false
27874 7980 Acked Write 5 Done Done Sent false
27875 7981 Acked Read 10 Done Done Sent false
27876 7982 Acked Read 9 Done Done Sent false
27877 7983 Acked Read 9 Done Done Sent false
27878 7984 Acked Write 8 Done Done Sent false
27879 7985 Acked Read 5 Done Done Sent false
27880 7986 Acked Write 8 Done Done Sent false
27881 7987 Acked Read 3 Done Done Sent false
27882 7988 Acked Read 2 Done Done Sent false
27883 7989 Acked Read 10 Done Done Sent false
27884 7990 Acked Flush 0 Done Done Sent false
27885 7991 Acked Read 6 Done Done Sent false
27886 7992 Acked Read 5 Done Done Sent false
27887 7993 Acked Read 10 Done Done Sent false
27888 7994 Acked Write 6 Done Done Sent false
27889 7995 Acked Read 9 Done Done Sent false
27890 7996 Acked Write 1 Done Done Sent false
27891 7997 Acked Read 4 Done Done Sent false
27892 7998 Acked Read 10 Done Done Sent false
27893 7999 Acked Write 1 Done Done Sent false
27894 8000 Acked Read 2 Done Done Sent false
27895 8001 Acked Write 9 Done Done Sent false
27896 8002 Acked Read 2 Done Done Sent false
27897 8003 Acked Write 10 Done Done Sent false
27898 8004 Acked Read 8 Done Done Sent false
27899 8005 Acked Read 7 Done Done Sent false
27900 8006 Acked Write 8 Done Done Sent false
27901 8007 Acked Read 4 Done Done Sent false
27902 8008 Acked Write 2 Done Done Sent false
27903 8009 Acked Write 6 Done Done Sent false
27904 8010 Acked Flush 0 Done Done Sent false
27905 8011 Acked Write 6 Done Done Sent false
27906 8012 Acked Read 1 Done Done Sent false
27907 8013 Acked Write 8 Done Done Sent false
27908 8014 Acked Read 4 Done Done Sent false
27909 8015 Acked Write 10 Done Done Sent false
27910 8016 Acked Read 8 Done Done Sent false
27911 8017 Acked Read 10 Done Done Sent false
27912 8018 Acked Read 7 Done Done Sent false
27913 8019 Acked Read 8 Done Done Sent false
27914 8020 Acked Write 5 Done Done Sent false
27915 8021 Acked Flush 0 Done Done Sent false
27916 8022 Acked Write 10 Done Done Sent false
27917 8023 Acked Flush 0 Done Done Sent false
27918 8024 Acked Flush 0 Done Done Sent false
27919 8025 Acked Write 10 Done Done Sent false
27920 8026 Acked Read 9 Done Done Sent false
27921 8027 Acked Read 3 Done Done Sent false
27922 8028 Acked Read 7 Done Done Sent false
27923 8029 Acked Write 5 Done Done Sent false
27924 8030 Acked Read 9 Done Done Sent false
27925 8031 Acked Read 5 Done Done Sent false
27926 8032 Acked Write 7 Done Done Sent false
27927 8033 Acked Write 5 Done Done Sent false
27928 8034 Acked Read 7 Done Done Sent false
27929 8035 Acked Read 8 Done Done Sent false
27930 8036 Acked Read 10 Done Done Sent false
27931 8037 Acked Read 4 Done Done Sent false
27932 8038 Acked Write 4 Done Done Sent false
27933 8039 Acked Write 9 Done Done Sent false
27934 8040 Acked Read 5 Done Done Sent false
27935 8041 Acked Read 1 Done Done Sent false
27936 8042 Acked Read 5 Done Done Sent false
27937 8043 Acked Write 2 Done Done Sent false
27938 8044 Acked Read 1 Done Done Sent false
27939 8045 Acked Flush 0 Done Done Sent false
27940 8046 Acked Read 10 Done Done Sent false
27941 8047 Acked Read 2 Done Done Sent false
27942 8048 Acked Write 2 Done Done Sent false
27943 8049 Acked Write 5 Done Done Sent false
27944 8050 Acked Read 1 Done Done Sent false
27945 8051 Acked Write 10 Done Done Sent false
27946 8052 Acked Write 9 Done Done Sent false
27947 8053 Acked Read 7 Done Done Sent false
27948 8054 Acked Write 10 Done Done Sent false
27949 8055 Acked Read 5 Done Done Sent false
27950 8056 Acked Read 5 Done Done Sent false
27951 8057 Acked Read 5 Done Done Sent false
27952 8058 Acked Write 9 Done Done Sent false
27953 8059 Acked Write 6 Done Done Sent false
27954 8060 Acked Read 2 Done Done Sent false
27955 8061 Acked Flush 0 Done Done Sent false
27956 8062 Acked Read 7 Done Done Sent false
27957 8063 Acked Read 2 Done Done Sent false
27958 8064 Acked Write 6 Done Done Sent false
27959 8065 Acked Write 2 Done Done Sent false
27960 8066 Acked Write 9 Done Done Sent false
27961 8067 Acked Read 2 Done Done Sent false
27962 8068 Acked Read 1 Done Done Sent false
27963 8069 Acked Flush 0 Done Done Sent false
27964 8070 Acked Write 5 Done Done Sent false
27965 8071 Acked Write 5 Done Done Sent false
27966 8072 Acked Read 9 Done Done Sent false
27967 8073 Acked Read 9 Done Done Sent false
27968 8074 Acked Read 4 Done Done Sent false
27969 8075 Acked Flush 0 Done Done Sent false
27970 8076 Acked Write 3 Done Done Sent false
27971 8077 Acked Write 1 Done Done Sent false
27972 8078 Acked Write 10 Done Done Sent false
27973 8079 Acked Read 2 Done Done Sent false
27974 8080 Acked Read 4 Done Done Sent false
27975 8081 Acked Read 4 Done Done Sent false
27976 8082 Acked Write 8 Done Done Sent false
27977 8083 Acked Write 7 Done Done Sent false
27978 8084 Acked Write 4 Done Done Sent false
27979 8085 Acked Read 4 Done Done Sent false
27980 8086 Acked Read 2 Done Done Sent false
27981 8087 Acked Write 4 Done Done Sent false
27982 8088 Acked Write 3 Done Done Sent false
27983 8089 Acked Read 9 Done Done Sent false
27984 8090 Acked Write 9 Done Done Sent false
27985 8091 Acked Read 2 Done Done Sent false
27986 8092 Acked Read 1 Done Done Sent false
27987 8093 Acked Read 6 Done Done Sent false
27988 8094 Acked Read 9 Done Done Sent false
27989 8095 Acked Read 6 Done Done Sent false
27990 8096 Acked Write 9 Done Done Sent false
27991 8097 Acked Write 9 Done Done Sent false
27992 8098 Acked Write 5 Done Done Sent false
27993 8099 Acked Read 8 Done Done Sent false
27994 8100 Acked Write 2 Done Done Sent false
27995 8101 Acked Read 2 Done Done Sent false
27996 8102 Acked Flush 0 Done Done Sent false
27997 8103 Acked Read 4 Done Done Sent false
27998 8104 Acked Write 6 Done Done Sent false
27999 8105 Acked Read 8 Done Done Sent false
28000 8106 Acked Write 7 Done Done Sent false
28001 8107 Acked Write 5 Done Done Sent false
28002 8108 Acked Write 3 Done Done Sent false
28003 8109 Acked Write 7 Done Done Sent false
28004 8110 Acked Read 1 Done Done Sent false
28005 8111 Acked Write 1 Done Done Sent false
28006 8112 Acked Write 9 Done Done Sent false
28007 8113 Acked Flush 0 Done Done Sent false
28008 8114 Acked Write 3 Done Done Sent false
28009 8115 Acked Write 8 Done Done Sent false
28010 8116 Acked Read 10 Done Done Sent false
28011 8117 Acked Write 5 Done Done Sent false
28012 8118 Acked Read 4 Done Done Sent false
28013 8119 Acked Read 2 Done Done Sent false
28014 8120 Acked Read 6 Done Done Sent false
28015 8121 Acked Read 5 Done Done Sent false
28016 8122 Acked Flush 0 Done Done Sent false
28017 8123 Acked Write 9 Done Done Sent false
28018 8124 Acked Read 10 Done Done Sent false
28019 8125 Acked Flush 0 Done Done Sent false
28020 8126 Acked Read 1 Done Done Sent false
28021 8127 Acked Read 2 Done Done Sent false
28022 8128 Acked Flush 0 Done Done Sent false
28023 8129 Acked Write 4 Done Done Sent false
28024 8130 Acked Write 8 Done Done Sent false
28025 8131 Acked Write 7 Done Done Sent false
28026 8132 Acked Read 6 Done Done Sent false
28027 8133 Acked Read 4 Done Done Sent false
28028 8134 Acked Flush 0 Done Done Sent false
28029 8135 Acked Read 3 Done Done Sent false
28030 8136 Acked Read 9 Done Done Sent false
28031 8137 Acked Read 3 Done Done Sent false
28032 8138 Acked Read 8 Done Done Sent false
28033 8139 Acked Write 9 Done Done Sent false
28034 8140 Acked Flush 0 Done Done Sent false
28035 8141 Acked Write 4 Done Done Sent false
28036 8142 Acked Read 9 Done Done Sent false
28037 8143 Acked Read 4 Done Done Sent false
28038 8144 Acked Flush 0 Done Done Sent false
28039 8145 Acked Read 4 Done Done Sent false
28040 8146 Acked Read 4 Done Done Sent false
28041 8147 Acked Write 4 Done Done Sent false
28042 8148 Acked Write 5 Done Done Sent false
28043 8149 Acked Read 1 Done Done Sent false
28044 8150 Acked Read 7 Done Done Sent false
28045 8151 Acked Read 7 Done Done Sent false
28046 8152 Acked Write 4 Done Done Sent false
28047 8153 Acked Write 10 Done Done Sent false
28048 8154 Acked Flush 0 Done Done Sent false
28049 8155 Acked Write 10 Done Done Sent false
28050 8156 Acked Read 2 Done Done Sent false
28051 8157 Acked Write 10 Done Done Sent false
28052 8158 Acked Read 7 Done Done Sent false
28053 8159 Acked Read 6 Done Done Sent false
28054 8160 Acked Flush 0 Done Done Sent false
28055 8161 Acked Write 8 Done Done Sent false
28056 8162 Acked Write 9 Done Done Sent false
28057 8163 Acked Write 1 Done Done Sent false
28058 8164 Acked Read 3 Done Done Sent false
28059 8165 Acked Read 7 Done Done Sent false
28060 8166 Acked Read 4 Done Done Sent false
28061 8167 Acked Write 6 Done Done Sent false
28062 8168 Acked Read 6 Done Done Sent false
28063 8169 Acked Read 5 Done Done Sent false
28064 8170 Acked Write 9 Done Done Sent false
28065 8171 Acked Read 3 Done Done Sent false
28066 8172 Acked Read 2 Done Done Sent false
28067 8173 Acked Write 7 Done Done Sent false
28068 8174 Acked Flush 0 Done Done Sent false
28069 8175 Acked Read 8 Done Done Sent false
28070 8176 Acked Write 9 Done Done Sent false
28071 8177 Acked Read 3 Done Done Sent false
28072 8178 Acked Read 7 Done Done Sent false
28073 8179 Acked Write 9 Done Done Sent false
28074 8180 Acked Read 3 Done Done Sent false
28075 8181 Acked Write 4 Done Done Sent false
28076 8182 Acked Read 2 Done Done Sent false
28077 8183 Acked Read 3 Done Done Sent false
28078 8184 Acked Write 7 Done Done Sent false
28079 8185 Acked Write 4 Done Done Sent false
28080 8186 Acked Flush 0 Done Done Sent false
28081 8187 Acked Read 6 Done Done Sent false
28082 8188 Acked Read 4 Done Done Sent false
28083 8189 Acked Read 9 Done Done Sent false
28084 8190 Acked Write 4 Done Done Sent false
28085 8191 Acked Read 3 Done Done Sent false
28086 8192 Acked Write 9 Done Done Sent false
28087 8193 Acked Read 6 Done Done Sent false
28088 8194 Acked Write 4 Done Done Sent false
28089 8195 Acked Write 10 Done Done Sent false
28090 8196 Acked Write 5 Done Done Sent false
28091 8197 Acked Read 7 Done Done Sent false
28092 8198 Acked Write 2 Done Done Sent false
28093 8199 Acked Write 4 Done Done Sent false
28094 8200 Acked Write 9 Done Done Sent false
28095 8201 Acked Read 5 Done Done Sent false
28096 8202 Acked Read 6 Done Done Sent false
28097 8203 Acked Read 5 Done Done Sent false
28098 8204 Acked Read 1 Done Done Sent false
28099 8205 Acked Write 4 Done Done Sent false
28100 8206 Acked Read 2 Done Done Sent false
28101 8207 Acked Read 9 Done Done Sent false
28102 8208 Acked Flush 0 Done Done Sent false
28103 8209 Acked Write 10 Done Done Sent false
28104 8210 Acked Read 7 Done Done Sent false
28105 8211 Acked Read 10 Done Done Sent false
28106 8212 Acked Read 1 Done Done Sent false
28107 8213 Acked Flush 0 Done Done Sent false
28108 8214 Acked Write 10 Done Done Sent false
28109 8215 Acked Read 10 Done Done Sent false
28110 8216 Acked Write 4 Done Done Sent false
28111 8217 Acked Read 1 Done Done Sent false
28112 8218 Acked Write 3 Done Done Sent false
28113 8219 Acked Read 9 Done Done Sent false
28114 8220 Acked Read 5 Done Done Sent false
28115 8221 Acked Flush 0 Done Done Sent false
28116 8222 Acked Read 1 Done Done Sent false
28117 8223 Acked Flush 0 Done Done Sent false
28118 8224 Acked Write 6 Done Done Sent false
28119 8225 Acked Write 5 Done Done Sent false
28120 8226 Acked Read 10 Done Done Sent false
28121 8227 Acked Write 4 Done Done Sent false
28122 8228 Acked Flush 0 Done Done Sent false
28123 8229 Acked Write 4 Done Done Sent false
28124 8230 Acked Write 10 Done Done Sent false
28125 8231 Acked Write 1 Done Done Sent false
28126 8232 Acked Read 8 Done Done Sent false
28127 8233 Acked Write 4 Done Done Sent false
28128 8234 Acked Write 8 Done Done Sent false
28129 8235 Acked Write 4 Done Done Sent false
28130 8236 Acked Read 2 Done Done Sent false
28131 8237 Acked Write 7 Done Done Sent false
28132 8238 Acked Read 3 Done Done Sent false
28133 8239 Acked Flush 0 Done Done Sent false
28134 8240 Acked Write 2 Done Done Sent false
28135 8241 Acked Flush 0 Done Done Sent false
28136 8242 Acked Flush 0 Done Done Sent false
28137 STATES DS:0 DS:1 DS:2 TOTAL
28138 Sent 0 0 301 301
28139 Done 301 301 0 602
28140 Skipped 0 0 0 0
28141 Error 0 0 0 0
28142 Last Flush: Some(JobId(8242)) Some(JobId(8242)) Some(JobId(7941))
28143 Downstairs last five completed: 7941 7940 7939 7938 7937
28144 Upstairs last five completed: 8242 8241 8240 8239 8238
28145 CLIENT: Up:0 ds:301 act:2
281462025-02-25T20:52:44.011ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
281472025-02-25T20:52:44.643ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
281482025-02-25T20:52:45.144ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
281492025-02-25T20:52:45.646ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
281502025-02-25T20:52:46.646ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
28151 ----------------------------------------------------------------
28152 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28153 States: Active Active Offline
28154 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28155 7942 Acked Write 1 Done Done Sent false
28156 7943 Acked Read 4 Done Done Sent false
28157 7944 Acked Read 3 Done Done Sent false
28158 7945 Acked Write 2 Done Done Sent false
28159 7946 Acked Read 4 Done Done Sent false
28160 7947 Acked Flush 0 Done Done Sent false
28161 7948 Acked Read 3 Done Done Sent false
28162 7949 Acked Write 7 Done Done Sent false
28163 7950 Acked Read 7 Done Done Sent false
28164 7951 Acked Read 3 Done Done Sent false
28165 7952 Acked Flush 0 Done Done Sent false
28166 7953 Acked Read 1 Done Done Sent false
28167 7954 Acked Read 4 Done Done Sent false
28168 7955 Acked Read 4 Done Done Sent false
28169 7956 Acked Read 4 Done Done Sent false
28170 7957 Acked Write 10 Done Done Sent false
28171 7958 Acked Read 5 Done Done Sent false
28172 7959 Acked Write 1 Done Done Sent false
28173 7960 Acked Write 9 Done Done Sent false
28174 7961 Acked Write 1 Done Done Sent false
28175 7962 Acked Read 7 Done Done Sent false
28176 7963 Acked Read 3 Done Done Sent false
28177 7964 Acked Flush 0 Done Done Sent false
28178 7965 Acked Write 1 Done Done Sent false
28179 7966 Acked Read 2 Done Done Sent false
28180 7967 Acked Read 3 Done Done Sent false
28181 7968 Acked Read 5 Done Done Sent false
28182 7969 Acked Read 9 Done Done Sent false
28183 7970 Acked Write 8 Done Done Sent false
28184 7971 Acked Write 1 Done Done Sent false
28185 7972 Acked Read 9 Done Done Sent false
28186 7973 Acked Flush 0 Done Done Sent false
28187 7974 Acked Write 7 Done Done Sent false
28188 7975 Acked Write 3 Done Done Sent false
28189 7976 Acked Read 4 Done Done Sent false
28190 7977 Acked Write 8 Done Done Sent false
28191 7978 Acked Flush 0 Done Done Sent false
28192 7979 Acked Read 6 Done Done Sent false
28193 7980 Acked Write 5 Done Done Sent false
28194 7981 Acked Read 10 Done Done Sent false
28195 7982 Acked Read 9 Done Done Sent false
28196 7983 Acked Read 9 Done Done Sent false
28197 7984 Acked Write 8 Done Done Sent false
28198 7985 Acked Read 5 Done Done Sent false
28199 7986 Acked Write 8 Done Done Sent false
28200 7987 Acked Read 3 Done Done Sent false
28201 7988 Acked Read 2 Done Done Sent false
28202 7989 Acked Read 10 Done Done Sent false
28203 7990 Acked Flush 0 Done Done Sent false
28204 7991 Acked Read 6 Done Done Sent false
28205 7992 Acked Read 5 Done Done Sent false
28206 7993 Acked Read 10 Done Done Sent false
28207 7994 Acked Write 6 Done Done Sent false
28208 7995 Acked Read 9 Done Done Sent false
28209 7996 Acked Write 1 Done Done Sent false
28210 7997 Acked Read 4 Done Done Sent false
28211 7998 Acked Read 10 Done Done Sent false
28212 7999 Acked Write 1 Done Done Sent false
28213 8000 Acked Read 2 Done Done Sent false
28214 8001 Acked Write 9 Done Done Sent false
28215 8002 Acked Read 2 Done Done Sent false
28216 8003 Acked Write 10 Done Done Sent false
28217 8004 Acked Read 8 Done Done Sent false
28218 8005 Acked Read 7 Done Done Sent false
28219 8006 Acked Write 8 Done Done Sent false
28220 8007 Acked Read 4 Done Done Sent false
28221 8008 Acked Write 2 Done Done Sent false
28222 8009 Acked Write 6 Done Done Sent false
28223 8010 Acked Flush 0 Done Done Sent false
28224 8011 Acked Write 6 Done Done Sent false
28225 8012 Acked Read 1 Done Done Sent false
28226 8013 Acked Write 8 Done Done Sent false
28227 8014 Acked Read 4 Done Done Sent false
28228 8015 Acked Write 10 Done Done Sent false
28229 8016 Acked Read 8 Done Done Sent false
28230 8017 Acked Read 10 Done Done Sent false
28231 8018 Acked Read 7 Done Done Sent false
28232 8019 Acked Read 8 Done Done Sent false
28233 8020 Acked Write 5 Done Done Sent false
28234 8021 Acked Flush 0 Done Done Sent false
28235 8022 Acked Write 10 Done Done Sent false
28236 8023 Acked Flush 0 Done Done Sent false
28237 8024 Acked Flush 0 Done Done Sent false
28238 8025 Acked Write 10 Done Done Sent false
28239 8026 Acked Read 9 Done Done Sent false
28240 8027 Acked Read 3 Done Done Sent false
28241 8028 Acked Read 7 Done Done Sent false
28242 8029 Acked Write 5 Done Done Sent false
28243 8030 Acked Read 9 Done Done Sent false
28244 8031 Acked Read 5 Done Done Sent false
28245 8032 Acked Write 7 Done Done Sent false
28246 8033 Acked Write 5 Done Done Sent false
28247 8034 Acked Read 7 Done Done Sent false
28248 8035 Acked Read 8 Done Done Sent false
28249 8036 Acked Read 10 Done Done Sent false
28250 8037 Acked Read 4 Done Done Sent false
28251 8038 Acked Write 4 Done Done Sent false
28252 8039 Acked Write 9 Done Done Sent false
28253 8040 Acked Read 5 Done Done Sent false
28254 8041 Acked Read 1 Done Done Sent false
28255 8042 Acked Read 5 Done Done Sent false
28256 8043 Acked Write 2 Done Done Sent false
28257 8044 Acked Read 1 Done Done Sent false
28258 8045 Acked Flush 0 Done Done Sent false
28259 8046 Acked Read 10 Done Done Sent false
28260 8047 Acked Read 2 Done Done Sent false
28261 8048 Acked Write 2 Done Done Sent false
28262 8049 Acked Write 5 Done Done Sent false
28263 8050 Acked Read 1 Done Done Sent false
28264 8051 Acked Write 10 Done Done Sent false
28265 8052 Acked Write 9 Done Done Sent false
28266 8053 Acked Read 7 Done Done Sent false
28267 8054 Acked Write 10 Done Done Sent false
28268 8055 Acked Read 5 Done Done Sent false
28269 8056 Acked Read 5 Done Done Sent false
28270 8057 Acked Read 5 Done Done Sent false
28271 8058 Acked Write 9 Done Done Sent false
28272 8059 Acked Write 6 Done Done Sent false
28273 8060 Acked Read 2 Done Done Sent false
28274 8061 Acked Flush 0 Done Done Sent false
28275 8062 Acked Read 7 Done Done Sent false
28276 8063 Acked Read 2 Done Done Sent false
28277 8064 Acked Write 6 Done Done Sent false
28278 8065 Acked Write 2 Done Done Sent false
28279 8066 Acked Write 9 Done Done Sent false
28280 8067 Acked Read 2 Done Done Sent false
28281 8068 Acked Read 1 Done Done Sent false
28282 8069 Acked Flush 0 Done Done Sent false
28283 8070 Acked Write 5 Done Done Sent false
28284 8071 Acked Write 5 Done Done Sent false
28285 8072 Acked Read 9 Done Done Sent false
28286 8073 Acked Read 9 Done Done Sent false
28287 8074 Acked Read 4 Done Done Sent false
28288 8075 Acked Flush 0 Done Done Sent false
28289 8076 Acked Write 3 Done Done Sent false
28290 8077 Acked Write 1 Done Done Sent false
28291 8078 Acked Write 10 Done Done Sent false
28292 8079 Acked Read 2 Done Done Sent false
28293 8080 Acked Read 4 Done Done Sent false
28294 8081 Acked Read 4 Done Done Sent false
28295 8082 Acked Write 8 Done Done Sent false
28296 8083 Acked Write 7 Done Done Sent false
28297 8084 Acked Write 4 Done Done Sent false
28298 8085 Acked Read 4 Done Done Sent false
28299 8086 Acked Read 2 Done Done Sent false
28300 8087 Acked Write 4 Done Done Sent false
28301 8088 Acked Write 3 Done Done Sent false
28302 8089 Acked Read 9 Done Done Sent false
28303 8090 Acked Write 9 Done Done Sent false
28304 8091 Acked Read 2 Done Done Sent false
28305 8092 Acked Read 1 Done Done Sent false
28306 8093 Acked Read 6 Done Done Sent false
28307 8094 Acked Read 9 Done Done Sent false
28308 8095 Acked Read 6 Done Done Sent false
28309 8096 Acked Write 9 Done Done Sent false
28310 8097 Acked Write 9 Done Done Sent false
28311 8098 Acked Write 5 Done Done Sent false
28312 8099 Acked Read 8 Done Done Sent false
28313 8100 Acked Write 2 Done Done Sent false
28314 8101 Acked Read 2 Done Done Sent false
28315 8102 Acked Flush 0 Done Done Sent false
28316 8103 Acked Read 4 Done Done Sent false
28317 8104 Acked Write 6 Done Done Sent false
28318 8105 Acked Read 8 Done Done Sent false
28319 8106 Acked Write 7 Done Done Sent false
28320 8107 Acked Write 5 Done Done Sent false
28321 8108 Acked Write 3 Done Done Sent false
28322 8109 Acked Write 7 Done Done Sent false
28323 8110 Acked Read 1 Done Done Sent false
28324 8111 Acked Write 1 Done Done Sent false
28325 8112 Acked Write 9 Done Done Sent false
28326 8113 Acked Flush 0 Done Done Sent false
28327 8114 Acked Write 3 Done Done Sent false
28328 8115 Acked Write 8 Done Done Sent false
28329 8116 Acked Read 10 Done Done Sent false
28330 8117 Acked Write 5 Done Done Sent false
28331 8118 Acked Read 4 Done Done Sent false
28332 8119 Acked Read 2 Done Done Sent false
28333 8120 Acked Read 6 Done Done Sent false
28334 8121 Acked Read 5 Done Done Sent false
28335 8122 Acked Flush 0 Done Done Sent false
28336 8123 Acked Write 9 Done Done Sent false
28337 8124 Acked Read 10 Done Done Sent false
28338 8125 Acked Flush 0 Done Done Sent false
28339 8126 Acked Read 1 Done Done Sent false
28340 8127 Acked Read 2 Done Done Sent false
28341 8128 Acked Flush 0 Done Done Sent false
28342 8129 Acked Write 4 Done Done Sent false
28343 8130 Acked Write 8 Done Done Sent false
28344 8131 Acked Write 7 Done Done Sent false
28345 8132 Acked Read 6 Done Done Sent false
28346 8133 Acked Read 4 Done Done Sent false
28347 8134 Acked Flush 0 Done Done Sent false
28348 8135 Acked Read 3 Done Done Sent false
28349 8136 Acked Read 9 Done Done Sent false
28350 8137 Acked Read 3 Done Done Sent false
28351 8138 Acked Read 8 Done Done Sent false
28352 8139 Acked Write 9 Done Done Sent false
28353 8140 Acked Flush 0 Done Done Sent false
28354 8141 Acked Write 4 Done Done Sent false
28355 8142 Acked Read 9 Done Done Sent false
28356 8143 Acked Read 4 Done Done Sent false
28357 8144 Acked Flush 0 Done Done Sent false
28358 8145 Acked Read 4 Done Done Sent false
28359 8146 Acked Read 4 Done Done Sent false
28360 8147 Acked Write 4 Done Done Sent false
28361 8148 Acked Write 5 Done Done Sent false
28362 8149 Acked Read 1 Done Done Sent false
28363 8150 Acked Read 7 Done Done Sent false
28364 8151 Acked Read 7 Done Done Sent false
28365 8152 Acked Write 4 Done Done Sent false
28366 8153 Acked Write 10 Done Done Sent false
28367 8154 Acked Flush 0 Done Done Sent false
28368 8155 Acked Write 10 Done Done Sent false
28369 8156 Acked Read 2 Done Done Sent false
28370 8157 Acked Write 10 Done Done Sent false
28371 8158 Acked Read 7 Done Done Sent false
28372 8159 Acked Read 6 Done Done Sent false
28373 8160 Acked Flush 0 Done Done Sent false
28374 8161 Acked Write 8 Done Done Sent false
28375 8162 Acked Write 9 Done Done Sent false
28376 8163 Acked Write 1 Done Done Sent false
28377 8164 Acked Read 3 Done Done Sent false
28378 8165 Acked Read 7 Done Done Sent false
28379 8166 Acked Read 4 Done Done Sent false
28380 8167 Acked Write 6 Done Done Sent false
28381 8168 Acked Read 6 Done Done Sent false
28382 8169 Acked Read 5 Done Done Sent false
28383 8170 Acked Write 9 Done Done Sent false
28384 8171 Acked Read 3 Done Done Sent false
28385 8172 Acked Read 2 Done Done Sent false
28386 8173 Acked Write 7 Done Done Sent false
28387 8174 Acked Flush 0 Done Done Sent false
28388 8175 Acked Read 8 Done Done Sent false
28389 8176 Acked Write 9 Done Done Sent false
28390 8177 Acked Read 3 Done Done Sent false
28391 8178 Acked Read 7 Done Done Sent false
28392 8179 Acked Write 9 Done Done Sent false
28393 8180 Acked Read 3 Done Done Sent false
28394 8181 Acked Write 4 Done Done Sent false
28395 8182 Acked Read 2 Done Done Sent false
28396 8183 Acked Read 3 Done Done Sent false
28397 8184 Acked Write 7 Done Done Sent false
28398 8185 Acked Write 4 Done Done Sent false
28399 8186 Acked Flush 0 Done Done Sent false
28400 8187 Acked Read 6 Done Done Sent false
28401 8188 Acked Read 4 Done Done Sent false
28402 8189 Acked Read 9 Done Done Sent false
28403 8190 Acked Write 4 Done Done Sent false
28404 8191 Acked Read 3 Done Done Sent false
28405 8192 Acked Write 9 Done Done Sent false
28406 8193 Acked Read 6 Done Done Sent false
28407 8194 Acked Write 4 Done Done Sent false
28408 8195 Acked Write 10 Done Done Sent false
28409 8196 Acked Write 5 Done Done Sent false
28410 8197 Acked Read 7 Done Done Sent false
28411 8198 Acked Write 2 Done Done Sent false
28412 8199 Acked Write 4 Done Done Sent false
28413 8200 Acked Write 9 Done Done Sent false
28414 8201 Acked Read 5 Done Done Sent false
28415 8202 Acked Read 6 Done Done Sent false
28416 8203 Acked Read 5 Done Done Sent false
28417 8204 Acked Read 1 Done Done Sent false
28418 8205 Acked Write 4 Done Done Sent false
28419 8206 Acked Read 2 Done Done Sent false
28420 8207 Acked Read 9 Done Done Sent false
28421 8208 Acked Flush 0 Done Done Sent false
28422 8209 Acked Write 10 Done Done Sent false
28423 8210 Acked Read 7 Done Done Sent false
28424 8211 Acked Read 10 Done Done Sent false
28425 8212 Acked Read 1 Done Done Sent false
28426 8213 Acked Flush 0 Done Done Sent false
28427 8214 Acked Write 10 Done Done Sent false
28428 8215 Acked Read 10 Done Done Sent false
28429 8216 Acked Write 4 Done Done Sent false
28430 8217 Acked Read 1 Done Done Sent false
28431 8218 Acked Write 3 Done Done Sent false
28432 8219 Acked Read 9 Done Done Sent false
28433 8220 Acked Read 5 Done Done Sent false
28434 8221 Acked Flush 0 Done Done Sent false
28435 8222 Acked Read 1 Done Done Sent false
28436 8223 Acked Flush 0 Done Done Sent false
28437 8224 Acked Write 6 Done Done Sent false
28438 8225 Acked Write 5 Done Done Sent false
28439 8226 Acked Read 10 Done Done Sent false
28440 8227 Acked Write 4 Done Done Sent false
28441 8228 Acked Flush 0 Done Done Sent false
28442 8229 Acked Write 4 Done Done Sent false
28443 8230 Acked Write 10 Done Done Sent false
28444 8231 Acked Write 1 Done Done Sent false
28445 8232 Acked Read 8 Done Done Sent false
28446 8233 Acked Write 4 Done Done Sent false
28447 8234 Acked Write 8 Done Done Sent false
28448 8235 Acked Write 4 Done Done Sent false
28449 8236 Acked Read 2 Done Done Sent false
28450 8237 Acked Write 7 Done Done Sent false
28451 8238 Acked Read 3 Done Done Sent false
28452 8239 Acked Flush 0 Done Done Sent false
28453 8240 Acked Write 2 Done Done Sent false
28454 8241 Acked Flush 0 Done Done Sent false
28455 8242 Acked Flush 0 Done Done Sent false
28456 STATES DS:0 DS:1 DS:2 TOTAL
28457 Sent 0 0 301 301
28458 Done 301 301 0 602
28459 Skipped 0 0 0 0
28460 Error 0 0 0 0
28461 Last Flush: Some(JobId(8242)) Some(JobId(8242)) Some(JobId(7941))
28462 Downstairs last five completed: 7941 7940 7939 7938 7937
28463 Upstairs last five completed: 8242 8241 8240 8239 8238
28464 CLIENT: Up:0 ds:301 act:2
284652025-02-25T20:52:47.647ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284662025-02-25T20:52:49.011ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284672025-02-25T20:52:49.650ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284682025-02-25T20:52:50.553ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284692025-02-25T20:52:50.553ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61647) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284702025-02-25T20:52:50.554ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284712025-02-25T20:52:50.555ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284722025-02-25T20:52:50.555ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284732025-02-25T20:52:50.555ZINFOcrucible: send last flush ID to this DS: Some(JobId(7941)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284742025-02-25T20:52:50.555ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284752025-02-25T20:52:50.555ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7941)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284762025-02-25T20:52:50.555ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7941)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
28477 ----------------------------------------------------------------
28478 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28479 Downstairs last five completed: 8242 8241 8240 8239 8238
28480 Upstairs last five completed: 8242 8241 8240 8239 8238
28481 CLIENT: Up:0 ds:0 act:3
28482 Replay: All jobs finished
28483 Main task has work to do, go find it
28484 got dsc Stop(2)
28485 stop 2
28486 Feb 25 20:52:50.812 INFO request completed, [8830] Got stop action so:false kr:false
28487 latency_us: 382, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 964b14cc-44d2-41c2-9877-d6fae3923a36, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
284882025-02-25T20:52:50.813ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284892025-02-25T20:52:50.813ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
284902025-02-25T20:52:50.813ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
28491 Feb 25 20:52:50.813 INFO request completed, latency_us: 176, response_code: 200, uri: /state/cid/2, method: GET, req_id: c554afbe-91f5-445f-bce2-9edf7953ac17, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
284922025-02-25T20:52:50.813ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
28493 [8830] Exited with: ExitStatus(unix_wait_status(9))
28494 [8830][2] reports Exit
28495 Feb 25 20:52:54.815 INFO request completed, latency_us: 388, response_code: 200, uri: /state/cid/2, method: GET, req_id: ee008ba7-3634-4a4f-af97-9a231a2943fb, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
28496 001/300 Flush
28497 002/300 Read block 4577 len 40960
28498 003/300 Read block 1203 len 24576
28499 004/300 Flush
28500 005/300 Read block 2637 len 20480
28501 006/300 Write block 621 len 28672 data: 5 3 4 4 4 3 3
28502 007/300 Read block 166 len 8192
28503 008/300 Write block 4928 len 24576 data: 3 3 3 3 5 4
28504 009/300 Read block 2216 len 36864
28505 010/300 Write block 1890 len 20480 data: 7 6 6 7 7
28506 011/300 Read block 2718 len 16384
28507 012/300 Read block 1462 len 32768
28508 013/300 Flush
28509 014/300 Read block 3331 len 40960
28510 015/300 Read block 2640 len 32768
28511 016/300 Write block 1913 len 4096 data: 4
28512 017/300 Write block 4645 len 36864 data: 4 4 3 3 2 2 3 5 5
28513 018/300 Write block 3081 len 20480 data: 4 4 5 5 6
28514 019/300 Write block 1118 len 28672 data: 7 6 6 7 7 7 6
28515 020/300 Read block 4189 len 16384
28516 021/300 Read block 1557 len 36864
28517 022/300 Read block 386 len 16384
28518 023/300 Read block 2054 len 36864
28519 024/300 Write block 4713 len 40960 data: 5 4 5 7 7 6 5 6 7 7
28520 025/300 Write block 2255 len 36864 data: 9 10 8 8 9 6 7 5 5
28521 026/300 Read block 3204 len 8192
28522 027/300 Write block 1651 len 40960 data: 6 5 5 6 6 7 9 9 7 7
28523 028/300 Write block 3030 len 28672 data: 5 4 4 3 4 5 5
28524 029/300 Write block 4337 len 12288 data: 3 3 4
28525 030/300 Flush
28526 031/300 Read block 4968 len 12288
28527 032/300 Write block 1430 len 12288 data: 6 6 6
28528 033/300 Read block 2794 len 32768
28529 034/300 Read block 834 len 12288
28530 035/300 Read block 3641 len 16384
28531 036/300 Read block 1173 len 32768
28532 037/300 Write block 2999 len 28672 data: 4 4 4 4 5 5 5
28533 038/300 Read block 4252 len 16384
28534 039/300 Read block 2194 len 40960
28535 040/300 Write block 4567 len 16384 data: 3 5 5 6
28536 041/300 Read block 1968 len 32768
28537 042/300 Read block 4630 len 28672
28538 043/300 Write block 688 len 28672 data: 3 3 5 7 6 6 5
28539 044/300 Write block 3317 len 16384 data: 6 4 5 5
28540 045/300 Write block 3482 len 24576 data: 6 4 6 6 6 6
28541 046/300 Read block 2675 len 20480
28542 047/300 Read block 4026 len 24576
28543 048/300 Write block 2268 len 28672 data: 2 2 3 4 6 6 6
28544 049/300 Write block 1028 len 8192 data: 7 6
28545 050/300 Flush
28546 051/300 Flush
28547 052/300 Read block 2642 len 32768
28548 053/300 Write block 666 len 12288 data: 7 7 8
28549 054/300 Write block 1923 len 4096 data: 3
28550 055/300 Read block 3258 len 4096
28551 056/300 Write block 1433 len 20480 data: 5 5 4 5 5
28552 057/300 Read block 2397 len 40960
28553 058/300 Read block 1579 len 16384
28554 059/300 Write block 514 len 20480 data: 5 5 5 4 3
28555 060/300 Flush
28556 061/300 Read block 3504 len 12288
28557 062/300 Write block 3881 len 36864 data: 7 8 7 7 8 7 6 6 6
28558 063/300 Read block 2198 len 32768
28559 064/300 Read block 4409 len 16384
28560 065/300 Read block 3009 len 12288
28561 066/300 Read block 1723 len 12288
28562 067/300 Write block 2617 len 16384 data: 4 5 6 7
28563 068/300 Write block 3088 len 32768 data: 6 5 3 3 3 4 4 4
28564 069/300 Flush
28565 070/300 Write block 1234 len 28672 data: 4 6 6 6 7 5 5
28566 071/300 Read block 2934 len 8192
28567 072/300 Write block 3181 len 40960 data: 7 7 7 6 6 7 4 3 5 5
28568 073/300 Write block 4232 len 12288 data: 6 7 8
28569 074/300 Write block 4432 len 16384 data: 4 5 5 5
28570 075/300 Read block 3911 len 32768
28571 076/300 Read block 4493 len 40960
28572 077/300 Write block 628 len 12288 data: 3 3 4
28573 078/300 Read block 1667 len 28672
28574 079/300 Write block 1309 len 12288 data: 4 4 4
28575 080/300 Read block 752 len 12288
28576 081/300 Write block 817 len 8192 data: 6 6
28577 082/300 Write block 4246 len 40960 data: 7 4 3 4 4 4 4 5 6 6
28578 083/300 Read block 1241 len 40960
28579 084/300 Write block 2699 len 16384 data: 3 4 5 5
28580 085/300 Write block 2696 len 20480 data: 3 3 3 4 5
28581 086/300 Read block 3848 len 4096
28582 087/300 Write block 166 len 20480 data: 5 3 3 4 4
28583 088/300 Write block 3316 len 4096 data: 7
28584 089/300 Read block 1010 len 12288
28585 090/300 Write block 2077 len 24576 data: 4 4 4 4 4 5
28586 091/300 Write block 562 len 32768 data: 5 3 3 3 3 3 3 3
28587 092/300 Read block 774 len 28672
28588 093/300 Read block 1211 len 28672
28589 094/300 Write block 4390 len 40960 data: 5 7 6 6 7 7 6 6 7 7
28590 095/300 Read block 2246 len 8192
28591 096/300 Write block 2883 len 28672 data: 7 4 3 2 3 3 6
28592 097/300 Read block 2053 len 36864
28593 098/300 Read block 2181 len 4096
28594 099/300 Write block 482 len 32768 data: 5 5 6 7 7 4 6 7
28595 100/300 Read block 3550 len 8192
28596 101/300 Read block 4328 len 12288
28597 102/300 Write block 4105 len 36864 data: 4 4 4 3 3 3 3 3 3
28598 103/300 Read block 68 len 24576
28599 104/300 Read block 2395 len 32768
28600 105/300 Read block 1469 len 20480
28601 106/300 Write block 624 len 12288 data: 5 5 4
28602 107/300 Flush
28603 108/300 Write block 1992 len 40960 data: 8 7 7 4 5 5 4 4 7 6
28604 109/300 Read block 592 len 36864
28605 110/300 Flush
28606 111/300 Read block 1523 len 28672
28607 112/300 Write block 831 len 24576 data: 3 3 3 4 3 5
28608 113/300 Read block 4869 len 40960
28609 114/300 Write block 3390 len 36864 data: 5 5 5 5 5 5 5 5 4
28610 115/300 Write block 3048 len 16384 data: 4 5 5 5
28611 116/300 Write block 1608 len 4096 data: 6
28612 117/300 Read block 2878 len 8192
28613 118/300 Write block 2405 len 36864 data: 3 4 4 3 3 3 3 5 6
28614 119/300 Flush
28615 120/300 Flush
28616 121/300 Flush
28617 122/300 Flush
28618 123/300 Write block 2824 len 32768 data: 5 5 7 6 6 6 6 5
28619 124/300 Flush
28620 125/300 Read block 4748 len 20480
28621 126/300 Read block 357 len 40960
28622 127/300 Read block 4579 len 36864
28623 128/300 Write block 428 len 8192 data: 6 6
28624 129/300 Write block 4637 len 40960 data: 6 5 6 6 4 4 5 4 5 5
28625 130/300 Read block 4253 len 8192
28626 131/300 Flush
28627 132/300 Write block 3786 len 16384 data: 8 7 6 8
28628 133/300 Read block 3772 len 4096
28629 134/300 Write block 1178 len 4096 data: 4
28630 135/300 Write block 4608 len 8192 data: 4 5
28631 136/300 Read block 3772 len 32768
28632 137/300 Write block 3099 len 24576 data: 5 5 6 6 8 8
28633 138/300 Write block 40 len 20480 data: 5 5 3 2 2
28634 139/300 Read block 2229 len 36864
28635 140/300 Read block 1972 len 12288
28636 141/300 Read block 3858 len 32768
28637 142/300 Read block 699 len 36864
28638 143/300 Read block 3869 len 8192
28639 144/300 Write block 588 len 16384 data: 7 7 9 8
28640 145/300 Write block 1075 len 16384 data: 6 4 3 4
28641 146/300 Write block 3471 len 32768 data: 4 4 4 3 4 6 7 6
28642 147/300 Write block 3849 len 20480 data: 3 5 6 8 9
28643 148/300 Read block 1869 len 36864
28644 149/300 Read block 2937 len 16384
28645 150/300 Read block 3074 len 8192
28646 151/300 Read block 866 len 12288
28647 152/300 Write block 3945 len 12288 data: 6 7 6
28648 153/300 Write block 404 len 24576 data: 5 5 7 7 6 6
28649 154/300 Read block 36 len 4096
28650 155/300 Write block 4592 len 28672 data: 5 5 5 7 7 3 5
28651 156/300 Read block 3208 len 28672
28652 157/300 Write block 1652 len 24576 data: 6 6 7 7 8 10
28653 158/300 Write block 4623 len 4096 data: 3
28654 159/300 Read block 4475 len 20480
28655 160/300 Read block 3548 len 36864
28656 161/300 Read block 1939 len 36864
28657 162/300 Write block 4583 len 32768 data: 3 3 3 5 3 3 5 5
28658 163/300 Read block 4925 len 24576
28659 164/300 Write block 2106 len 20480 data: 6 7 7 5 3
28660 165/300 Read block 2540 len 40960
28661 166/300 Read block 1075 len 32768
28662 167/300 Write block 1387 len 28672 data: 8 9 7 7 7 6 7
28663 168/300 Write block 3235 len 8192 data: 4 5
28664 169/300 Read block 2023 len 16384
28665 170/300 Read block 3164 len 16384
28666 171/300 Read block 2444 len 4096
28667 172/300 Write block 2951 len 40960 data: 4 4 5 5 5 7 9 11 9 8
28668 173/300 Read block 3643 len 16384
28669 174/300 Write block 3940 len 36864 data: 2 2 4 4 5 7 8 7 6
28670 175/300 Read block 948 len 20480
28671 176/300 Read block 665 len 36864
28672 177/300 Write block 3944 len 24576 data: 6 8 9 8 7 6
28673 178/300 Write block 4495 len 4096 data: 3
28674 179/300 Read block 2912 len 28672
28675 180/300 Read block 3749 len 16384
28676 181/300 Write block 4575 len 16384 data: 5 4 5 5
28677 182/300 Read block 4361 len 8192
28678 183/300 Read block 3231 len 20480
28679 184/300 Read block 136 len 8192
28680 185/300 Read block 2905 len 24576
28681 186/300 Read block 2586 len 36864
28682 187/300 Read block 4733 len 4096
28683 188/300 Write block 3643 len 16384 data: 8 9 7 9
28684 189/300 Write block 918 len 40960 data: 7 8 8 8 8 8 7 6 7 6
28685 190/300 Write block 382 len 24576 data: 3 6 6 6 6 6
28686 191/300 Read block 944 len 12288
28687 192/300 Write block 4672 len 20480 data: 4 5 5 5 5
28688 193/300 Write block 3586 len 32768 data: 5 5 5 5 4 4 2 4
28689 194/300 Read block 3920 len 32768
28690 195/300 Read block 4507 len 36864
28691 196/300 Flush
28692 197/300 Write block 2006 len 4096 data: 5
28693 198/300 Write block 4 len 24576 data: 5 5 7 7 7 8
28694 199/300 Write block 330 len 4096 data: 4
28695 200/300 Read block 1951 len 28672
28696 201/300 Read block 1903 len 12288
28697 202/300 Read block 3174 len 36864
28698 203/300 Write block 169 len 32768 data: 5 5 5 6 8 9 8 9
28699 204/300 Read block 2157 len 20480
28700 205/300 Flush
28701 206/300 Flush
28702 207/300 Write block 2446 len 32768 data: 8 8 7 8 10 10 8 9
28703 208/300 Write block 3740 len 4096 data: 4
28704 209/300 Write block 854 len 4096 data: 7
28705 210/300 Read block 3700 len 4096
28706 211/300 Write block 3766 len 28672 data: 7 5 3 5 4 3 4
28707 212/300 Read block 3490 len 16384
28708 213/300 Read block 2803 len 32768
28709 214/300 Read block 1606 len 28672
28710 215/300 Write block 458 len 12288 data: 3 3 3
28711 216/300 Read block 1747 len 32768
28712 217/300 Write block 4745 len 4096 data: 6
28713 218/300 Read block 3815 len 20480
28714 219/300 Read block 791 len 36864
28715 220/300 Read block 3100 len 28672
28716 221/300 Read block 4249 len 36864
28717 222/300 Read block 2353 len 40960
28718 223/300 Write block 2283 len 12288 data: 7 7 8
28719 224/300 Write block 1382 len 16384 data: 4 4 4 6
28720 225/300 Read block 3994 len 20480
28721 226/300 Read block 3650 len 40960
28722 227/300 Flush
28723 228/300 Flush
28724 229/300 Read block 525 len 16384
28725 230/300 Write block 1501 len 12288 data: 3 3 3
28726 231/300 Read block 4657 len 40960
28727 232/300 Write block 1121 len 32768 data: 8 8 8 7 5 5 5 4
28728 233/300 Read block 996 len 36864
28729 234/300 Write block 3068 len 20480 data: 7 6 6 5 4
28730 235/300 Flush
28731 236/300 Read block 4990 len 32768
28732 237/300 Write block 2300 len 24576 data: 3 3 4 4 3 4
28733 238/300 Write block 4292 len 8192 data: 5 5
28734 239/300 Write block 2540 len 40960 data: 4 4 4 5 5 7 7 5 6 6
28735 240/300 Flush
28736 241/300 Read block 3991 len 24576
28737 242/300 Write block 3904 len 32768 data: 8 8 7 7 6 5 2 2
28738 243/300 Read block 415 len 24576
28739 244/300 Read block 3407 len 36864
28740 245/300 Flush
28741 246/300 Flush
28742 247/300 Read block 4058 len 24576
28743 248/300 Read block 884 len 40960
28744 249/300 Read block 847 len 36864
28745 250/300 Read block 3383 len 4096
28746 251/300 Write block 2682 len 16384 data: 8 8 8 5
28747 252/300 Write block 2301 len 28672 data: 4 5 5 4 5 5 6
28748 253/300 Read block 468 len 20480
28749 254/300 Flush
28750 255/300 Write block 1761 len 16384 data: 5 5 5 5
28751 256/300 Flush
28752 257/300 Read block 3858 len 8192
28753 258/300 Read block 35 len 28672
28754 259/300 Write block 4534 len 16384 data: 5 4 4 3
28755 260/300 Write block 4033 len 4096 data: 5
28756 261/300 Write block 4036 len 28672 data: 4 5 5 4 4 5 3
28757 262/300 Read block 4431 len 20480
28758 263/300 Read block 3681 len 36864
28759 264/300 Write block 931 len 32768 data: 5 5 5 6 4 5 5 3
28760 265/300 Read block 882 len 24576
28761 266/300 Write block 2133 len 8192 data: 5 5
28762 267/300 Read block 1610 len 36864
28763 268/300 Read block 1349 len 36864
28764 269/300 Read block 2548 len 4096
28765 270/300 Read block 2593 len 36864
28766 271/300 Flush
28767 272/300 Write block 4492 len 16384 data: 3 3 3 4
28768 273/300 Write block 2020 len 16384 data: 6 7 5 3
28769 274/300 Read block 1557 len 4096
28770 275/300 Flush
28771 276/300 Read block 4370 len 4096
28772 277/300 Read block 679 len 24576
28773 278/300 Read block 4691 len 16384
28774 279/300 Write block 3398 len 20480 data: 5 5 4 4 4
28775 280/300 Read block 2765 len 28672
28776 281/300 Write block 532 len 8192 data: 7 6
28777 282/300 Write block 904 len 16384 data: 5 5 3 3
28778 283/300 Write block 2277 len 40960 data: 4 4 3 3 4 5 8 8 9 7
28779 284/300 Read block 3056 len 16384
28780 285/300 Write block 3088 len 40960 data: 7 6 4 4 4 5 5 5 4 4
28781 286/300 Write block 1648 len 40960 data: 4 5 4 7 7 7 8 8 9 11
28782 287/300 Flush
28783 288/300 Write block 3886 len 4096 data: 8
28784 289/300 Read block 4191 len 24576
28785 290/300 Read block 524 len 4096
28786 291/300 Write block 3715 len 32768 data: 8 7 6 7 8 9 7 9
28787 292/300 Flush
28788 293/300 Read block 1561 len 4096
28789 294/300 Write block 236 len 32768 data: 4 5 5 6 5 5 4 4
28790 295/300 Read block 2777 len 40960
28791 296/300 Write block 2913 len 12288 data: 4 5 5
28792 297/300 Flush
28793 298/300 Read block 321 len 12288
28794 299/300 Write block 1030 len 32768 data: 6 4 4 5 5 4 5 6
28795 300/300 Read block 1274 len 40960
28796 301/300 Read block 2479 len 8192
28797 Main task has work to do, go find it
28798 got dsc Start(2)
28799 start 2
28800 Feb 25 20:52:55.065 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: 50ee547b-6736-4447-8d5a-f8400cf97b4e, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
28801 Replay: started 2, returned:Ok(())
28802 ----------------------------------------------------------------
28803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28804 States: Active Active Offline
28805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28806 8243 Acked Flush 0 Done Done Sent false
28807 8244 Acked Read 10 Done Done Sent false
28808 8245 Acked Read 6 Done Done Sent false
28809 8246 Acked Flush 0 Done Done Sent false
28810 8247 Acked Read 5 Done Done Sent false
28811 8248 Acked Write 7 Done Done Sent false
28812 8249 Acked Read 2 Done Done Sent false
28813 8250 Acked Write 6 Done Done Sent false
28814 8251 Acked Read 9 Done Done Sent false
28815 8252 Acked Write 5 Done Done Sent false
28816 8253 Acked Read 4 Done Done Sent false
28817 8254 Acked Read 8 Done Done Sent false
28818 8255 Acked Flush 0 Done Done Sent false
28819 8256 Acked Read 10 Done Done Sent false
28820 8257 Acked Read 8 Done Done Sent false
28821 8258 Acked Write 1 Done Done Sent false
28822 8259 Acked Write 9 Done Done Sent false
28823 8260 Acked Write 5 Done Done Sent false
28824 8261 Acked Write 7 Done Done Sent false
28825 8262 Acked Read 4 Done Done Sent false
28826 8263 Acked Read 9 Done Done Sent false
28827 8264 Acked Read 4 Done Done Sent false
28828 8265 Acked Read 9 Done Done Sent false
28829 8266 Acked Write 10 Done Done Sent false
28830 8267 Acked Write 9 Done Done Sent false
28831 8268 Acked Read 2 Done Done Sent false
28832 8269 Acked Write 10 Done Done Sent false
28833 8270 Acked Write 7 Done Done Sent false
28834 8271 Acked Write 3 Done Done Sent false
28835 8272 Acked Flush 0 Done Done Sent false
28836 8273 Acked Read 3 Done Done Sent false
28837 8274 Acked Write 3 Done Done Sent false
28838 8275 Acked Read 8 Done Done Sent false
28839 8276 Acked Read 3 Done Done Sent false
28840 8277 Acked Read 4 Done Done Sent false
28841 8278 Acked Read 8 Done Done Sent false
28842 8279 Acked Write 7 Done Done Sent false
28843 8280 Acked Read 4 Done Done Sent false
28844 8281 Acked Read 10 Done Done Sent false
28845 8282 Acked Write 4 Done Done Sent false
28846 8283 Acked Read 8 Done Done Sent false
28847 8284 Acked Read 7 Done Done Sent false
28848 8285 Acked Write 7 Done Done Sent false
28849 8286 Acked Write 4 Done Done Sent false
28850 8287 Acked Write 6 Done Done Sent false
28851 8288 Acked Read 5 Done Done Sent false
28852 8289 Acked Read 6 Done Done Sent false
28853 8290 Acked Write 7 Done Done Sent false
28854 8291 Acked Write 2 Done Done Sent false
28855 8292 Acked Flush 0 Done Done Sent false
28856 8293 Acked Flush 0 Done Done Sent false
28857 8294 Acked Read 8 Done Done Sent false
28858 8295 Acked Write 3 Done Done Sent false
28859 8296 Acked Write 1 Done Done Sent false
28860 8297 Acked Read 1 Done Done Sent false
28861 8298 Acked Write 5 Done Done Sent false
28862 8299 Acked Read 10 Done Done Sent false
28863 8300 Acked Read 4 Done Done Sent false
28864 8301 Acked Write 5 Done Done Sent false
28865 8302 Acked Flush 0 Done Done Sent false
28866 8303 Acked Read 3 Done Done Sent false
28867 8304 Acked Write 9 Done Done Sent false
28868 8305 Acked Read 8 Done Done Sent false
28869 8306 Acked Read 4 Done Done Sent false
28870 8307 Acked Read 3 Done Done Sent false
28871 8308 Acked Read 3 Done Done Sent false
28872 8309 Acked Write 4 Done Done Sent false
28873 8310 Acked Write 8 Done Done Sent false
28874 8311 Acked Flush 0 Done Done Sent false
28875 8312 Acked Write 7 Done Done Sent false
28876 8313 Acked Read 2 Done Done Sent false
28877 8314 Acked Write 10 Done Done Sent false
28878 8315 Acked Write 3 Done Done Sent false
28879 8316 Acked Write 4 Done Done Sent false
28880 8317 Acked Read 8 Done Done Sent false
28881 8318 Acked Read 10 Done Done Sent false
28882 8319 Acked Write 3 Done Done Sent false
28883 8320 Acked Read 7 Done Done Sent false
28884 8321 Acked Write 3 Done Done Sent false
28885 8322 Acked Read 3 Done Done Sent false
28886 8323 Acked Write 2 Done Done Sent false
28887 8324 Acked Write 10 Done Done Sent false
28888 8325 Acked Read 10 Done Done Sent false
28889 8326 Acked Write 4 Done Done Sent false
28890 8327 Acked Write 5 Done Done Sent false
28891 8328 Acked Read 1 Done Done Sent false
28892 8329 Acked Write 5 Done Done Sent false
28893 8330 Acked Write 1 Done Done Sent false
28894 8331 Acked Read 3 Done Done Sent false
28895 8332 Acked Write 6 Done Done Sent false
28896 8333 Acked Write 8 Done Done Sent false
28897 8334 Acked Read 7 Done Done Sent false
28898 8335 Acked Read 7 Done Done Sent false
28899 8336 Acked Write 10 Done Done Sent false
28900 8337 Acked Read 2 Done Done Sent false
28901 8338 Acked Write 7 Done Done Sent false
28902 8339 Acked Read 9 Done Done Sent false
28903 8340 Acked Read 1 Done Done Sent false
28904 8341 Acked Write 8 Done Done Sent false
28905 8342 Acked Read 2 Done Done Sent false
28906 8343 Acked Read 3 Done Done Sent false
28907 8344 Acked Write 9 Done Done Sent false
28908 8345 Acked Read 6 Done Done Sent false
28909 8346 Acked Read 8 Done Done Sent false
28910 8347 Acked Read 5 Done Done Sent false
28911 8348 Acked Write 3 Done Done Sent false
28912 8349 Acked Flush 0 Done Done Sent false
28913 8350 Acked Write 10 Done Done Sent false
28914 8351 Acked Read 9 Done Done Sent false
28915 8352 Acked Flush 0 Done Done Sent false
28916 8353 Acked Read 7 Done Done Sent false
28917 8354 Acked Write 6 Done Done Sent false
28918 8355 Acked Read 10 Done Done Sent false
28919 8356 Acked Write 9 Done Done Sent false
28920 8357 Acked Write 4 Done Done Sent false
28921 8358 Acked Write 1 Done Done Sent false
28922 8359 Acked Read 2 Done Done Sent false
28923 8360 Acked Write 9 Done Done Sent false
28924 8361 Acked Flush 0 Done Done Sent false
28925 8362 Acked Flush 0 Done Done Sent false
28926 8363 Acked Flush 0 Done Done Sent false
28927 8364 Acked Flush 0 Done Done Sent false
28928 8365 Acked Write 8 Done Done Sent false
28929 8366 Acked Flush 0 Done Done Sent false
28930 8367 Acked Read 5 Done Done Sent false
28931 8368 Acked Read 10 Done Done Sent false
28932 8369 Acked Read 9 Done Done Sent false
28933 8370 Acked Write 2 Done Done Sent false
28934 8371 Acked Write 10 Done Done Sent false
28935 8372 Acked Read 2 Done Done Sent false
28936 8373 Acked Flush 0 Done Done Sent false
28937 8374 Acked Write 4 Done Done Sent false
28938 8375 Acked Read 1 Done Done Sent false
28939 8376 Acked Write 1 Done Done Sent false
28940 8377 Acked Write 2 Done Done Sent false
28941 8378 Acked Read 8 Done Done Sent false
28942 8379 Acked Write 6 Done Done Sent false
28943 8380 Acked Write 5 Done Done Sent false
28944 8381 Acked Read 9 Done Done Sent false
28945 8382 Acked Read 3 Done Done Sent false
28946 8383 Acked Read 8 Done Done Sent false
28947 8384 Acked Read 9 Done Done Sent false
28948 8385 Acked Read 2 Done Done Sent false
28949 8386 Acked Write 4 Done Done Sent false
28950 8387 Acked Write 4 Done Done Sent false
28951 8388 Acked Write 8 Done Done Sent false
28952 8389 Acked Write 5 Done Done Sent false
28953 8390 Acked Read 9 Done Done Sent false
28954 8391 Acked Read 4 Done Done Sent false
28955 8392 Acked Read 2 Done Done Sent false
28956 8393 Acked Read 3 Done Done Sent false
28957 8394 Acked Write 3 Done Done Sent false
28958 8395 Acked Write 6 Done Done Sent false
28959 8396 Acked Read 1 Done Done Sent false
28960 8397 Acked Write 7 Done Done Sent false
28961 8398 Acked Read 7 Done Done Sent false
28962 8399 Acked Write 6 Done Done Sent false
28963 8400 Acked Write 1 Done Done Sent false
28964 8401 Acked Read 5 Done Done Sent false
28965 8402 Acked Read 9 Done Done Sent false
28966 8403 Acked Read 9 Done Done Sent false
28967 8404 Acked Write 8 Done Done Sent false
28968 8405 Acked Read 6 Done Done Sent false
28969 8406 Acked Write 5 Done Done Sent false
28970 8407 Acked Read 10 Done Done Sent false
28971 8408 Acked Read 8 Done Done Sent false
28972 8409 Acked Write 7 Done Done Sent false
28973 8410 Acked Write 2 Done Done Sent false
28974 8411 Acked Read 4 Done Done Sent false
28975 8412 Acked Read 4 Done Done Sent false
28976 8413 Acked Read 1 Done Done Sent false
28977 8414 Acked Write 10 Done Done Sent false
28978 8415 Acked Read 4 Done Done Sent false
28979 8416 Acked Write 9 Done Done Sent false
28980 8417 Acked Read 5 Done Done Sent false
28981 8418 Acked Read 9 Done Done Sent false
28982 8419 Acked Write 6 Done Done Sent false
28983 8420 Acked Write 1 Done Done Sent false
28984 8421 Acked Read 7 Done Done Sent false
28985 8422 Acked Read 4 Done Done Sent false
28986 8423 Acked Write 4 Done Done Sent false
28987 8424 Acked Read 2 Done Done Sent false
28988 8425 Acked Read 5 Done Done Sent false
28989 8426 Acked Read 2 Done Done Sent false
28990 8427 Acked Read 6 Done Done Sent false
28991 8428 Acked Read 9 Done Done Sent false
28992 8429 Acked Read 1 Done Done Sent false
28993 8430 Acked Write 4 Done Done Sent false
28994 8431 Acked Write 10 Done Done Sent false
28995 8432 Acked Write 6 Done Done Sent false
28996 8433 Acked Read 3 Done Done Sent false
28997 8434 Acked Write 5 Done Done Sent false
28998 8435 Acked Write 8 Done Done Sent false
28999 8436 Acked Read 8 Done Done Sent false
29000 8437 Acked Read 9 Done Done Sent false
29001 8438 Acked Flush 0 Done Done Sent false
29002 8439 Acked Write 1 Done Done Sent false
29003 8440 Acked Write 6 Done Done Sent false
29004 8441 Acked Write 1 Done Done Sent false
29005 8442 Acked Read 7 Done Done Sent false
29006 8443 Acked Read 3 Done Done Sent false
29007 8444 Acked Read 9 Done Done Sent false
29008 8445 Acked Write 8 Done Done Sent false
29009 8446 Acked Read 5 Done Done Sent false
29010 8447 Acked Flush 0 Done Done Sent false
29011 8448 Acked Flush 0 Done Done Sent false
29012 8449 Acked Write 8 Done Done Sent false
29013 8450 Acked Write 1 Done Done Sent false
29014 8451 Acked Write 1 Done Done Sent false
29015 8452 Acked Read 1 Done Done Sent false
29016 8453 Acked Write 7 Done Done Sent false
29017 8454 Acked Read 4 Done Done Sent false
29018 8455 Acked Read 8 Done Done Sent false
29019 8456 Acked Read 7 Done Done Sent false
29020 8457 Acked Write 3 Done Done Sent false
29021 8458 Acked Read 8 Done Done Sent false
29022 8459 Acked Write 1 Done Done Sent false
29023 8460 Acked Read 5 Done Done Sent false
29024 8461 Acked Read 9 Done Done Sent false
29025 8462 Acked Read 7 Done Done Sent false
29026 8463 Acked Read 9 Done Done Sent false
29027 8464 Acked Read 10 Done Done Sent false
29028 8465 Acked Write 3 Done Done Sent false
29029 8466 Acked Write 4 Done Done Sent false
29030 8467 Acked Read 5 Done Done Sent false
29031 8468 Acked Read 10 Done Done Sent false
29032 8469 Acked Flush 0 Done Done Sent false
29033 8470 Acked Flush 0 Done Done Sent false
29034 8471 Acked Read 4 Done Done Sent false
29035 8472 Acked Write 3 Done Done Sent false
29036 8473 Acked Read 10 Done Done Sent false
29037 8474 Acked Write 8 Done Done Sent false
29038 8475 Acked Read 9 Done Done Sent false
29039 8476 Acked Write 5 Done Done Sent false
29040 8477 Acked Flush 0 Done Done Sent false
29041 8478 Acked Read 8 Done Done Sent false
29042 8479 Acked Write 6 Done Done Sent false
29043 8480 Acked Write 2 Done Done Sent false
29044 8481 Acked Write 10 Done Done Sent false
29045 8482 Acked Flush 0 Done Done Sent false
29046 8483 Acked Read 6 Done Done Sent false
29047 8484 Acked Write 8 Done Done Sent false
29048 8485 Acked Read 6 Done Done Sent false
29049 8486 Acked Read 9 Done Done Sent false
29050 8487 Acked Flush 0 Done Done Sent false
29051 8488 Acked Flush 0 Done Done Sent false
29052 8489 Acked Read 6 Done Done Sent false
29053 8490 Acked Read 10 Done Done Sent false
29054 8491 Acked Read 9 Done Done Sent false
29055 8492 Acked Read 1 Done Done Sent false
29056 8493 Acked Write 4 Done Done Sent false
29057 8494 Acked Write 7 Done Done Sent false
29058 8495 Acked Read 5 Done Done Sent false
29059 8496 Acked Flush 0 Done Done Sent false
29060 8497 Acked Write 4 Done Done Sent false
29061 8498 Acked Flush 0 Done Done Sent false
29062 8499 Acked Read 2 Done Done Sent false
29063 8500 Acked Read 7 Done Done Sent false
29064 8501 Acked Write 4 Done Done Sent false
29065 8502 Acked Write 1 Done Done Sent false
29066 8503 Acked Write 7 Done Done Sent false
29067 8504 Acked Read 5 Done Done Sent false
29068 8505 Acked Read 9 Done Done Sent false
29069 8506 Acked Write 8 Done Done Sent false
29070 8507 Acked Read 6 Done Done Sent false
29071 8508 Acked Write 2 Done Done Sent false
29072 8509 Acked Read 9 Done Done Sent false
29073 8510 Acked Read 9 Done Done Sent false
29074 8511 Acked Read 1 Done Done Sent false
29075 8512 Acked Read 9 Done Done Sent false
29076 8513 Acked Flush 0 Done Done Sent false
29077 8514 Acked Write 4 Done Done Sent false
29078 8515 Acked Write 4 Done Done Sent false
29079 8516 Acked Read 1 Done Done Sent false
29080 8517 Acked Flush 0 Done Done Sent false
29081 8518 Acked Read 1 Done Done Sent false
29082 8519 Acked Read 6 Done Done Sent false
29083 8520 Acked Read 4 Done Done Sent false
29084 8521 Acked Write 5 Done Done Sent false
29085 8522 Acked Read 7 Done Done Sent false
29086 8523 Acked Write 2 Done Done Sent false
29087 8524 Acked Write 4 Done Done Sent false
29088 8525 Acked Write 10 Done Done Sent false
29089 8526 Acked Read 4 Done Done Sent false
29090 8527 Acked Write 10 Done Done Sent false
29091 8528 Acked Write 10 Done Done Sent false
29092 8529 Acked Flush 0 Done Done Sent false
29093 8530 Acked Write 1 Done Done Sent false
29094 8531 Acked Read 6 Done Done Sent false
29095 8532 Acked Read 1 Done Done Sent false
29096 8533 Acked Write 8 Done Done Sent false
29097 8534 Acked Flush 0 Done Done Sent false
29098 8535 Acked Read 1 Done Done Sent false
29099 8536 Acked Write 8 Done Done Sent false
29100 8537 Acked Read 10 Done Done Sent false
29101 8538 Acked Write 3 Done Done Sent false
29102 8539 Acked Flush 0 Done Done Sent false
29103 8540 Acked Read 3 Done Done Sent false
29104 8541 Acked Write 8 Done Done Sent false
29105 8542 Acked Read 10 Done Done Sent false
29106 8543 Acked Read 2 Done Done Sent false
29107 STATES DS:0 DS:1 DS:2 TOTAL
29108 Sent 0 0 301 301
29109 Done 301 301 0 602
29110 Skipped 0 0 0 0
29111 Error 0 0 0 0
29112 Last Flush: Some(JobId(8539)) Some(JobId(8539)) Some(JobId(8242))
29113 Downstairs last five completed: 8242 8241 8240 8239 8238
29114 Upstairs last five completed: 8543 8542 8541 8540 8539
29115 CLIENT: Up:0 ds:301 act:2
291162025-02-25T20:52:55.660ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
291172025-02-25T20:52:56.660ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
29118 [8830] got start action
29119 [8830] I am going to restart
29120 Starting downstairs at port 8830
29121 Make output file at "/tmp/dsc/downstairs-8830.txt"
29122 [8830][2] reports Starting
29123 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
29124 [8830][2] reports Running
291252025-02-25T20:52:57.661ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
291262025-02-25T20:52:59.013ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
29127 ----------------------------------------------------------------
29128 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29129 States: Active Active Offline
29130 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29131 8243 Acked Flush 0 Done Done Sent false
29132 8244 Acked Read 10 Done Done Sent false
29133 8245 Acked Read 6 Done Done Sent false
29134 8246 Acked Flush 0 Done Done Sent false
29135 8247 Acked Read 5 Done Done Sent false
29136 8248 Acked Write 7 Done Done Sent false
29137 8249 Acked Read 2 Done Done Sent false
29138 8250 Acked Write 6 Done Done Sent false
29139 8251 Acked Read 9 Done Done Sent false
29140 8252 Acked Write 5 Done Done Sent false
29141 8253 Acked Read 4 Done Done Sent false
29142 8254 Acked Read 8 Done Done Sent false
29143 8255 Acked Flush 0 Done Done Sent false
29144 8256 Acked Read 10 Done Done Sent false
29145 8257 Acked Read 8 Done Done Sent false
29146 8258 Acked Write 1 Done Done Sent false
29147 8259 Acked Write 9 Done Done Sent false
29148 8260 Acked Write 5 Done Done Sent false
29149 8261 Acked Write 7 Done Done Sent false
29150 8262 Acked Read 4 Done Done Sent false
29151 8263 Acked Read 9 Done Done Sent false
29152 8264 Acked Read 4 Done Done Sent false
29153 8265 Acked Read 9 Done Done Sent false
29154 8266 Acked Write 10 Done Done Sent false
29155 8267 Acked Write 9 Done Done Sent false
29156 8268 Acked Read 2 Done Done Sent false
29157 8269 Acked Write 10 Done Done Sent false
29158 8270 Acked Write 7 Done Done Sent false
29159 8271 Acked Write 3 Done Done Sent false
29160 8272 Acked Flush 0 Done Done Sent false
29161 8273 Acked Read 3 Done Done Sent false
29162 8274 Acked Write 3 Done Done Sent false
29163 8275 Acked Read 8 Done Done Sent false
29164 8276 Acked Read 3 Done Done Sent false
29165 8277 Acked Read 4 Done Done Sent false
29166 8278 Acked Read 8 Done Done Sent false
29167 8279 Acked Write 7 Done Done Sent false
29168 8280 Acked Read 4 Done Done Sent false
29169 8281 Acked Read 10 Done Done Sent false
29170 8282 Acked Write 4 Done Done Sent false
29171 8283 Acked Read 8 Done Done Sent false
29172 8284 Acked Read 7 Done Done Sent false
29173 8285 Acked Write 7 Done Done Sent false
29174 8286 Acked Write 4 Done Done Sent false
29175 8287 Acked Write 6 Done Done Sent false
29176 8288 Acked Read 5 Done Done Sent false
29177 8289 Acked Read 6 Done Done Sent false
29178 8290 Acked Write 7 Done Done Sent false
29179 8291 Acked Write 2 Done Done Sent false
29180 8292 Acked Flush 0 Done Done Sent false
29181 8293 Acked Flush 0 Done Done Sent false
29182 8294 Acked Read 8 Done Done Sent false
29183 8295 Acked Write 3 Done Done Sent false
29184 8296 Acked Write 1 Done Done Sent false
29185 8297 Acked Read 1 Done Done Sent false
29186 8298 Acked Write 5 Done Done Sent false
29187 8299 Acked Read 10 Done Done Sent false
29188 8300 Acked Read 4 Done Done Sent false
29189 8301 Acked Write 5 Done Done Sent false
29190 8302 Acked Flush 0 Done Done Sent false
29191 8303 Acked Read 3 Done Done Sent false
29192 8304 Acked Write 9 Done Done Sent false
29193 8305 Acked Read 8 Done Done Sent false
29194 8306 Acked Read 4 Done Done Sent false
29195 8307 Acked Read 3 Done Done Sent false
29196 8308 Acked Read 3 Done Done Sent false
29197 8309 Acked Write 4 Done Done Sent false
29198 8310 Acked Write 8 Done Done Sent false
29199 8311 Acked Flush 0 Done Done Sent false
29200 8312 Acked Write 7 Done Done Sent false
29201 8313 Acked Read 2 Done Done Sent false
29202 8314 Acked Write 10 Done Done Sent false
29203 8315 Acked Write 3 Done Done Sent false
29204 8316 Acked Write 4 Done Done Sent false
29205 8317 Acked Read 8 Done Done Sent false
29206 8318 Acked Read 10 Done Done Sent false
29207 8319 Acked Write 3 Done Done Sent false
29208 8320 Acked Read 7 Done Done Sent false
29209 8321 Acked Write 3 Done Done Sent false
29210 8322 Acked Read 3 Done Done Sent false
29211 8323 Acked Write 2 Done Done Sent false
29212 8324 Acked Write 10 Done Done Sent false
29213 8325 Acked Read 10 Done Done Sent false
29214 8326 Acked Write 4 Done Done Sent false
29215 8327 Acked Write 5 Done Done Sent false
29216 8328 Acked Read 1 Done Done Sent false
29217 8329 Acked Write 5 Done Done Sent false
29218 8330 Acked Write 1 Done Done Sent false
29219 8331 Acked Read 3 Done Done Sent false
29220 8332 Acked Write 6 Done Done Sent false
29221 8333 Acked Write 8 Done Done Sent false
29222 8334 Acked Read 7 Done Done Sent false
29223 8335 Acked Read 7 Done Done Sent false
29224 8336 Acked Write 10 Done Done Sent false
29225 8337 Acked Read 2 Done Done Sent false
29226 8338 Acked Write 7 Done Done Sent false
29227 8339 Acked Read 9 Done Done Sent false
29228 8340 Acked Read 1 Done Done Sent false
29229 8341 Acked Write 8 Done Done Sent false
29230 8342 Acked Read 2 Done Done Sent false
29231 8343 Acked Read 3 Done Done Sent false
29232 8344 Acked Write 9 Done Done Sent false
29233 8345 Acked Read 6 Done Done Sent false
29234 8346 Acked Read 8 Done Done Sent false
29235 8347 Acked Read 5 Done Done Sent false
29236 8348 Acked Write 3 Done Done Sent false
29237 8349 Acked Flush 0 Done Done Sent false
29238 8350 Acked Write 10 Done Done Sent false
29239 8351 Acked Read 9 Done Done Sent false
29240 8352 Acked Flush 0 Done Done Sent false
29241 8353 Acked Read 7 Done Done Sent false
29242 8354 Acked Write 6 Done Done Sent false
29243 8355 Acked Read 10 Done Done Sent false
29244 8356 Acked Write 9 Done Done Sent false
29245 8357 Acked Write 4 Done Done Sent false
29246 8358 Acked Write 1 Done Done Sent false
29247 8359 Acked Read 2 Done Done Sent false
29248 8360 Acked Write 9 Done Done Sent false
29249 8361 Acked Flush 0 Done Done Sent false
29250 8362 Acked Flush 0 Done Done Sent false
29251 8363 Acked Flush 0 Done Done Sent false
29252 8364 Acked Flush 0 Done Done Sent false
29253 8365 Acked Write 8 Done Done Sent false
29254 8366 Acked Flush 0 Done Done Sent false
29255 8367 Acked Read 5 Done Done Sent false
29256 8368 Acked Read 10 Done Done Sent false
29257 8369 Acked Read 9 Done Done Sent false
29258 8370 Acked Write 2 Done Done Sent false
29259 8371 Acked Write 10 Done Done Sent false
29260 8372 Acked Read 2 Done Done Sent false
29261 8373 Acked Flush 0 Done Done Sent false
29262 8374 Acked Write 4 Done Done Sent false
29263 8375 Acked Read 1 Done Done Sent false
29264 8376 Acked Write 1 Done Done Sent false
29265 8377 Acked Write 2 Done Done Sent false
29266 8378 Acked Read 8 Done Done Sent false
29267 8379 Acked Write 6 Done Done Sent false
29268 8380 Acked Write 5 Done Done Sent false
29269 8381 Acked Read 9 Done Done Sent false
29270 8382 Acked Read 3 Done Done Sent false
29271 8383 Acked Read 8 Done Done Sent false
29272 8384 Acked Read 9 Done Done Sent false
29273 8385 Acked Read 2 Done Done Sent false
29274 8386 Acked Write 4 Done Done Sent false
29275 8387 Acked Write 4 Done Done Sent false
29276 8388 Acked Write 8 Done Done Sent false
29277 8389 Acked Write 5 Done Done Sent false
29278 8390 Acked Read 9 Done Done Sent false
29279 8391 Acked Read 4 Done Done Sent false
29280 8392 Acked Read 2 Done Done Sent false
29281 8393 Acked Read 3 Done Done Sent false
29282 8394 Acked Write 3 Done Done Sent false
29283 8395 Acked Write 6 Done Done Sent false
29284 8396 Acked Read 1 Done Done Sent false
29285 8397 Acked Write 7 Done Done Sent false
29286 8398 Acked Read 7 Done Done Sent false
29287 8399 Acked Write 6 Done Done Sent false
29288 8400 Acked Write 1 Done Done Sent false
29289 8401 Acked Read 5 Done Done Sent false
29290 8402 Acked Read 9 Done Done Sent false
29291 8403 Acked Read 9 Done Done Sent false
29292 8404 Acked Write 8 Done Done Sent false
29293 8405 Acked Read 6 Done Done Sent false
29294 8406 Acked Write 5 Done Done Sent false
29295 8407 Acked Read 10 Done Done Sent false
29296 8408 Acked Read 8 Done Done Sent false
29297 8409 Acked Write 7 Done Done Sent false
29298 8410 Acked Write 2 Done Done Sent false
29299 8411 Acked Read 4 Done Done Sent false
29300 8412 Acked Read 4 Done Done Sent false
29301 8413 Acked Read 1 Done Done Sent false
29302 8414 Acked Write 10 Done Done Sent false
29303 8415 Acked Read 4 Done Done Sent false
29304 8416 Acked Write 9 Done Done Sent false
29305 8417 Acked Read 5 Done Done Sent false
29306 8418 Acked Read 9 Done Done Sent false
29307 8419 Acked Write 6 Done Done Sent false
29308 8420 Acked Write 1 Done Done Sent false
29309 8421 Acked Read 7 Done Done Sent false
29310 8422 Acked Read 4 Done Done Sent false
29311 8423 Acked Write 4 Done Done Sent false
29312 8424 Acked Read 2 Done Done Sent false
29313 8425 Acked Read 5 Done Done Sent false
29314 8426 Acked Read 2 Done Done Sent false
29315 8427 Acked Read 6 Done Done Sent false
29316 8428 Acked Read 9 Done Done Sent false
29317 8429 Acked Read 1 Done Done Sent false
29318 8430 Acked Write 4 Done Done Sent false
29319 8431 Acked Write 10 Done Done Sent false
29320 8432 Acked Write 6 Done Done Sent false
29321 8433 Acked Read 3 Done Done Sent false
29322 8434 Acked Write 5 Done Done Sent false
29323 8435 Acked Write 8 Done Done Sent false
29324 8436 Acked Read 8 Done Done Sent false
29325 8437 Acked Read 9 Done Done Sent false
29326 8438 Acked Flush 0 Done Done Sent false
29327 8439 Acked Write 1 Done Done Sent false
29328 8440 Acked Write 6 Done Done Sent false
29329 8441 Acked Write 1 Done Done Sent false
29330 8442 Acked Read 7 Done Done Sent false
29331 8443 Acked Read 3 Done Done Sent false
29332 8444 Acked Read 9 Done Done Sent false
29333 8445 Acked Write 8 Done Done Sent false
29334 8446 Acked Read 5 Done Done Sent false
29335 8447 Acked Flush 0 Done Done Sent false
29336 8448 Acked Flush 0 Done Done Sent false
29337 8449 Acked Write 8 Done Done Sent false
29338 8450 Acked Write 1 Done Done Sent false
29339 8451 Acked Write 1 Done Done Sent false
29340 8452 Acked Read 1 Done Done Sent false
29341 8453 Acked Write 7 Done Done Sent false
29342 8454 Acked Read 4 Done Done Sent false
29343 8455 Acked Read 8 Done Done Sent false
29344 8456 Acked Read 7 Done Done Sent false
29345 8457 Acked Write 3 Done Done Sent false
29346 8458 Acked Read 8 Done Done Sent false
29347 8459 Acked Write 1 Done Done Sent false
29348 8460 Acked Read 5 Done Done Sent false
29349 8461 Acked Read 9 Done Done Sent false
29350 8462 Acked Read 7 Done Done Sent false
29351 8463 Acked Read 9 Done Done Sent false
29352 8464 Acked Read 10 Done Done Sent false
29353 8465 Acked Write 3 Done Done Sent false
29354 8466 Acked Write 4 Done Done Sent false
29355 8467 Acked Read 5 Done Done Sent false
29356 8468 Acked Read 10 Done Done Sent false
29357 8469 Acked Flush 0 Done Done Sent false
29358 8470 Acked Flush 0 Done Done Sent false
29359 8471 Acked Read 4 Done Done Sent false
29360 8472 Acked Write 3 Done Done Sent false
29361 8473 Acked Read 10 Done Done Sent false
29362 8474 Acked Write 8 Done Done Sent false
29363 8475 Acked Read 9 Done Done Sent false
29364 8476 Acked Write 5 Done Done Sent false
29365 8477 Acked Flush 0 Done Done Sent false
29366 8478 Acked Read 8 Done Done Sent false
29367 8479 Acked Write 6 Done Done Sent false
29368 8480 Acked Write 2 Done Done Sent false
29369 8481 Acked Write 10 Done Done Sent false
29370 8482 Acked Flush 0 Done Done Sent false
29371 8483 Acked Read 6 Done Done Sent false
29372 8484 Acked Write 8 Done Done Sent false
29373 8485 Acked Read 6 Done Done Sent false
29374 8486 Acked Read 9 Done Done Sent false
29375 8487 Acked Flush 0 Done Done Sent false
29376 8488 Acked Flush 0 Done Done Sent false
29377 8489 Acked Read 6 Done Done Sent false
29378 8490 Acked Read 10 Done Done Sent false
29379 8491 Acked Read 9 Done Done Sent false
29380 8492 Acked Read 1 Done Done Sent false
29381 8493 Acked Write 4 Done Done Sent false
29382 8494 Acked Write 7 Done Done Sent false
29383 8495 Acked Read 5 Done Done Sent false
29384 8496 Acked Flush 0 Done Done Sent false
29385 8497 Acked Write 4 Done Done Sent false
29386 8498 Acked Flush 0 Done Done Sent false
29387 8499 Acked Read 2 Done Done Sent false
29388 8500 Acked Read 7 Done Done Sent false
29389 8501 Acked Write 4 Done Done Sent false
29390 8502 Acked Write 1 Done Done Sent false
29391 8503 Acked Write 7 Done Done Sent false
29392 8504 Acked Read 5 Done Done Sent false
29393 8505 Acked Read 9 Done Done Sent false
29394 8506 Acked Write 8 Done Done Sent false
29395 8507 Acked Read 6 Done Done Sent false
29396 8508 Acked Write 2 Done Done Sent false
29397 8509 Acked Read 9 Done Done Sent false
29398 8510 Acked Read 9 Done Done Sent false
29399 8511 Acked Read 1 Done Done Sent false
29400 8512 Acked Read 9 Done Done Sent false
29401 8513 Acked Flush 0 Done Done Sent false
29402 8514 Acked Write 4 Done Done Sent false
29403 8515 Acked Write 4 Done Done Sent false
29404 8516 Acked Read 1 Done Done Sent false
29405 8517 Acked Flush 0 Done Done Sent false
29406 8518 Acked Read 1 Done Done Sent false
29407 8519 Acked Read 6 Done Done Sent false
29408 8520 Acked Read 4 Done Done Sent false
29409 8521 Acked Write 5 Done Done Sent false
29410 8522 Acked Read 7 Done Done Sent false
29411 8523 Acked Write 2 Done Done Sent false
29412 8524 Acked Write 4 Done Done Sent false
29413 8525 Acked Write 10 Done Done Sent false
29414 8526 Acked Read 4 Done Done Sent false
29415 8527 Acked Write 10 Done Done Sent false
29416 8528 Acked Write 10 Done Done Sent false
29417 8529 Acked Flush 0 Done Done Sent false
29418 8530 Acked Write 1 Done Done Sent false
29419 8531 Acked Read 6 Done Done Sent false
29420 8532 Acked Read 1 Done Done Sent false
29421 8533 Acked Write 8 Done Done Sent false
29422 8534 Acked Flush 0 Done Done Sent false
29423 8535 Acked Read 1 Done Done Sent false
29424 8536 Acked Write 8 Done Done Sent false
29425 8537 Acked Read 10 Done Done Sent false
29426 8538 Acked Write 3 Done Done Sent false
29427 8539 Acked Flush 0 Done Done Sent false
29428 8540 Acked Read 3 Done Done Sent false
29429 8541 Acked Write 8 Done Done Sent false
29430 8542 Acked Read 10 Done Done Sent false
29431 8543 Acked Read 2 Done Done Sent false
29432 STATES DS:0 DS:1 DS:2 TOTAL
29433 Sent 0 0 301 301
29434 Done 301 301 0 602
29435 Skipped 0 0 0 0
29436 Error 0 0 0 0
29437 Last Flush: Some(JobId(8539)) Some(JobId(8539)) Some(JobId(8242))
29438 Downstairs last five completed: 8242 8241 8240 8239 8238
29439 Upstairs last five completed: 8543 8542 8541 8540 8539
29440 CLIENT: Up:0 ds:301 act:2
294412025-02-25T20:52:59.665ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294422025-02-25T20:53:00.667ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294432025-02-25T20:53:00.814ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294442025-02-25T20:53:00.815ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63895) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294452025-02-25T20:53:00.816ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294462025-02-25T20:53:00.816ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294472025-02-25T20:53:00.816ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294482025-02-25T20:53:00.816ZINFOcrucible: send last flush ID to this DS: Some(JobId(8242)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294492025-02-25T20:53:00.816ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294502025-02-25T20:53:00.816ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8242)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294512025-02-25T20:53:00.817ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8242)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
29452 ----------------------------------------------------------------
29453 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29454 Downstairs last five completed: 8544 8543 8542 8541 8540
29455 Upstairs last five completed: 8544 8543 8542 8541 8540
29456 CLIENT: Up:0 ds:0 act:3
29457 Replay: All jobs finished
29458 Main task has work to do, go find it
29459 got dsc Stop(2)
29460 stop 2
29461 [8830] Got stop action so:false kr:false
29462 Feb 25 20:53:03.071 INFO request completed, latency_us: 379, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e98a9b9b-e9ad-41a9-806c-ae55955d9981, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
29463 Feb 25 20:53:03.071 INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/2, method: GET, req_id: 91cad797-b978-436f-9d69-18613f2c3a13, remote_addr: 127.0.0.1:43930, local_addr: 127.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:53:03.071805359Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
29464 0.0.1:9998
294652025-02-25T20:53:03.071ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294662025-02-25T20:53:03.071ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
294672025-02-25T20:53:03.071ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
29468 [8830] Exited with: ExitStatus(unix_wait_status(9))
29469 [8830][2] reports Exit
29470 Feb 25 20:53:07.075 INFO request completed, latency_us: 347, response_code: 200, uri: /state/cid/2, method: GET, req_id: d4159570-ebae-474c-af9d-476ad4b5384c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
29471 001/300 Read block 3198 len 16384
29472 002/300 Read block 701 len 20480
29473 003/300 Write block 3892 len 4096 data: 7
29474 004/300 Write block 1451 len 36864 data: 4 4 3 4 6 6 7 9 9
29475 005/300 Read block 3329 len 4096
29476 006/300 Read block 503 len 12288
29477 007/300 Flush
29478 008/300 Flush
29479 009/300 Write block 2867 len 12288 data: 5 6 6
29480 010/300 Read block 1929 len 8192
29481 011/300 Write block 2435 len 40960 data: 3 3 3 3 4 3 3 4 5 5
29482 012/300 Write block 1638 len 32768 data: 4 5 3 3 3 3 2 3
29483 013/300 Read block 3512 len 36864
29484 014/300 Read block 1371 len 12288
29485 015/300 Read block 1137 len 40960
29486 016/300 Write block 3680 len 16384 data: 7 6 3 3
29487 017/300 Write block 368 len 20480 data: 3 5 4 6 5
29488 018/300 Read block 696 len 16384
29489 019/300 Write block 2167 len 32768 data: 8 7 6 6 5 4 4 4
29490 020/300 Read block 3281 len 24576
29491 021/300 Write block 509 len 4096 data: 5
29492 022/300 Write block 3006 len 20480 data: 6 5 7 9 8
29493 023/300 Write block 4150 len 36864 data: 6 6 6 5 6 5 3 2 2
29494 024/300 Read block 1305 len 40960
29495 025/300 Read block 4965 len 20480
29496 026/300 Write block 166 len 36864 data: 6 4 4 6 6 6 7 9 10
29497 027/300 Write block 303 len 40960 data: 4 3 2 4 5 5 5 5 5 5
29498 028/300 Flush
29499 029/300 Read block 209 len 16384
29500 030/300 Read block 2854 len 28672
29501 031/300 Write block 527 len 32768 data: 4 4 5 5 6 8 7 5
29502 032/300 Read block 3330 len 16384
29503 033/300 Read block 3808 len 4096
29504 034/300 Write block 2895 len 24576 data: 9 8 8 8 9 9
29505 035/300 Write block 1660 len 8192 data: 8 6
29506 036/300 Write block 366 len 16384 data: 3 3 4 6
29507 037/300 Read block 2878 len 16384
29508 038/300 Flush
29509 039/300 Read block 1153 len 16384
29510 040/300 Read block 461 len 36864
29511 041/300 Read block 2249 len 8192
29512 042/300 Read block 4894 len 40960
29513 043/300 Read block 820 len 40960
29514 044/300 Read block 4200 len 4096
29515 045/300 Read block 1581 len 36864
29516 046/300 Write block 3041 len 36864 data: 5 6 6 6 5 4 5 5 6
29517 047/300 Read block 449 len 12288
29518 048/300 Read block 928 len 36864
29519 049/300 Write block 4763 len 8192 data: 5 4
29520 050/300 Read block 3062 len 32768
29521 051/300 Write block 496 len 36864 data: 4 4 6 6 6 7 6 6 5
29522 052/300 Flush
29523 053/300 Read block 2105 len 32768
29524 054/300 Write block 4967 len 40960 data: 6 7 6 6 6 5 5 6 6 5
29525 055/300 Write block 3808 len 36864 data: 4 4 5 5 5 5 5 3 3
29526 056/300 Read block 1900 len 36864
29527 057/300 Write block 1446 len 8192 data: 4 4
29528 058/300 Flush
29529 059/300 Write block 449 len 20480 data: 5 5 5 5 4
29530 060/300 Write block 2388 len 32768 data: 2 3 3 2 3 3 3 3
29531 061/300 Read block 3535 len 40960
29532 062/300 Read block 3934 len 12288
29533 063/300 Read block 987 len 28672
29534 064/300 Write block 2468 len 8192 data: 4 5
29535 065/300 Write block 3262 len 28672 data: 2 4 5 7 7 7 8
29536 066/300 Read block 4149 len 24576
29537 067/300 Read block 657 len 8192
29538 068/300 Read block 821 len 24576
29539 069/300 Write block 1538 len 16384 data: 3 3 4 4
29540 070/300 Read block 551 len 40960
29541 071/300 Write block 3794 len 40960 data: 5 5 5 6 6 6 6 3 4 5
29542 072/300 Read block 1897 len 40960
29543 073/300 Write block 3334 len 16384 data: 6 6 6 6
29544 074/300 Write block 4457 len 24576 data: 6 6 8 7 10 11
29545 075/300 Write block 2044 len 28672 data: 4 5 3 4 5 5 5
29546 076/300 Read block 4294 len 16384
29547 077/300 Write block 736 len 12288 data: 3 3 3
29548 078/300 Write block 4572 len 32768 data: 6 5 5 6 5 6 6 4
29549 079/300 Read block 3919 len 40960
29550 080/300 Read block 3701 len 20480
29551 081/300 Write block 4525 len 16384 data: 6 8 8 8
29552 082/300 Read block 2227 len 24576
29553 083/300 Write block 302 len 28672 data: 3 5 4 3 5 6 6
29554 084/300 Read block 2425 len 32768
29555 085/300 Read block 3671 len 24576
29556 086/300 Read block 4651 len 40960
29557 087/300 Read block 3023 len 28672
29558 088/300 Write block 1972 len 8192 data: 4 4
29559 089/300 Read block 1618 len 36864
29560 090/300 Flush
29561 091/300 Read block 3115 len 36864
29562 092/300 Write block 4916 len 16384 data: 3 4 6 6
29563 093/300 Write block 3904 len 36864 data: 9 9 8 8 7 6 3 3 3
29564 094/300 Read block 1042 len 4096
29565 095/300 Flush
29566 096/300 Read block 2616 len 4096
29567 097/300 Write block 826 len 28672 data: 4 2 4 4 3 4 4
29568 098/300 Read block 1848 len 24576
29569 099/300 Flush
29570 100/300 Write block 2120 len 36864 data: 4 3 5 4 5 5 6 6 5
29571 101/300 Write block 3511 len 24576 data: 3 3 4 4 6 6
29572 102/300 Read block 562 len 12288
29573 103/300 Read block 1396 len 20480
29574 104/300 Read block 3867 len 36864
29575 105/300 Write block 828 len 40960 data: 5 5 4 5 5 4 5 4 6 5
29576 106/300 Read block 4145 len 36864
29577 107/300 Read block 2268 len 4096
29578 108/300 Read block 2503 len 32768
29579 109/300 Read block 4690 len 32768
29580 110/300 Write block 1074 len 24576 data: 8 7 5 4 5 4
29581 111/300 Read block 4873 len 16384
29582 112/300 Write block 1548 len 32768 data: 5 5 5 5 4 5 4 6
29583 113/300 Write block 4331 len 40960 data: 5 6 5 5 4 4 4 4 5 4
29584 114/300 Write block 3480 len 40960 data: 6 6 7 5 7 7 7 7 7 7
29585 115/300 Read block 1560 len 16384
29586 116/300 Read block 699 len 16384
29587 117/300 Write block 2862 len 24576 data: 3 3 2 2 3 6
29588 118/300 Write block 3340 len 28672 data: 5 4 4 5 5 5 6
29589 119/300 Write block 270 len 28672 data: 7 6 6 6 6 5 7
29590 120/300 Write block 614 len 8192 data: 7 7
29591 121/300 Write block 4866 len 4096 data: 5
29592 122/300 Read block 1757 len 32768
29593 123/300 Read block 119 len 4096
29594 124/300 Write block 963 len 36864 data: 6 6 4 6 6 6 4 4 4
29595 125/300 Flush
29596 126/300 Read block 3412 len 32768
29597 127/300 Write block 4132 len 12288 data: 5 5 5
29598 128/300 Read block 149 len 28672
29599 129/300 Write block 836 len 16384 data: 7 6 5 4
29600 130/300 Write block 971 len 16384 data: 5 5 5 5
29601 131/300 Write block 4439 len 32768 data: 5 6 6 6 5 4 4 4
29602 132/300 Read block 4277 len 32768
29603 133/300 Write block 4335 len 12288 data: 5 5 5
29604 134/300 Write block 685 len 8192 data: 4 4
29605 135/300 Write block 1941 len 36864 data: 2 3 3 4 4 5 6 7 6
29606 136/300 Write block 128 len 8192 data: 6 6
29607 137/300 Read block 4081 len 28672
29608 138/300 Flush
29609 139/300 Read block 3109 len 4096
29610 140/300 Read block 3810 len 16384
29611 141/300 Read block 1075 len 36864
29612 142/300 Write block 798 len 12288 data: 4 3 3
29613 143/300 Write block 2500 len 4096 data: 6
29614 144/300 Flush
29615 145/300 Read block 1825 len 32768
29616 146/300 Write block 804 len 16384 data: 5 5 5 4
29617 147/300 Read block 325 len 32768
29618 148/300 Write block 1251 len 8192 data: 3 4
29619 149/300 Write block 1007 len 12288 data: 4 7 7
29620 150/300 Flush
29621 151/300 Read block 3790 len 32768
29622 152/300 Read block 3197 len 36864
29623 153/300 Read block 1676 len 40960
29624 154/300 Read block 4745 len 8192
29625 155/300 Write block 213 len 4096 data: 4
29626 156/300 Read block 2813 len 36864
29627 157/300 Write block 3428 len 16384 data: 3 5 4 6
29628 158/300 Write block 4193 len 32768 data: 5 5 6 5 5 4 3 2
29629 159/300 Read block 114 len 28672
29630 160/300 Flush
29631 161/300 Read block 3779 len 32768
29632 162/300 Read block 4548 len 20480
29633 163/300 Read block 4287 len 20480
29634 164/300 Read block 2120 len 12288
29635 165/300 Read block 4082 len 28672
29636 166/300 Read block 2904 len 40960
29637 167/300 Write block 703 len 4096 data: 6
29638 168/300 Read block 1768 len 20480
29639 169/300 Write block 4449 len 20480 data: 3 3 3 4 3
29640 170/300 Write block 3290 len 40960 data: 3 3 4 4 6 7 6 6 6 8
29641 171/300 Read block 4779 len 28672
29642 172/300 Write block 3444 len 16384 data: 8 6 6 6
29643 173/300 Read block 2842 len 8192
29644 174/300 Read block 1135 len 24576
29645 175/300 Read block 2068 len 12288
29646 176/300 Write block 1130 len 8192 data: 5 4
29647 177/300 Read block 3196 len 12288
29648 178/300 Write block 665 len 16384 data: 6 8 8 9
29649 179/300 Write block 832 len 8192 data: 6 5
29650 180/300 Read block 2850 len 16384
29651 181/300 Write block 2533 len 28672 data: 6 7 5 4 4 5 5
29652 182/300 Read block 4290 len 32768
29653 183/300 Write block 1942 len 36864 data: 4 4 5 5 6 7 8 7 7
29654 184/300 Write block 1175 len 36864 data: 5 5 4 5 4 3 4 6 5
29655 185/300 Read block 2329 len 16384
29656 186/300 Read block 1904 len 24576
29657 187/300 Flush
29658 188/300 Write block 4289 len 40960 data: 4 4 5 6 6 8 6 8 7 6
29659 189/300 Write block 3452 len 4096 data: 3
29660 190/300 Read block 711 len 20480
29661 191/300 Read block 1837 len 32768
29662 192/300 Read block 854 len 20480
29663 193/300 Write block 2748 len 36864 data: 7 7 7 8 6 6 8 8 9
29664 194/300 Read block 2795 len 4096
29665 195/300 Read block 3459 len 24576
29666 196/300 Write block 1161 len 4096 data: 4
29667 197/300 Read block 1509 len 4096
29668 198/300 Flush
29669 199/300 Write block 3862 len 32768 data: 3 3 2 4 5 5 5 5
29670 200/300 Read block 1198 len 24576
29671 201/300 Write block 604 len 8192 data: 5 6
29672 202/300 Read block 3742 len 16384
29673 203/300 Write block 3643 len 12288 data: 9 10 8
29674 204/300 Write block 4344 len 24576 data: 5 4 3 4 4 4
29675 205/300 Flush
29676 206/300 Read block 4747 len 20480
29677 207/300 Read block 2619 len 36864
29678 208/300 Read block 3097 len 12288
29679 209/300 Write block 1192 len 4096 data: 5
29680 210/300 Write block 2676 len 24576 data: 4 4 4 4 8 9
29681 211/300 Write block 3844 len 8192 data: 4 3
29682 212/300 Write block 1569 len 24576 data: 3 6 5 4 4 4
29683 213/300 Read block 85 len 8192
29684 214/300 Write block 2295 len 32768 data: 3 3 2 3 3 4 5 6
29685 215/300 Read block 1956 len 28672
29686 216/300 Write block 2256 len 4096 data: 11
29687 217/300 Write block 3124 len 32768 data: 7 7 7 6 5 5 5 5
29688 218/300 Read block 2500 len 4096
29689 219/300 Write block 4832 len 36864 data: 4 4 4 5 5 4 6 7 5
29690 220/300 Read block 2310 len 8192
29691 221/300 Read block 963 len 24576
29692 222/300 Write block 421 len 8192 data: 8 8
29693 223/300 Read block 2186 len 20480
29694 224/300 Write block 22 len 32768 data: 2 2 2 4 4 4 3 4
29695 225/300 Read block 1248 len 36864
29696 226/300 Read block 24 len 20480
29697 227/300 Read block 3094 len 12288
29698 228/300 Write block 3891 len 32768 data: 7 8 6 6 6 6 6 6
29699 229/300 Read block 990 len 24576
29700 230/300 Write block 3977 len 8192 data: 9 7
29701 231/300 Read block 532 len 8192
29702 232/300 Read block 359 len 36864
29703 233/300 Write block 3466 len 4096 data: 5
29704 234/300 Write block 4614 len 40960 data: 6 6 6 7 6 5 5 4 4 4
29705 235/300 Read block 128 len 28672
29706 236/300 Write block 4316 len 12288 data: 3 3 3
29707 237/300 Write block 2455 len 24576 data: 8 5 5 5 6 5
29708 238/300 Write block 662 len 32768 data: 9 8 8 7 9 9 10 5
29709 239/300 Read block 643 len 12288
29710 240/300 Read block 3358 len 40960
29711 241/300 Write block 2938 len 12288 data: 3 3 3
29712 242/300 Read block 3117 len 40960
29713 243/300 Write block 3465 len 4096 data: 5
29714 244/300 Write block 4863 len 24576 data: 3 3 4 6 4 4
29715 245/300 Flush
29716 246/300 Read block 4461 len 28672
29717 247/300 Write block 1083 len 24576 data: 3 2 3 3 4 5
29718 248/300 Read block 2216 len 4096
29719 249/300 Read block 4137 len 32768
29720 250/300 Read block 1594 len 8192
29721 251/300 Write block 1589 len 4096 data: 3
29722 252/300 Write block 1001 len 28672 data: 6 6 6 5 5 4 5
29723 253/300 Read block 694 len 40960
29724 254/300 Write block 1300 len 8192 data: 4 3
29725 255/300 Write block 443 len 40960 data: 7 5 5 5 5 6 6 6 6 6
29726 256/300 Read block 1451 len 4096
29727 257/300 Read block 1121 len 28672
29728 258/300 Read block 3746 len 28672
29729 259/300 Write block 4373 len 24576 data: 5 5 5 5 5 6
29730 260/300 Read block 2680 len 32768
29731 261/300 Write block 3917 len 4096 data: 6
29732 262/300 Flush
29733 263/300 Read block 121 len 20480
29734 264/300 Read block 3466 len 16384
29735 265/300 Write block 4002 len 40960 data: 5 5 3 3 3 3 3 3 3 3
29736 266/300 Write block 3049 len 4096 data: 7
29737 267/300 Write block 2144 len 24576 data: 5 5 5 5 7 6
29738 268/300 Read block 4881 len 12288
29739 269/300 Flush
29740 270/300 Flush
29741 271/300 Read block 2362 len 4096
29742 272/300 Write block 3188 len 8192 data: 4 6
29743 273/300 Write block 85 len 28672 data: 4 3 6 5 5 5 7
29744 274/300 Write block 2519 len 8192 data: 9 7
29745 275/300 Write block 3129 len 4096 data: 6
29746 276/300 Write block 92 len 12288 data: 8 8 8
29747 277/300 Write block 2342 len 28672 data: 11 11 6 7 6 5 7
29748 278/300 Read block 2910 len 8192
29749 279/300 Read block 1753 len 40960
29750 280/300 Write block 2680 len 12288 data: 9 10 9
29751 281/300 Read block 4725 len 20480
29752 282/300 Read block 2940 len 12288
29753 283/300 Write block 2613 len 36864 data: 6 5 6 4 5 6 7 8 7
29754 284/300 Read block 619 len 8192
29755 285/300 Flush
29756 286/300 Read block 3946 len 24576
29757 287/300 Flush
29758 288/300 Read block 2370 len 28672
29759 289/300 Read block 4609 len 36864
29760 290/300 Write block 4116 len 8192 data: 5 5
29761 291/300 Write block 453 len 28672 data: 5 4 4 3 3 4 4
29762 292/300 Read block 2217 len 32768
29763 293/300 Read block 1117 len 16384
29764 294/300 Flush
29765 295/300 Read block 4109 len 12288
29766 296/300 Read block 4659 len 16384
29767 297/300 Read block 4885 len 28672
29768 298/300 Read block 3500 len 40960
29769 299/300 Read block 2577 len 28672
29770 300/300 Flush
29771 301/300 Write block 318 len 28672 data: 4 4 5 7 8 6 6
29772 Main task has work to do, go find it
29773 got dsc Start(2)
29774 start 2
29775 Feb 25 20:53:07.323 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/2, method: POST, req_id: 96a7787e-d13f-4b98-9a82-0eea0333c789, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
29776 Replay: started 2, returned:Ok(())
29777 ----------------------------------------------------------------
29778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29779 States: Active Active Offline
29780 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29781 8545 Acked Read 4 Done Done Sent false
29782 8546 Acked Read 5 Done Done Sent false
29783 8547 Acked Write 1 Done Done Sent false
29784 8548 Acked Write 9 Done Done Sent false
29785 8549 Acked Read 1 Done Done Sent false
29786 8550 Acked Read 3 Done Done Sent false
29787 8551 Acked Flush 0 Done Done Sent false
29788 8552 Acked Flush 0 Done Done Sent false
29789 8553 Acked Write 3 Done Done Sent false
29790 8554 Acked Read 2 Done Done Sent false
29791 8555 Acked Write 10 Done Done Sent false
29792 8556 Acked Write 8 Done Done Sent false
29793 8557 Acked Read 9 Done Done Sent false
29794 8558 Acked Read 3 Done Done Sent false
29795 8559 Acked Read 10 Done Done Sent false
29796 8560 Acked Write 4 Done Done Sent false
29797 8561 Acked Write 5 Done Done Sent false
29798 8562 Acked Read 4 Done Done Sent false
29799 8563 Acked Write 8 Done Done Sent false
29800 8564 Acked Read 6 Done Done Sent false
29801 8565 Acked Write 1 Done Done Sent false
29802 8566 Acked Write 5 Done Done Sent false
29803 8567 Acked Write 9 Done Done Sent false
29804 8568 Acked Read 10 Done Done Sent false
29805 8569 Acked Read 5 Done Done Sent false
29806 8570 Acked Write 9 Done Done Sent false
29807 8571 Acked Write 10 Done Done Sent false
29808 8572 Acked Flush 0 Done Done Sent false
29809 8573 Acked Read 4 Done Done Sent false
29810 8574 Acked Read 7 Done Done Sent false
29811 8575 Acked Write 8 Done Done Sent false
29812 8576 Acked Read 4 Done Done Sent false
29813 8577 Acked Read 1 Done Done Sent false
29814 8578 Acked Write 6 Done Done Sent false
29815 8579 Acked Write 2 Done Done Sent false
29816 8580 Acked Write 4 Done Done Sent false
29817 8581 Acked Read 4 Done Done Sent false
29818 8582 Acked Flush 0 Done Done Sent false
29819 8583 Acked Read 4 Done Done Sent false
29820 8584 Acked Read 9 Done Done Sent false
29821 8585 Acked Read 2 Done Done Sent false
29822 8586 Acked Read 10 Done Done Sent false
29823 8587 Acked Read 10 Done Done Sent false
29824 8588 Acked Read 1 Done Done Sent false
29825 8589 Acked Read 9 Done Done Sent false
29826 8590 Acked Write 9 Done Done Sent false
29827 8591 Acked Read 3 Done Done Sent false
29828 8592 Acked Read 9 Done Done Sent false
29829 8593 Acked Write 2 Done Done Sent false
29830 8594 Acked Read 8 Done Done Sent false
29831 8595 Acked Write 9 Done Done Sent false
29832 8596 Acked Flush 0 Done Done Sent false
29833 8597 Acked Read 8 Done Done Sent false
29834 8598 Acked Write 10 Done Done Sent false
29835 8599 Acked Write 9 Done Done Sent false
29836 8600 Acked Read 9 Done Done Sent false
29837 8601 Acked Write 2 Done Done Sent false
29838 8602 Acked Flush 0 Done Done Sent false
29839 8603 Acked Write 5 Done Done Sent false
29840 8604 Acked Write 8 Done Done Sent false
29841 8605 Acked Read 10 Done Done Sent false
29842 8606 Acked Read 3 Done Done Sent false
29843 8607 Acked Read 7 Done Done Sent false
29844 8608 Acked Write 2 Done Done Sent false
29845 8609 Acked Write 7 Done Done Sent false
29846 8610 Acked Read 6 Done Done Sent false
29847 8611 Acked Read 2 Done Done Sent false
29848 8612 Acked Read 6 Done Done Sent false
29849 8613 Acked Write 4 Done Done Sent false
29850 8614 Acked Read 10 Done Done Sent false
29851 8615 Acked Write 10 Done Done Sent false
29852 8616 Acked Read 10 Done Done Sent false
29853 8617 Acked Write 4 Done Done Sent false
29854 8618 Acked Write 6 Done Done Sent false
29855 8619 Acked Write 7 Done Done Sent false
29856 8620 Acked Read 4 Done Done Sent false
29857 8621 Acked Write 3 Done Done Sent false
29858 8622 Acked Write 8 Done Done Sent false
29859 8623 Acked Read 10 Done Done Sent false
29860 8624 Acked Read 5 Done Done Sent false
29861 8625 Acked Write 4 Done Done Sent false
29862 8626 Acked Read 6 Done Done Sent false
29863 8627 Acked Write 7 Done Done Sent false
29864 8628 Acked Read 8 Done Done Sent false
29865 8629 Acked Read 6 Done Done Sent false
29866 8630 Acked Read 10 Done Done Sent false
29867 8631 Acked Read 7 Done Done Sent false
29868 8632 Acked Write 2 Done Done Sent false
29869 8633 Acked Read 9 Done Done Sent false
29870 8634 Acked Flush 0 Done Done Sent false
29871 8635 Acked Read 9 Done Done Sent false
29872 8636 Acked Write 4 Done Done Sent false
29873 8637 Acked Write 9 Done Done Sent false
29874 8638 Acked Read 1 Done Done Sent false
29875 8639 Acked Flush 0 Done Done Sent false
29876 8640 Acked Read 1 Done Done Sent false
29877 8641 Acked Write 7 Done Done Sent false
29878 8642 Acked Read 6 Done Done Sent false
29879 8643 Acked Flush 0 Done Done Sent false
29880 8644 Acked Write 9 Done Done Sent false
29881 8645 Acked Write 6 Done Done Sent false
29882 8646 Acked Read 3 Done Done Sent false
29883 8647 Acked Read 5 Done Done Sent false
29884 8648 Acked Read 9 Done Done Sent false
29885 8649 Acked Write 10 Done Done Sent false
29886 8650 Acked Read 9 Done Done Sent false
29887 8651 Acked Read 1 Done Done Sent false
29888 8652 Acked Read 8 Done Done Sent false
29889 8653 Acked Read 8 Done Done Sent false
29890 8654 Acked Write 6 Done Done Sent false
29891 8655 Acked Read 4 Done Done Sent false
29892 8656 Acked Write 8 Done Done Sent false
29893 8657 Acked Write 10 Done Done Sent false
29894 8658 Acked Write 10 Done Done Sent false
29895 8659 Acked Read 4 Done Done Sent false
29896 8660 Acked Read 4 Done Done Sent false
29897 8661 Acked Write 6 Done Done Sent false
29898 8662 Acked Write 7 Done Done Sent false
29899 8663 Acked Write 7 Done Done Sent false
29900 8664 Acked Write 2 Done Done Sent false
29901 8665 Acked Write 1 Done Done Sent false
29902 8666 Acked Read 8 Done Done Sent false
29903 8667 Acked Read 1 Done Done Sent false
29904 8668 Acked Write 9 Done Done Sent false
29905 8669 Acked Flush 0 Done Done Sent false
29906 8670 Acked Read 8 Done Done Sent false
29907 8671 Acked Write 3 Done Done Sent false
29908 8672 Acked Read 7 Done Done Sent false
29909 8673 Acked Write 4 Done Done Sent false
29910 8674 Acked Write 4 Done Done Sent false
29911 8675 Acked Write 8 Done Done Sent false
29912 8676 Acked Read 8 Done Done Sent false
29913 8677 Acked Write 3 Done Done Sent false
29914 8678 Acked Write 2 Done Done Sent false
29915 8679 Acked Write 9 Done Done Sent false
29916 8680 Acked Write 2 Done Done Sent false
29917 8681 Acked Read 7 Done Done Sent false
29918 8682 Acked Flush 0 Done Done Sent false
29919 8683 Acked Read 1 Done Done Sent false
29920 8684 Acked Read 4 Done Done Sent false
29921 8685 Acked Read 9 Done Done Sent false
29922 8686 Acked Write 3 Done Done Sent false
29923 8687 Acked Write 1 Done Done Sent false
29924 8688 Acked Flush 0 Done Done Sent false
29925 8689 Acked Read 8 Done Done Sent false
29926 8690 Acked Write 4 Done Done Sent false
29927 8691 Acked Read 8 Done Done Sent false
29928 8692 Acked Write 2 Done Done Sent false
29929 8693 Acked Write 3 Done Done Sent false
29930 8694 Acked Flush 0 Done Done Sent false
29931 8695 Acked Read 8 Done Done Sent false
29932 8696 Acked Read 9 Done Done Sent false
29933 8697 Acked Read 10 Done Done Sent false
29934 8698 Acked Read 2 Done Done Sent false
29935 8699 Acked Write 1 Done Done Sent false
29936 8700 Acked Read 9 Done Done Sent false
29937 8701 Acked Write 4 Done Done Sent false
29938 8702 Acked Write 8 Done Done Sent false
29939 8703 Acked Read 7 Done Done Sent false
29940 8704 Acked Flush 0 Done Done Sent false
29941 8705 Acked Read 8 Done Done Sent false
29942 8706 Acked Read 5 Done Done Sent false
29943 8707 Acked Read 5 Done Done Sent false
29944 8708 Acked Read 3 Done Done Sent false
29945 8709 Acked Read 7 Done Done Sent false
29946 8710 Acked Read 10 Done Done Sent false
29947 8711 Acked Write 1 Done Done Sent false
29948 8712 Acked Read 5 Done Done Sent false
29949 8713 Acked Write 5 Done Done Sent false
29950 8714 Acked Write 10 Done Done Sent false
29951 8715 Acked Read 7 Done Done Sent false
29952 8716 Acked Write 4 Done Done Sent false
29953 8717 Acked Read 2 Done Done Sent false
29954 8718 Acked Read 6 Done Done Sent false
29955 8719 Acked Read 3 Done Done Sent false
29956 8720 Acked Write 2 Done Done Sent false
29957 8721 Acked Read 3 Done Done Sent false
29958 8722 Acked Write 4 Done Done Sent false
29959 8723 Acked Write 2 Done Done Sent false
29960 8724 Acked Read 4 Done Done Sent false
29961 8725 Acked Write 7 Done Done Sent false
29962 8726 Acked Read 8 Done Done Sent false
29963 8727 Acked Write 9 Done Done Sent false
29964 8728 Acked Write 9 Done Done Sent false
29965 8729 Acked Read 4 Done Done Sent false
29966 8730 Acked Read 6 Done Done Sent false
29967 8731 Acked Flush 0 Done Done Sent false
29968 8732 Acked Write 10 Done Done Sent false
29969 8733 Acked Write 1 Done Done Sent false
29970 8734 Acked Read 5 Done Done Sent false
29971 8735 Acked Read 8 Done Done Sent false
29972 8736 Acked Read 5 Done Done Sent false
29973 8737 Acked Write 9 Done Done Sent false
29974 8738 Acked Read 1 Done Done Sent false
29975 8739 Acked Read 6 Done Done Sent false
29976 8740 Acked Write 1 Done Done Sent false
29977 8741 Acked Read 1 Done Done Sent false
29978 8742 Acked Flush 0 Done Done Sent false
29979 8743 Acked Write 8 Done Done Sent false
29980 8744 Acked Read 6 Done Done Sent false
29981 8745 Acked Write 2 Done Done Sent false
29982 8746 Acked Read 4 Done Done Sent false
29983 8747 Acked Write 3 Done Done Sent false
29984 8748 Acked Write 6 Done Done Sent false
29985 8749 Acked Flush 0 Done Done Sent false
29986 8750 Acked Read 5 Done Done Sent false
29987 8751 Acked Read 9 Done Done Sent false
29988 8752 Acked Read 3 Done Done Sent false
29989 8753 Acked Write 1 Done Done Sent false
29990 8754 Acked Write 6 Done Done Sent false
29991 8755 Acked Write 2 Done Done Sent false
29992 8756 Acked Write 6 Done Done Sent false
29993 8757 Acked Read 2 Done Done Sent false
29994 8758 Acked Write 8 Done Done Sent false
29995 8759 Acked Read 7 Done Done Sent false
29996 8760 Acked Write 1 Done Done Sent false
29997 8761 Acked Write 8 Done Done Sent false
29998 8762 Acked Read 1 Done Done Sent false
29999 8763 Acked Write 9 Done Done Sent false
30000 8764 Acked Read 2 Done Done Sent false
30001 8765 Acked Read 6 Done Done Sent false
30002 8766 Acked Write 2 Done Done Sent false
30003 8767 Acked Read 5 Done Done Sent false
30004 8768 Acked Write 8 Done Done Sent false
30005 8769 Acked Read 9 Done Done Sent false
30006 8770 Acked Read 5 Done Done Sent false
30007 8771 Acked Read 3 Done Done Sent false
30008 8772 Acked Write 8 Done Done Sent false
30009 8773 Acked Read 6 Done Done Sent false
30010 8774 Acked Write 2 Done Done Sent false
30011 8775 Acked Read 2 Done Done Sent false
30012 8776 Acked Read 9 Done Done Sent false
30013 8777 Acked Write 1 Done Done Sent false
30014 8778 Acked Write 10 Done Done Sent false
30015 8779 Acked Read 7 Done Done Sent false
30016 8780 Acked Write 3 Done Done Sent false
30017 8781 Acked Write 6 Done Done Sent false
30018 8782 Acked Write 8 Done Done Sent false
30019 8783 Acked Read 3 Done Done Sent false
30020 8784 Acked Read 10 Done Done Sent false
30021 8785 Acked Write 3 Done Done Sent false
30022 8786 Acked Read 10 Done Done Sent false
30023 8787 Acked Write 1 Done Done Sent false
30024 8788 Acked Write 6 Done Done Sent false
30025 8789 Acked Flush 0 Done Done Sent false
30026 8790 Acked Read 7 Done Done Sent false
30027 8791 Acked Write 6 Done Done Sent false
30028 8792 Acked Read 1 Done Done Sent false
30029 8793 Acked Read 8 Done Done Sent false
30030 8794 Acked Read 2 Done Done Sent false
30031 8795 Acked Write 1 Done Done Sent false
30032 8796 Acked Write 7 Done Done Sent false
30033 8797 Acked Read 10 Done Done Sent false
30034 8798 Acked Write 2 Done Done Sent false
30035 8799 Acked Write 10 Done Done Sent false
30036 8800 Acked Read 1 Done Done Sent false
30037 8801 Acked Read 7 Done Done Sent false
30038 8802 Acked Read 7 Done Done Sent false
30039 8803 Acked Write 6 Done Done Sent false
30040 8804 Acked Read 8 Done Done Sent false
30041 8805 Acked Write 1 Done Done Sent false
30042 8806 Acked Flush 0 Done Done Sent false
30043 8807 Acked Read 5 Done Done Sent false
30044 8808 Acked Read 4 Done Done Sent false
30045 8809 Acked Write 10 Done Done Sent false
30046 8810 Acked Write 1 Done Done Sent false
30047 8811 Acked Write 6 Done Done Sent false
30048 8812 Acked Read 3 Done Done Sent false
30049 8813 Acked Flush 0 Done Done Sent false
30050 8814 Acked Flush 0 Done Done Sent false
30051 8815 Acked Read 1 Done Done Sent false
30052 8816 Acked Write 2 Done Done Sent false
30053 8817 Acked Write 7 Done Done Sent false
30054 8818 Acked Write 2 Done Done Sent false
30055 8819 Acked Write 1 Done Done Sent false
30056 8820 Acked Write 3 Done Done Sent false
30057 8821 Acked Write 7 Done Done Sent false
30058 8822 Acked Read 2 Done Done Sent false
30059 8823 Acked Read 10 Done Done Sent false
30060 8824 Acked Write 3 Done Done Sent false
30061 8825 Acked Read 5 Done Done Sent false
30062 8826 Acked Read 3 Done Done Sent false
30063 8827 Acked Write 9 Done Done Sent false
30064 8828 Acked Read 2 Done Done Sent false
30065 8829 Acked Flush 0 Done Done Sent false
30066 8830 Acked Read 6 Done Done Sent false
30067 8831 Acked Flush 0 Done Done Sent false
30068 8832 Acked Read 7 Done Done Sent false
30069 8833 Acked Read 9 Done Done Sent false
30070 8834 Acked Write 2 Done Done Sent false
30071 8835 Acked Write 7 Done Done Sent false
30072 8836 Acked Read 8 Done Done Sent false
30073 8837 Acked Read 4 Done Done Sent false
30074 8838 Acked Flush 0 Done Done Sent false
30075 8839 Acked Read 3 Done Done Sent false
30076 8840 Acked Read 4 Done Done Sent false
30077 8841 Acked Read 7 Done Done Sent false
30078 8842 Acked Read 10 Done Done Sent false
30079 8843 Acked Read 7 Done Done Sent false
30080 8844 Acked Flush 0 Done Done Sent false
30081 8845 Acked Write 7 Done Done Sent false
30082 STATES DS:0 DS:1 DS:2 TOTAL
30083 Sent 0 0 301 301
30084 Done 301 301 0 602
30085 Skipped 0 0 0 0
30086 Error 0 0 0 0
30087 Last Flush: Some(JobId(8844)) Some(JobId(8844)) Some(JobId(8544))
30088 Downstairs last five completed: 8544 8543 8542 8541 8540
30089 Upstairs last five completed: 8845 8844 8843 8842 8841
30090 CLIENT: Up:0 ds:301 act:2
300912025-02-25T20:53:07.824ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
300922025-02-25T20:53:09.016ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
30093 [8830] got start action
30094 [8830] I am going to restart
30095 Starting downstairs at port 8830
30096 Make output file at "/tmp/dsc/downstairs-8830.txt"
30097 [8830][2] reports Starting
30098 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
30099 [8830][2] reports Running
301002025-02-25T20:53:09.683ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
301012025-02-25T20:53:10.685ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
30102 ----------------------------------------------------------------
30103 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30104 States: Active Active Offline
30105 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30106 8545 Acked Read 4 Done Done Sent false
30107 8546 Acked Read 5 Done Done Sent false
30108 8547 Acked Write 1 Done Done Sent false
30109 8548 Acked Write 9 Done Done Sent false
30110 8549 Acked Read 1 Done Done Sent false
30111 8550 Acked Read 3 Done Done Sent false
30112 8551 Acked Flush 0 Done Done Sent false
30113 8552 Acked Flush 0 Done Done Sent false
30114 8553 Acked Write 3 Done Done Sent false
30115 8554 Acked Read 2 Done Done Sent false
30116 8555 Acked Write 10 Done Done Sent false
30117 8556 Acked Write 8 Done Done Sent false
30118 8557 Acked Read 9 Done Done Sent false
30119 8558 Acked Read 3 Done Done Sent false
30120 8559 Acked Read 10 Done Done Sent false
30121 8560 Acked Write 4 Done Done Sent false
30122 8561 Acked Write 5 Done Done Sent false
30123 8562 Acked Read 4 Done Done Sent false
30124 8563 Acked Write 8 Done Done Sent false
30125 8564 Acked Read 6 Done Done Sent false
30126 8565 Acked Write 1 Done Done Sent false
30127 8566 Acked Write 5 Done Done Sent false
30128 8567 Acked Write 9 Done Done Sent false
30129 8568 Acked Read 10 Done Done Sent false
30130 8569 Acked Read 5 Done Done Sent false
30131 8570 Acked Write 9 Done Done Sent false
30132 8571 Acked Write 10 Done Done Sent false
30133 8572 Acked Flush 0 Done Done Sent false
30134 8573 Acked Read 4 Done Done Sent false
30135 8574 Acked Read 7 Done Done Sent false
30136 8575 Acked Write 8 Done Done Sent false
30137 8576 Acked Read 4 Done Done Sent false
30138 8577 Acked Read 1 Done Done Sent false
30139 8578 Acked Write 6 Done Done Sent false
30140 8579 Acked Write 2 Done Done Sent false
30141 8580 Acked Write 4 Done Done Sent false
30142 8581 Acked Read 4 Done Done Sent false
30143 8582 Acked Flush 0 Done Done Sent false
30144 8583 Acked Read 4 Done Done Sent false
30145 8584 Acked Read 9 Done Done Sent false
30146 8585 Acked Read 2 Done Done Sent false
30147 8586 Acked Read 10 Done Done Sent false
30148 8587 Acked Read 10 Done Done Sent false
30149 8588 Acked Read 1 Done Done Sent false
30150 8589 Acked Read 9 Done Done Sent false
30151 8590 Acked Write 9 Done Done Sent false
30152 8591 Acked Read 3 Done Done Sent false
30153 8592 Acked Read 9 Done Done Sent false
30154 8593 Acked Write 2 Done Done Sent false
30155 8594 Acked Read 8 Done Done Sent false
30156 8595 Acked Write 9 Done Done Sent false
30157 8596 Acked Flush 0 Done Done Sent false
30158 8597 Acked Read 8 Done Done Sent false
30159 8598 Acked Write 10 Done Done Sent false
30160 8599 Acked Write 9 Done Done Sent false
30161 8600 Acked Read 9 Done Done Sent false
30162 8601 Acked Write 2 Done Done Sent false
30163 8602 Acked Flush 0 Done Done Sent false
30164 8603 Acked Write 5 Done Done Sent false
30165 8604 Acked Write 8 Done Done Sent false
30166 8605 Acked Read 10 Done Done Sent false
30167 8606 Acked Read 3 Done Done Sent false
30168 8607 Acked Read 7 Done Done Sent false
30169 8608 Acked Write 2 Done Done Sent false
30170 8609 Acked Write 7 Done Done Sent false
30171 8610 Acked Read 6 Done Done Sent false
30172 8611 Acked Read 2 Done Done Sent false
30173 8612 Acked Read 6 Done Done Sent false
30174 8613 Acked Write 4 Done Done Sent false
30175 8614 Acked Read 10 Done Done Sent false
30176 8615 Acked Write 10 Done Done Sent false
30177 8616 Acked Read 10 Done Done Sent false
30178 8617 Acked Write 4 Done Done Sent false
30179 8618 Acked Write 6 Done Done Sent false
30180 8619 Acked Write 7 Done Done Sent false
30181 8620 Acked Read 4 Done Done Sent false
30182 8621 Acked Write 3 Done Done Sent false
30183 8622 Acked Write 8 Done Done Sent false
30184 8623 Acked Read 10 Done Done Sent false
30185 8624 Acked Read 5 Done Done Sent false
30186 8625 Acked Write 4 Done Done Sent false
30187 8626 Acked Read 6 Done Done Sent false
30188 8627 Acked Write 7 Done Done Sent false
30189 8628 Acked Read 8 Done Done Sent false
30190 8629 Acked Read 6 Done Done Sent false
30191 8630 Acked Read 10 Done Done Sent false
30192 8631 Acked Read 7 Done Done Sent false
30193 8632 Acked Write 2 Done Done Sent false
30194 8633 Acked Read 9 Done Done Sent false
30195 8634 Acked Flush 0 Done Done Sent false
30196 8635 Acked Read 9 Done Done Sent false
30197 8636 Acked Write 4 Done Done Sent false
30198 8637 Acked Write 9 Done Done Sent false
30199 8638 Acked Read 1 Done Done Sent false
30200 8639 Acked Flush 0 Done Done Sent false
30201 8640 Acked Read 1 Done Done Sent false
30202 8641 Acked Write 7 Done Done Sent false
30203 8642 Acked Read 6 Done Done Sent false
30204 8643 Acked Flush 0 Done Done Sent false
30205 8644 Acked Write 9 Done Done Sent false
30206 8645 Acked Write 6 Done Done Sent false
30207 8646 Acked Read 3 Done Done Sent false
30208 8647 Acked Read 5 Done Done Sent false
30209 8648 Acked Read 9 Done Done Sent false
30210 8649 Acked Write 10 Done Done Sent false
30211 8650 Acked Read 9 Done Done Sent false
30212 8651 Acked Read 1 Done Done Sent false
30213 8652 Acked Read 8 Done Done Sent false
30214 8653 Acked Read 8 Done Done Sent false
30215 8654 Acked Write 6 Done Done Sent false
30216 8655 Acked Read 4 Done Done Sent false
30217 8656 Acked Write 8 Done Done Sent false
30218 8657 Acked Write 10 Done Done Sent false
30219 8658 Acked Write 10 Done Done Sent false
30220 8659 Acked Read 4 Done Done Sent false
30221 8660 Acked Read 4 Done Done Sent false
30222 8661 Acked Write 6 Done Done Sent false
30223 8662 Acked Write 7 Done Done Sent false
30224 8663 Acked Write 7 Done Done Sent false
30225 8664 Acked Write 2 Done Done Sent false
30226 8665 Acked Write 1 Done Done Sent false
30227 8666 Acked Read 8 Done Done Sent false
30228 8667 Acked Read 1 Done Done Sent false
30229 8668 Acked Write 9 Done Done Sent false
30230 8669 Acked Flush 0 Done Done Sent false
30231 8670 Acked Read 8 Done Done Sent false
30232 8671 Acked Write 3 Done Done Sent false
30233 8672 Acked Read 7 Done Done Sent false
30234 8673 Acked Write 4 Done Done Sent false
30235 8674 Acked Write 4 Done Done Sent false
30236 8675 Acked Write 8 Done Done Sent false
30237 8676 Acked Read 8 Done Done Sent false
30238 8677 Acked Write 3 Done Done Sent false
30239 8678 Acked Write 2 Done Done Sent false
30240 8679 Acked Write 9 Done Done Sent false
30241 8680 Acked Write 2 Done Done Sent false
30242 8681 Acked Read 7 Done Done Sent false
30243 8682 Acked Flush 0 Done Done Sent false
30244 8683 Acked Read 1 Done Done Sent false
30245 8684 Acked Read 4 Done Done Sent false
30246 8685 Acked Read 9 Done Done Sent false
30247 8686 Acked Write 3 Done Done Sent false
30248 8687 Acked Write 1 Done Done Sent false
30249 8688 Acked Flush 0 Done Done Sent false
30250 8689 Acked Read 8 Done Done Sent false
30251 8690 Acked Write 4 Done Done Sent false
30252 8691 Acked Read 8 Done Done Sent false
30253 8692 Acked Write 2 Done Done Sent false
30254 8693 Acked Write 3 Done Done Sent false
30255 8694 Acked Flush 0 Done Done Sent false
30256 8695 Acked Read 8 Done Done Sent false
30257 8696 Acked Read 9 Done Done Sent false
30258 8697 Acked Read 10 Done Done Sent false
30259 8698 Acked Read 2 Done Done Sent false
30260 8699 Acked Write 1 Done Done Sent false
30261 8700 Acked Read 9 Done Done Sent false
30262 8701 Acked Write 4 Done Done Sent false
30263 8702 Acked Write 8 Done Done Sent false
30264 8703 Acked Read 7 Done Done Sent false
30265 8704 Acked Flush 0 Done Done Sent false
30266 8705 Acked Read 8 Done Done Sent false
30267 8706 Acked Read 5 Done Done Sent false
30268 8707 Acked Read 5 Done Done Sent false
30269 8708 Acked Read 3 Done Done Sent false
30270 8709 Acked Read 7 Done Done Sent false
30271 8710 Acked Read 10 Done Done Sent false
30272 8711 Acked Write 1 Done Done Sent false
30273 8712 Acked Read 5 Done Done Sent false
30274 8713 Acked Write 5 Done Done Sent false
30275 8714 Acked Write 10 Done Done Sent false
30276 8715 Acked Read 7 Done Done Sent false
30277 8716 Acked Write 4 Done Done Sent false
30278 8717 Acked Read 2 Done Done Sent false
30279 8718 Acked Read 6 Done Done Sent false
30280 8719 Acked Read 3 Done Done Sent false
30281 8720 Acked Write 2 Done Done Sent false
30282 8721 Acked Read 3 Done Done Sent false
30283 8722 Acked Write 4 Done Done Sent false
30284 8723 Acked Write 2 Done Done Sent false
30285 8724 Acked Read 4 Done Done Sent false
30286 8725 Acked Write 7 Done Done Sent false
30287 8726 Acked Read 8 Done Done Sent false
30288 8727 Acked Write 9 Done Done Sent false
30289 8728 Acked Write 9 Done Done Sent false
30290 8729 Acked Read 4 Done Done Sent false
30291 8730 Acked Read 6 Done Done Sent false
30292 8731 Acked Flush 0 Done Done Sent false
30293 8732 Acked Write 10 Done Done Sent false
30294 8733 Acked Write 1 Done Done Sent false
30295 8734 Acked Read 5 Done Done Sent false
30296 8735 Acked Read 8 Done Done Sent false
30297 8736 Acked Read 5 Done Done Sent false
30298 8737 Acked Write 9 Done Done Sent false
30299 8738 Acked Read 1 Done Done Sent false
30300 8739 Acked Read 6 Done Done Sent false
30301 8740 Acked Write 1 Done Done Sent false
30302 8741 Acked Read 1 Done Done Sent false
30303 8742 Acked Flush 0 Done Done Sent false
30304 8743 Acked Write 8 Done Done Sent false
30305 8744 Acked Read 6 Done Done Sent false
30306 8745 Acked Write 2 Done Done Sent false
30307 8746 Acked Read 4 Done Done Sent false
30308 8747 Acked Write 3 Done Done Sent false
30309 8748 Acked Write 6 Done Done Sent false
30310 8749 Acked Flush 0 Done Done Sent false
30311 8750 Acked Read 5 Done Done Sent false
30312 8751 Acked Read 9 Done Done Sent false
30313 8752 Acked Read 3 Done Done Sent false
30314 8753 Acked Write 1 Done Done Sent false
30315 8754 Acked Write 6 Done Done Sent false
30316 8755 Acked Write 2 Done Done Sent false
30317 8756 Acked Write 6 Done Done Sent false
30318 8757 Acked Read 2 Done Done Sent false
30319 8758 Acked Write 8 Done Done Sent false
30320 8759 Acked Read 7 Done Done Sent false
30321 8760 Acked Write 1 Done Done Sent false
30322 8761 Acked Write 8 Done Done Sent false
30323 8762 Acked Read 1 Done Done Sent false
30324 8763 Acked Write 9 Done Done Sent false
30325 8764 Acked Read 2 Done Done Sent false
30326 8765 Acked Read 6 Done Done Sent false
30327 8766 Acked Write 2 Done Done Sent false
30328 8767 Acked Read 5 Done Done Sent false
30329 8768 Acked Write 8 Done Done Sent false
30330 8769 Acked Read 9 Done Done Sent false
30331 8770 Acked Read 5 Done Done Sent false
30332 8771 Acked Read 3 Done Done Sent false
30333 8772 Acked Write 8 Done Done Sent false
30334 8773 Acked Read 6 Done Done Sent false
30335 8774 Acked Write 2 Done Done Sent false
30336 8775 Acked Read 2 Done Done Sent false
30337 8776 Acked Read 9 Done Done Sent false
30338 8777 Acked Write 1 Done Done Sent false
30339 8778 Acked Write 10 Done Done Sent false
30340 8779 Acked Read 7 Done Done Sent false
30341 8780 Acked Write 3 Done Done Sent false
30342 8781 Acked Write 6 Done Done Sent false
30343 8782 Acked Write 8 Done Done Sent false
30344 8783 Acked Read 3 Done Done Sent false
30345 8784 Acked Read 10 Done Done Sent false
30346 8785 Acked Write 3 Done Done Sent false
30347 8786 Acked Read 10 Done Done Sent false
30348 8787 Acked Write 1 Done Done Sent false
30349 8788 Acked Write 6 Done Done Sent false
30350 8789 Acked Flush 0 Done Done Sent false
30351 8790 Acked Read 7 Done Done Sent false
30352 8791 Acked Write 6 Done Done Sent false
30353 8792 Acked Read 1 Done Done Sent false
30354 8793 Acked Read 8 Done Done Sent false
30355 8794 Acked Read 2 Done Done Sent false
30356 8795 Acked Write 1 Done Done Sent false
30357 8796 Acked Write 7 Done Done Sent false
30358 8797 Acked Read 10 Done Done Sent false
30359 8798 Acked Write 2 Done Done Sent false
30360 8799 Acked Write 10 Done Done Sent false
30361 8800 Acked Read 1 Done Done Sent false
30362 8801 Acked Read 7 Done Done Sent false
30363 8802 Acked Read 7 Done Done Sent false
30364 8803 Acked Write 6 Done Done Sent false
30365 8804 Acked Read 8 Done Done Sent false
30366 8805 Acked Write 1 Done Done Sent false
30367 8806 Acked Flush 0 Done Done Sent false
30368 8807 Acked Read 5 Done Done Sent false
30369 8808 Acked Read 4 Done Done Sent false
30370 8809 Acked Write 10 Done Done Sent false
30371 8810 Acked Write 1 Done Done Sent false
30372 8811 Acked Write 6 Done Done Sent false
30373 8812 Acked Read 3 Done Done Sent false
30374 8813 Acked Flush 0 Done Done Sent false
30375 8814 Acked Flush 0 Done Done Sent false
30376 8815 Acked Read 1 Done Done Sent false
30377 8816 Acked Write 2 Done Done Sent false
30378 8817 Acked Write 7 Done Done Sent false
30379 8818 Acked Write 2 Done Done Sent false
30380 8819 Acked Write 1 Done Done Sent false
30381 8820 Acked Write 3 Done Done Sent false
30382 8821 Acked Write 7 Done Done Sent false
30383 8822 Acked Read 2 Done Done Sent false
30384 8823 Acked Read 10 Done Done Sent false
30385 8824 Acked Write 3 Done Done Sent false
30386 8825 Acked Read 5 Done Done Sent false
30387 8826 Acked Read 3 Done Done Sent false
30388 8827 Acked Write 9 Done Done Sent false
30389 8828 Acked Read 2 Done Done Sent false
30390 8829 Acked Flush 0 Done Done Sent false
30391 8830 Acked Read 6 Done Done Sent false
30392 8831 Acked Flush 0 Done Done Sent false
30393 8832 Acked Read 7 Done Done Sent false
30394 8833 Acked Read 9 Done Done Sent false
30395 8834 Acked Write 2 Done Done Sent false
30396 8835 Acked Write 7 Done Done Sent false
30397 8836 Acked Read 8 Done Done Sent false
30398 8837 Acked Read 4 Done Done Sent false
30399 8838 Acked Flush 0 Done Done Sent false
30400 8839 Acked Read 3 Done Done Sent false
30401 8840 Acked Read 4 Done Done Sent false
30402 8841 Acked Read 7 Done Done Sent false
30403 8842 Acked Read 10 Done Done Sent false
30404 8843 Acked Read 7 Done Done Sent false
30405 8844 Acked Flush 0 Done Done Sent false
30406 8845 Acked Write 7 Done Done Sent false
30407 STATES DS:0 DS:1 DS:2 TOTAL
30408 Sent 0 0 301 301
30409 Done 301 301 0 602
30410 Skipped 0 0 0 0
30411 Error 0 0 0 0
30412 Last Flush: Some(JobId(8844)) Some(JobId(8844)) Some(JobId(8544))
30413 Downstairs last five completed: 8544 8543 8542 8541 8540
30414 Upstairs last five completed: 8845 8844 8843 8842 8841
30415 CLIENT: Up:0 ds:301 act:2
304162025-02-25T20:53:11.826ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304172025-02-25T20:53:12.689ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304182025-02-25T20:53:13.073ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304192025-02-25T20:53:13.073ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40061) = io task client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304202025-02-25T20:53:13.074ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304212025-02-25T20:53:13.075ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304222025-02-25T20:53:13.075ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304232025-02-25T20:53:13.075ZINFOcrucible: send last flush ID to this DS: Some(JobId(8544)) = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304242025-02-25T20:53:13.075ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304252025-02-25T20:53:13.075ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8544)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304262025-02-25T20:53:13.075ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8544)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
30427 ----------------------------------------------------------------
30428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30429 Downstairs last five completed: 8846 8845 8844 8843 8842
30430 Upstairs last five completed: 8846 8845 8844 8843 8842
30431 CLIENT: Up:0 ds:0 act:3
30432 Replay: All jobs finished
30433 Main task has work to do, go find it
30434 got dsc Stop(0)
30435 stop 0
30436 Feb 25 20:53:15.328 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false
30437 method: POST, req_id: 8ed9d29c-1c5e-44e5-88d0-e390b8b31360, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
30438 Feb 25 20:53:15.328 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/0, method: GET, req_id: 231e2c3f-82e5-4a6b-b66d-f32ddda20b6a, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
304392025-02-25T20:53:15.328ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304402025-02-25T20:53:15.329ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304412025-02-25T20:53:15.329ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
304422025-02-25T20:53:15.329ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
30443 [8810] Exited with: ExitStatus(unix_wait_status(9))
30444 [8810][0] reports Exit
30445 Feb 25 20:53:19.331 INFO request completed, latency_us: 239, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6c4ef758-b18e-4966-897a-4f7cebd5f08f, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
30446 001/300 Write block 3368 len 4096 data: 5
30447 002/300 Flush
30448 003/300 Read block 3942 len 36864
30449 004/300 Flush
30450 005/300 Write block 2951 len 12288 data: 5 5 6
30451 006/300 Write block 4226 len 24576 data: 5 7 7 7 6 6
30452 007/300 Read block 650 len 4096
30453 008/300 Write block 608 len 16384 data: 4 4 4 4
30454 009/300 Write block 1132 len 40960 data: 4 5 6 6 5 4 3 5 5 6
30455 010/300 Read block 3472 len 24576
30456 011/300 Write block 4974 len 20480 data: 7 7 6 3 3
30457 012/300 Read block 2621 len 12288
30458 013/300 Flush
30459 014/300 Write block 357 len 24576 data: 5 4 4 4 4 3
30460 015/300 Read block 2808 len 28672
30461 016/300 Read block 531 len 4096
30462 017/300 Read block 3241 len 28672
30463 018/300 Write block 1614 len 32768 data: 7 6 7 9 7 6 5 6
30464 019/300 Read block 1253 len 8192
30465 020/300 Flush
30466 021/300 Read block 1077 len 16384
30467 022/300 Read block 2978 len 28672
30468 023/300 Write block 1077 len 4096 data: 5
30469 024/300 Write block 2638 len 16384 data: 6 5 4 5
30470 025/300 Read block 2545 len 24576
30471 026/300 Read block 3116 len 4096
30472 027/300 Read block 1335 len 24576
30473 028/300 Write block 728 len 36864 data: 10 10 8 7 4 4 4 4 4
30474 029/300 Read block 1187 len 40960
30475 030/300 Read block 3805 len 36864
30476 031/300 Write block 837 len 12288 data: 7 6 5
30477 032/300 Flush
30478 033/300 Read block 4970 len 24576
30479 034/300 Read block 4027 len 32768
30480 035/300 Read block 1198 len 16384
30481 036/300 Write block 697 len 12288 data: 4 4 4
30482 037/300 Write block 4103 len 4096 data: 7
30483 038/300 Write block 494 len 36864 data: 6 6 5 5 7 7 7 8 7
30484 039/300 Read block 4017 len 24576
30485 040/300 Write block 3727 len 16384 data: 3 3 3 3
30486 041/300 Flush
30487 042/300 Read block 964 len 20480
30488 043/300 Read block 782 len 4096
30489 044/300 Read block 3644 len 12288
30490 045/300 Write block 3260 len 16384 data: 2 2 3 5
30491 046/300 Read block 1494 len 32768
30492 047/300 Write block 518 len 20480 data: 4 3 3 2 3
30493 048/300 Flush
30494 049/300 Write block 707 len 28672 data: 8 6 6 6 6 5 7
30495 050/300 Read block 4468 len 28672
30496 051/300 Write block 1403 len 36864 data: 3 2 2 2 2 3 5 5 5
30497 052/300 Flush
30498 053/300 Read block 3955 len 36864
30499 054/300 Flush
30500 055/300 Write block 2371 len 12288 data: 2 2 5
30501 056/300 Write block 206 len 32768 data: 5 3 4 5 4 5 4 5
30502 057/300 Write block 720 len 24576 data: 5 7 7 5 6 8
30503 058/300 Read block 2141 len 40960
30504 059/300 Write block 3226 len 16384 data: 2 2 2 3
30505 060/300 Write block 3853 len 24576 data: 10 10 9 7 6 4
30506 061/300 Write block 83 len 4096 data: 6
30507 062/300 Read block 119 len 4096
30508 063/300 Write block 3081 len 24576 data: 5 5 6 6 7 5
30509 064/300 Read block 3034 len 24576
30510 065/300 Read block 3933 len 4096
30511 066/300 Flush
30512 067/300 Read block 3228 len 20480
30513 068/300 Flush
30514 069/300 Read block 3354 len 12288
30515 070/300 Read block 1454 len 32768
30516 071/300 Read block 3212 len 8192
30517 072/300 Write block 2288 len 32768 data: 5 5 3 3 2 3 3 4
30518 073/300 Read block 2768 len 28672
30519 074/300 Read block 3611 len 24576
30520 075/300 Read block 2775 len 36864
30521 076/300 Read block 4897 len 8192
30522 077/300 Read block 2534 len 28672
30523 078/300 Read block 3707 len 32768
30524 079/300 Write block 1041 len 20480 data: 4 6 7 8 7
30525 080/300 Read block 3717 len 36864
30526 081/300 Write block 959 len 8192 data: 8 10
30527 082/300 Write block 3102 len 16384 data: 7 9 9 9
30528 083/300 Read block 3127 len 24576
30529 084/300 Read block 2938 len 28672
30530 085/300 Read block 4354 len 12288
30531 086/300 Write block 4745 len 28672 data: 7 4 5 6 7 7 7
30532 087/300 Read block 2774 len 8192
30533 088/300 Write block 922 len 28672 data: 9 9 8 7 8 7 5
30534 089/300 Flush
30535 090/300 Read block 2503 len 40960
30536 091/300 Write block 4031 len 12288 data: 5 5 6
30537 092/300 Write block 373 len 12288 data: 5 6 6
30538 093/300 Write block 4795 len 20480 data: 9 9 9 9 9
30539 094/300 Write block 624 len 12288 data: 6 6 5
30540 095/300 Read block 2098 len 28672
30541 096/300 Read block 765 len 12288
30542 097/300 Read block 507 len 12288
30543 098/300 Read block 2313 len 28672
30544 099/300 Write block 3288 len 8192 data: 4 3
30545 100/300 Read block 2051 len 12288
30546 101/300 Read block 2763 len 8192
30547 102/300 Read block 236 len 20480
30548 103/300 Read block 1248 len 28672
30549 104/300 Write block 2598 len 40960 data: 4 4 4 2 2 2 3 5 9 9
30550 105/300 Read block 985 len 20480
30551 106/300 Read block 438 len 4096
30552 107/300 Write block 1081 len 12288 data: 3 3 4
30553 108/300 Write block 4604 len 4096 data: 6
30554 109/300 Flush
30555 110/300 Read block 3682 len 16384
30556 111/300 Read block 408 len 32768
30557 112/300 Flush
30558 113/300 Flush
30559 114/300 Read block 2950 len 40960
30560 115/300 Read block 2384 len 16384
30561 116/300 Write block 1692 len 32768 data: 3 3 4 4 3 3 3 4
30562 117/300 Read block 3285 len 40960
30563 118/300 Flush
30564 119/300 Read block 925 len 24576
30565 120/300 Write block 2984 len 36864 data: 7 7 10 11 13 9 9 8 9
30566 121/300 Read block 4454 len 16384
30567 122/300 Flush
30568 123/300 Flush
30569 124/300 Flush
30570 125/300 Read block 2215 len 28672
30571 126/300 Write block 2663 len 32768 data: 5 5 6 5 6 7 4 3
30572 127/300 Write block 4223 len 28672 data: 5 4 3 6 8 8 8
30573 128/300 Write block 265 len 4096 data: 5
30574 129/300 Write block 4771 len 4096 data: 8
30575 130/300 Write block 3204 len 8192 data: 5 5
30576 131/300 Read block 4876 len 20480
30577 132/300 Write block 1389 len 32768 data: 8 8 8 7 8 5 4 3
30578 133/300 Read block 3917 len 28672
30579 134/300 Write block 4111 len 24576 data: 4 4 4 3 4 6
30580 135/300 Read block 3364 len 24576
30581 136/300 Read block 7 len 4096
30582 137/300 Write block 2811 len 12288 data: 4 3 4
30583 138/300 Write block 3548 len 40960 data: 4 4 4 4 3 2 3 5 6 7
30584 139/300 Write block 4291 len 20480 data: 6 7 7 9 7
30585 140/300 Read block 2771 len 20480
30586 141/300 Read block 2561 len 12288
30587 142/300 Write block 4377 len 28672 data: 6 7 5 5 5 5 6
30588 143/300 Write block 2243 len 20480 data: 7 10 8 7 6
30589 144/300 Read block 816 len 8192
30590 145/300 Read block 4285 len 36864
30591 146/300 Read block 435 len 40960
30592 147/300 Read block 2244 len 4096
30593 148/300 Read block 72 len 8192
30594 149/300 Write block 168 len 12288 data: 5 7 7
30595 150/300 Write block 4385 len 28672 data: 4 5 6 5 6 6 8
30596 151/300 Read block 584 len 40960
30597 152/300 Write block 3920 len 16384 data: 6 6 6 5
30598 153/300 Write block 4245 len 36864 data: 6 8 5 4 5 5 5 5 6
30599 154/300 Read block 3171 len 40960
30600 155/300 Write block 1206 len 36864 data: 5 4 6 6 7 7 7 6 6
30601 156/300 Read block 3661 len 4096
30602 157/300 Read block 3036 len 8192
30603 158/300 Read block 260 len 32768
30604 159/300 Flush
30605 160/300 Write block 429 len 16384 data: 7 8 7 6
30606 161/300 Flush
30607 162/300 Read block 1897 len 24576
30608 163/300 Read block 2392 len 28672
30609 164/300 Read block 832 len 20480
30610 165/300 Read block 2253 len 40960
30611 166/300 Read block 3754 len 28672
30612 167/300 Read block 817 len 4096
30613 168/300 Read block 2732 len 40960
30614 169/300 Flush
30615 170/300 Read block 1632 len 24576
30616 171/300 Write block 263 len 8192 data: 5 5
30617 172/300 Read block 281 len 40960
30618 173/300 Read block 3856 len 8192
30619 174/300 Read block 4084 len 28672
30620 175/300 Write block 758 len 12288 data: 3 3 3
30621 176/300 Write block 2920 len 32768 data: 6 8 8 6 5 4 4 4
30622 177/300 Read block 347 len 32768
30623 178/300 Write block 2349 len 32768 data: 7 7 6 5 6 5 5 6
30624 179/300 Read block 4178 len 36864
30625 180/300 Read block 2927 len 24576
30626 181/300 Write block 2285 len 8192 data: 10 8
30627 182/300 Read block 4190 len 36864
30628 183/300 Read block 2810 len 20480
30629 184/300 Flush
30630 185/300 Write block 3820 len 16384 data: 7 5 5 5
30631 186/300 Read block 886 len 12288
30632 187/300 Write block 600 len 4096 data: 3
30633 188/300 Read block 242 len 20480
30634 189/300 Flush
30635 190/300 Write block 1004 len 12288 data: 6 6 5
30636 191/300 Write block 1162 len 12288 data: 4 3 4
30637 192/300 Read block 1213 len 8192
30638 193/300 Write block 2053 len 36864 data: 5 4 5 5 4 4 4 4 3
30639 194/300 Read block 2451 len 36864
30640 195/300 Read block 848 len 28672
30641 196/300 Flush
30642 197/300 Write block 2974 len 28672 data: 5 4 4 4 2 2 4
30643 198/300 Flush
30644 199/300 Read block 3176 len 36864
30645 200/300 Flush
30646 201/300 Write block 127 len 8192 data: 7 7
30647 202/300 Read block 1384 len 20480
30648 203/300 Write block 1582 len 32768 data: 7 9 9 10 9 7 4 4
30649 204/300 Read block 2185 len 40960
30650 205/300 Read block 3417 len 12288
30651 206/300 Write block 3771 len 8192 data: 4 5
30652 207/300 Read block 2335 len 16384
30653 208/300 Read block 467 len 40960
30654 209/300 Read block 4350 len 32768
30655 210/300 Read block 4701 len 36864
30656 211/300 Flush
30657 212/300 Read block 4483 len 8192
30658 213/300 Read block 3405 len 36864
30659 214/300 Read block 4548 len 12288
30660 215/300 Read block 738 len 4096
30661 216/300 Read block 4573 len 4096
30662 217/300 Read block 1358 len 28672
30663 218/300 Flush
30664 219/300 Write block 3555 len 24576 data: 6 7 8 7 6 6
30665 220/300 Flush
30666 221/300 Read block 347 len 24576
30667 222/300 Read block 3906 len 40960
30668 223/300 Write block 893 len 28672 data: 3 3 4 5 6 6 6
30669 224/300 Write block 4450 len 40960 data: 4 4 5 4 7 6 6 7 7 9
30670 225/300 Read block 101 len 16384
30671 226/300 Read block 908 len 12288
30672 227/300 Write block 689 len 20480 data: 4 6 8 7 7
30673 228/300 Read block 4743 len 36864
30674 229/300 Read block 4016 len 40960
30675 230/300 Read block 1642 len 36864
30676 231/300 Read block 861 len 36864
30677 232/300 Flush
30678 233/300 Read block 2927 len 36864
30679 234/300 Write block 1168 len 40960 data: 8 8 5 4 3 4 5 6 6 5
30680 235/300 Flush
30681 236/300 Read block 1276 len 40960
30682 237/300 Write block 3327 len 24576 data: 4 6 6 6 6 5
30683 238/300 Read block 4762 len 12288
30684 239/300 Read block 2039 len 12288
30685 240/300 Flush
30686 241/300 Read block 13 len 24576
30687 242/300 Read block 915 len 40960
30688 243/300 Read block 1131 len 4096
30689 244/300 Read block 95 len 8192
30690 245/300 Write block 4159 len 32768 data: 5 7 10 12 11 11 11 12
30691 246/300 Read block 3977 len 40960
30692 247/300 Write block 40 len 40960 data: 6 6 4 3 3 3 4 5 4 4
30693 248/300 Read block 51 len 28672
30694 249/300 Read block 49 len 16384
30695 250/300 Read block 1517 len 12288
30696 251/300 Read block 2427 len 12288
30697 252/300 Read block 2282 len 12288
30698 253/300 Read block 2312 len 24576
30699 254/300 Read block 2462 len 16384
30700 255/300 Write block 2435 len 28672 data: 4 4 4 4 5 4 4
30701 256/300 Write block 2913 len 32768 data: 5 6 6 5 5 5 6 7
30702 257/300 Read block 1654 len 40960
30703 258/300 Read block 2711 len 40960
30704 259/300 Write block 617 len 4096 data: 8
30705 260/300 Read block 4938 len 24576
30706 261/300 Write block 2123 len 36864 data: 5 6 6 7 7 6 4 3 4
30707 262/300 Write block 2062 len 24576 data: 3 5 5 5 5 5
30708 263/300 Write block 1882 len 28672 data: 4 3 3 2 5 5 5
30709 264/300 Read block 4464 len 36864
30710 265/300 Write block 327 len 20480 data: 3 4 5 5 5
30711 266/300 Read block 2859 len 28672
30712 267/300 Write block 4514 len 16384 data: 6 6 5 5
30713 268/300 Write block 2594 len 4096 data: 6
30714 269/300 Write block 2255 len 4096 data: 10
30715 270/300 Read block 3067 len 36864
30716 271/300 Read block 4396 len 40960
30717 272/300 Write block 3101 len 20480 data: 7 8 10 10 10
30718 273/300 Read block 274 len 12288
30719 274/300 Read block 1081 len 28672
30720 275/300 Write block 2281 len 12288 data: 5 6 9
30721 276/300 Write block 806 len 12288 data: 6 5 3
30722 277/300 Write block 2385 len 36864 data: 5 5 4 3 4 4 3 4 4
30723 278/300 Write block 2097 len 12288 data: 4 3 5
30724 279/300 Read block 4755 len 32768
30725 280/300 Read block 3172 len 40960
30726 281/300 Read block 2238 len 16384
30727 282/300 Read block 574 len 4096
30728 283/300 Read block 1613 len 16384
30729 284/300 Read block 1472 len 24576
30730 285/300 Read block 4344 len 28672
30731 286/300 Write block 1732 len 8192 data: 5 4
30732 287/300 Write block 3989 len 24576 data: 6 6 6 6 5 5
30733 288/300 Read block 2456 len 40960
30734 289/300 Flush
30735 290/300 Read block 1336 len 28672
30736 291/300 Read block 3974 len 36864
30737 292/300 Write block 680 len 16384 data: 3 3 3 3
30738 293/300 Read block 646 len 16384
30739 294/300 Write block 2620 len 28672 data: 9 8 7 8 7 9 9
30740 295/300 Flush
30741 296/300 Read block 788 len 32768
30742 297/300 Write block 3979 len 40960 data: 8 6 6 6 5 5 5 6 5 6
30743 298/300 Read block 529 len 16384
30744 299/300 Read block 3124 len 4096
30745 300/300 Flush
30746 301/300 Write block 1870 len 12288 data: 6 5 5
30747 Main task has work to do, go find it
30748 got dsc Start(0)
30749 start 0
30750 Feb 25 20:53:19.592 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/0, method: POST, req_id: f9189ae4-93fe-4670-9136-f92ea530aa1a, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
30751 Replay: started 0, returned:Ok(())
30752 ----------------------------------------------------------------
30753 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30754 States: Offline Active Active
30755 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30756 8847 Acked Write 1 Sent Done Done false
30757 8848 Acked Flush 0 Sent Done Done false
30758 8849 Acked Read 9 Sent Done Done false
30759 8850 Acked Flush 0 Sent Done Done false
30760 8851 Acked Write 3 Sent Done Done false
30761 8852 Acked Write 6 Sent Done Done false
30762 8853 Acked Read 1 Sent Done Done false
30763 8854 Acked Write 4 Sent Done Done false
30764 8855 Acked Write 10 Sent Done Done false
30765 8856 Acked Read 6 Sent Done Done false
30766 8857 Acked Write 5 Sent Done Done false
30767 8858 Acked Read 3 Sent Done Done false
30768 8859 Acked Flush 0 Sent Done Done false
30769 8860 Acked Write 6 Sent Done Done false
30770 8861 Acked Read 7 Sent Done Done false
30771 8862 Acked Read 1 Sent Done Done false
30772 8863 Acked Read 7 Sent Done Done false
30773 8864 Acked Write 8 Sent Done Done false
30774 8865 Acked Read 2 Sent Done Done false
30775 8866 Acked Flush 0 Sent Done Done false
30776 8867 Acked Read 4 Sent Done Done false
30777 8868 Acked Read 7 Sent Done Done false
30778 8869 Acked Write 1 Sent Done Done false
30779 8870 Acked Write 4 Sent Done Done false
30780 8871 Acked Read 6 Sent Done Done false
30781 8872 Acked Read 1 Sent Done Done false
30782 8873 Acked Read 6 Sent Done Done false
30783 8874 Acked Write 9 Sent Done Done false
30784 8875 Acked Read 10 Sent Done Done false
30785 8876 Acked Read 9 Sent Done Done false
30786 8877 Acked Write 3 Sent Done Done false
30787 8878 Acked Flush 0 Sent Done Done false
30788 8879 Acked Read 6 Sent Done Done false
30789 8880 Acked Read 8 Sent Done Done false
30790 8881 Acked Read 4 Sent Done Done false
30791 8882 Acked Write 3 Sent Done Done false
30792 8883 Acked Write 1 Sent Done Done false
30793 8884 Acked Write 9 Sent Done Done false
30794 8885 Acked Read 6 Sent Done Done false
30795 8886 Acked Write 4 Sent Done Done false
30796 8887 Acked Flush 0 Sent Done Done false
30797 8888 Acked Read 5 Sent Done Done false
30798 8889 Acked Read 1 Sent Done Done false
30799 8890 Acked Read 3 Sent Done Done false
30800 8891 Acked Write 4 Sent Done Done false
30801 8892 Acked Read 8 Sent Done Done false
30802 8893 Acked Write 5 Sent Done Done false
30803 8894 Acked Flush 0 Sent Done Done false
30804 8895 Acked Write 7 Sent Done Done false
30805 8896 Acked Read 7 Sent Done Done false
30806 8897 Acked Write 9 Sent Done Done false
30807 8898 Acked Flush 0 Sent Done Done false
30808 8899 Acked Read 9 Sent Done Done false
30809 8900 Acked Flush 0 Sent Done Done false
30810 8901 Acked Write 3 Sent Done Done false
30811 8902 Acked Write 8 Sent Done Done false
30812 8903 Acked Write 6 Sent Done Done false
30813 8904 Acked Read 10 Sent Done Done false
30814 8905 Acked Write 4 Sent Done Done false
30815 8906 Acked Write 6 Sent Done Done false
30816 8907 Acked Write 1 Sent Done Done false
30817 8908 Acked Read 1 Sent Done Done false
30818 8909 Acked Write 6 Sent Done Done false
30819 8910 Acked Read 6 Sent Done Done false
30820 8911 Acked Read 1 Sent Done Done false
30821 8912 Acked Flush 0 Sent Done Done false
30822 8913 Acked Read 5 Sent Done Done false
30823 8914 Acked Flush 0 Sent Done Done false
30824 8915 Acked Read 3 Sent Done Done false
30825 8916 Acked Read 8 Sent Done Done false
30826 8917 Acked Read 2 Sent Done Done false
30827 8918 Acked Write 8 Sent Done Done false
30828 8919 Acked Read 7 Sent Done Done false
30829 8920 Acked Read 6 Sent Done Done false
30830 8921 Acked Read 9 Sent Done Done false
30831 8922 Acked Read 2 Sent Done Done false
30832 8923 Acked Read 7 Sent Done Done false
30833 8924 Acked Read 8 Sent Done Done false
30834 8925 Acked Write 5 Sent Done Done false
30835 8926 Acked Read 9 Sent Done Done false
30836 8927 Acked Write 2 Sent Done Done false
30837 8928 Acked Write 4 Sent Done Done false
30838 8929 Acked Read 6 Sent Done Done false
30839 8930 Acked Read 7 Sent Done Done false
30840 8931 Acked Read 3 Sent Done Done false
30841 8932 Acked Write 7 Sent Done Done false
30842 8933 Acked Read 2 Sent Done Done false
30843 8934 Acked Write 7 Sent Done Done false
30844 8935 Acked Flush 0 Sent Done Done false
30845 8936 Acked Read 10 Sent Done Done false
30846 8937 Acked Write 3 Sent Done Done false
30847 8938 Acked Write 3 Sent Done Done false
30848 8939 Acked Write 5 Sent Done Done false
30849 8940 Acked Write 3 Sent Done Done false
30850 8941 Acked Read 7 Sent Done Done false
30851 8942 Acked Read 3 Sent Done Done false
30852 8943 Acked Read 3 Sent Done Done false
30853 8944 Acked Read 7 Sent Done Done false
30854 8945 Acked Write 2 Sent Done Done false
30855 8946 Acked Read 3 Sent Done Done false
30856 8947 Acked Read 2 Sent Done Done false
30857 8948 Acked Read 5 Sent Done Done false
30858 8949 Acked Read 7 Sent Done Done false
30859 8950 Acked Write 10 Sent Done Done false
30860 8951 Acked Read 5 Sent Done Done false
30861 8952 Acked Read 1 Sent Done Done false
30862 8953 Acked Write 3 Sent Done Done false
30863 8954 Acked Write 1 Sent Done Done false
30864 8955 Acked Flush 0 Sent Done Done false
30865 8956 Acked Read 4 Sent Done Done false
30866 8957 Acked Read 8 Sent Done Done false
30867 8958 Acked Flush 0 Sent Done Done false
30868 8959 Acked Flush 0 Sent Done Done false
30869 8960 Acked Read 10 Sent Done Done false
30870 8961 Acked Read 4 Sent Done Done false
30871 8962 Acked Write 8 Sent Done Done false
30872 8963 Acked Read 10 Sent Done Done false
30873 8964 Acked Flush 0 Sent Done Done false
30874 8965 Acked Read 6 Sent Done Done false
30875 8966 Acked Write 9 Sent Done Done false
30876 8967 Acked Read 4 Sent Done Done false
30877 8968 Acked Flush 0 Sent Done Done false
30878 8969 Acked Flush 0 Sent Done Done false
30879 8970 Acked Flush 0 Sent Done Done false
30880 8971 Acked Read 7 Sent Done Done false
30881 8972 Acked Write 8 Sent Done Done false
30882 8973 Acked Write 7 Sent Done Done false
30883 8974 Acked Write 1 Sent Done Done false
30884 8975 Acked Write 1 Sent Done Done false
30885 8976 Acked Write 2 Sent Done Done false
30886 8977 Acked Read 5 Sent Done Done false
30887 8978 Acked Write 8 Sent Done Done false
30888 8979 Acked Read 7 Sent Done Done false
30889 8980 Acked Write 6 Sent Done Done false
30890 8981 Acked Read 6 Sent Done Done false
30891 8982 Acked Read 1 Sent Done Done false
30892 8983 Acked Write 3 Sent Done Done false
30893 8984 Acked Write 10 Sent Done Done false
30894 8985 Acked Write 5 Sent Done Done false
30895 8986 Acked Read 5 Sent Done Done false
30896 8987 Acked Read 3 Sent Done Done false
30897 8988 Acked Write 7 Sent Done Done false
30898 8989 Acked Write 5 Sent Done Done false
30899 8990 Acked Read 2 Sent Done Done false
30900 8991 Acked Read 9 Sent Done Done false
30901 8992 Acked Read 10 Sent Done Done false
30902 8993 Acked Read 1 Sent Done Done false
30903 8994 Acked Read 2 Sent Done Done false
30904 8995 Acked Write 3 Sent Done Done false
30905 8996 Acked Write 7 Sent Done Done false
30906 8997 Acked Read 10 Sent Done Done false
30907 8998 Acked Write 4 Sent Done Done false
30908 8999 Acked Write 9 Sent Done Done false
30909 9000 Acked Read 10 Sent Done Done false
30910 9001 Acked Write 9 Sent Done Done false
30911 9002 Acked Read 1 Sent Done Done false
30912 9003 Acked Read 2 Sent Done Done false
30913 9004 Acked Read 8 Sent Done Done false
30914 9005 Acked Flush 0 Sent Done Done false
30915 9006 Acked Write 4 Sent Done Done false
30916 9007 Acked Flush 0 Sent Done Done false
30917 9008 Acked Read 6 Sent Done Done false
30918 9009 Acked Read 7 Sent Done Done false
30919 9010 Acked Read 5 Sent Done Done false
30920 9011 Acked Read 10 Sent Done Done false
30921 9012 Acked Read 7 Sent Done Done false
30922 9013 Acked Read 1 Sent Done Done false
30923 9014 Acked Read 10 Sent Done Done false
30924 9015 Acked Flush 0 Sent Done Done false
30925 9016 Acked Read 6 Sent Done Done false
30926 9017 Acked Write 2 Sent Done Done false
30927 9018 Acked Read 10 Sent Done Done false
30928 9019 Acked Read 2 Sent Done Done false
30929 9020 Acked Read 7 Sent Done Done false
30930 9021 Acked Write 3 Sent Done Done false
30931 9022 Acked Write 8 Sent Done Done false
30932 9023 Acked Read 8 Sent Done Done false
30933 9024 Acked Write 8 Sent Done Done false
30934 9025 Acked Read 9 Sent Done Done false
30935 9026 Acked Read 6 Sent Done Done false
30936 9027 Acked Write 2 Sent Done Done false
30937 9028 Acked Read 9 Sent Done Done false
30938 9029 Acked Read 5 Sent Done Done false
30939 9030 Acked Flush 0 Sent Done Done false
30940 9031 Acked Write 4 Sent Done Done false
30941 9032 Acked Read 3 Sent Done Done false
30942 9033 Acked Write 1 Sent Done Done false
30943 9034 Acked Read 5 Sent Done Done false
30944 9035 Acked Flush 0 Sent Done Done false
30945 9036 Acked Write 3 Sent Done Done false
30946 9037 Acked Write 3 Sent Done Done false
30947 9038 Acked Read 2 Sent Done Done false
30948 9039 Acked Write 9 Sent Done Done false
30949 9040 Acked Read 9 Sent Done Done false
30950 9041 Acked Read 7 Sent Done Done false
30951 9042 Acked Flush 0 Sent Done Done false
30952 9043 Acked Write 7 Sent Done Done false
30953 9044 Acked Flush 0 Sent Done Done false
30954 9045 Acked Read 9 Sent Done Done false
30955 9046 Acked Flush 0 Sent Done Done false
30956 9047 Acked Write 2 Sent Done Done false
30957 9048 Acked Read 5 Sent Done Done false
30958 9049 Acked Write 8 Sent Done Done false
30959 9050 Acked Read 10 Sent Done Done false
30960 9051 Acked Read 3 Sent Done Done false
30961 9052 Acked Write 2 Sent Done Done false
30962 9053 Acked Read 4 Sent Done Done false
30963 9054 Acked Read 10 Sent Done Done false
30964 9055 Acked Read 8 Sent Done Done false
30965 9056 Acked Read 9 Sent Done Done false
30966 9057 Acked Flush 0 Sent Done Done false
30967 9058 Acked Read 2 Sent Done Done false
30968 9059 Acked Read 9 Sent Done Done false
30969 9060 Acked Read 3 Sent Done Done false
30970 9061 Acked Read 1 Sent Done Done false
30971 9062 Acked Read 1 Sent Done Done false
30972 9063 Acked Read 7 Sent Done Done false
30973 9064 Acked Flush 0 Sent Done Done false
30974 9065 Acked Write 6 Sent Done Done false
30975 9066 Acked Flush 0 Sent Done Done false
30976 9067 Acked Read 6 Sent Done Done false
30977 9068 Acked Read 10 Sent Done Done false
30978 9069 Acked Write 7 Sent Done Done false
30979 9070 Acked Write 10 Sent Done Done false
30980 9071 Acked Read 4 Sent Done Done false
30981 9072 Acked Read 3 Sent Done Done false
30982 9073 Acked Write 5 Sent Done Done false
30983 9074 Acked Read 9 Sent Done Done false
30984 9075 Acked Read 10 Sent Done Done false
30985 9076 Acked Read 9 Sent Done Done false
30986 9077 Acked Read 9 Sent Done Done false
30987 9078 Acked Flush 0 Sent Done Done false
30988 9079 Acked Read 9 Sent Done Done false
30989 9080 Acked Write 10 Sent Done Done false
30990 9081 Acked Flush 0 Sent Done Done false
30991 9082 Acked Read 10 Sent Done Done false
30992 9083 Acked Write 6 Sent Done Done false
30993 9084 Acked Read 3 Sent Done Done false
30994 9085 Acked Read 3 Sent Done Done false
30995 9086 Acked Flush 0 Sent Done Done false
30996 9087 Acked Read 6 Sent Done Done false
30997 9088 Acked Read 10 Sent Done Done false
30998 9089 Acked Read 1 Sent Done Done false
30999 9090 Acked Read 2 Sent Done Done false
31000 9091 Acked Write 8 Sent Done Done false
31001 9092 Acked Read 10 Sent Done Done false
31002 9093 Acked Write 10 Sent Done Done false
31003 9094 Acked Read 7 Sent Done Done false
31004 9095 Acked Read 4 Sent Done Done false
31005 9096 Acked Read 3 Sent Done Done false
31006 9097 Acked Read 3 Sent Done Done false
31007 9098 Acked Read 3 Sent Done Done false
31008 9099 Acked Read 6 Sent Done Done false
31009 9100 Acked Read 4 Sent Done Done false
31010 9101 Acked Write 7 Sent Done Done false
31011 9102 Acked Write 8 Sent Done Done false
31012 9103 Acked Read 10 Sent Done Done false
31013 9104 Acked Read 10 Sent Done Done false
31014 9105 Acked Write 1 Sent Done Done false
31015 9106 Acked Read 6 Sent Done Done false
31016 9107 Acked Write 9 Sent Done Done false
31017 9108 Acked Write 6 Sent Done Done false
31018 9109 Acked Write 7 Sent Done Done false
31019 9110 Acked Read 9 Sent Done Done false
31020 9111 Acked Write 5 Sent Done Done false
31021 9112 Acked Read 7 Sent Done Done false
31022 9113 Acked Write 4 Sent Done Done false
31023 9114 Acked Write 1 Sent Done Done false
31024 9115 Acked Write 1 Sent Done Done false
31025 9116 Acked Read 9 Sent Done Done false
31026 9117 Acked Read 10 Sent Done Done false
31027 9118 Acked Write 5 Sent Done Done false
31028 9119 Acked Read 3 Sent Done Done false
31029 9120 Acked Read 7 Sent Done Done false
31030 9121 Acked Write 3 Sent Done Done false
31031 9122 Acked Write 3 Sent Done Done false
31032 9123 Acked Write 9 Sent Done Done false
31033 9124 Acked Write 3 Sent Done Done false
31034 9125 Acked Read 8 Sent Done Done false
31035 9126 Acked Read 10 Sent Done Done false
31036 9127 Acked Read 4 Sent Done Done false
31037 9128 Acked Read 1 Sent Done Done false
31038 9129 Acked Read 4 Sent Done Done false
31039 9130 Acked Read 6 Sent Done Done false
31040 9131 Acked Read 7 Sent Done Done false
31041 9132 Acked Write 2 Sent Done Done false
31042 9133 Acked Write 6 Sent Done Done false
31043 9134 Acked Read 10 Sent Done Done false
31044 9135 Acked Flush 0 Sent Done Done false
31045 9136 Acked Read 7 Sent Done Done false
31046 9137 Acked Read 9 Sent Done Done false
31047 9138 Acked Write 4 Sent Done Done false
31048 9139 Acked Read 4 Sent Done Done false
31049 9140 Acked Write 7 Sent Done Done false
31050 9141 Acked Flush 0 Sent Done Done false
31051 9142 Acked Read 8 Sent Done Done false
31052 9143 Acked Write 10 Sent Done Done false
31053 9144 Acked Read 4 Sent Done Done false
31054 9145 Acked Read 1 Sent Done Done false
31055 9146 Acked Flush 0 Sent Done Done false
31056 9147 Acked Write 3 Sent Done Done false
31057 STATES DS:0 DS:1 DS:2 TOTAL
31058 Sent 301 0 0 301
31059 Done 0 301 301 602
31060 Skipped 0 0 0 0
31061 Error 0 0 0 0
31062 Last Flush: Some(JobId(8846)) Some(JobId(9146)) Some(JobId(9146))
31063 Downstairs last five completed: 8846 8845 8844 8843 8842
31064 Upstairs last five completed: 9147 9146 9145 9144 9143
31065 CLIENT: Up:0 ds:301 act:2
310662025-02-25T20:53:20.094ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
310672025-02-25T20:53:20.703ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
310682025-02-25T20:53:21.705ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
310692025-02-25T20:53:22.707ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31070 [8810] got start action
31071 [8810] I am going to restart
31072 Starting downstairs at port 8810
31073 Make output file at "/tmp/dsc/downstairs-8810.txt"
31074 [8810][0] reports Starting
31075 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1233)
31076 [8810][0] reports Running
31077 ----------------------------------------------------------------
31078 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31079 States: Offline Active Active
31080 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31081 8847 Acked Write 1 Sent Done Done false
31082 8848 Acked Flush 0 Sent Done Done false
31083 8849 Acked Read 9 Sent Done Done false
31084 8850 Acked Flush 0 Sent Done Done false
31085 8851 Acked Write 3 Sent Done Done false
31086 8852 Acked Write 6 Sent Done Done false
31087 8853 Acked Read 1 Sent Done Done false
31088 8854 Acked Write 4 Sent Done Done false
31089 8855 Acked Write 10 Sent Done Done false
31090 8856 Acked Read 6 Sent Done Done false
31091 8857 Acked Write 5 Sent Done Done false
31092 8858 Acked Read 3 Sent Done Done false
31093 8859 Acked Flush 0 Sent Done Done false
31094 8860 Acked Write 6 Sent Done Done false
31095 8861 Acked Read 7 Sent Done Done false
31096 8862 Acked Read 1 Sent Done Done false
31097 8863 Acked Read 7 Sent Done Done false
31098 8864 Acked Write 8 Sent Done Done false
31099 8865 Acked Read 2 Sent Done Done false
31100 8866 Acked Flush 0 Sent Done Done false
31101 8867 Acked Read 4 Sent Done Done false
31102 8868 Acked Read 7 Sent Done Done false
31103 8869 Acked Write 1 Sent Done Done false
31104 8870 Acked Write 4 Sent Done Done false
31105 8871 Acked Read 6 Sent Done Done false
31106 8872 Acked Read 1 Sent Done Done false
31107 8873 Acked Read 6 Sent Done Done false
31108 8874 Acked Write 9 Sent Done Done false
31109 8875 Acked Read 10 Sent Done Done false
31110 8876 Acked Read 9 Sent Done Done false
31111 8877 Acked Write 3 Sent Done Done false
31112 8878 Acked Flush 0 Sent Done Done false
31113 8879 Acked Read 6 Sent Done Done false
31114 8880 Acked Read 8 Sent Done Done false
31115 8881 Acked Read 4 Sent Done Done false
31116 8882 Acked Write 3 Sent Done Done false
31117 8883 Acked Write 1 Sent Done Done false
31118 8884 Acked Write 9 Sent Done Done false
31119 8885 Acked Read 6 Sent Done Done false
31120 8886 Acked Write 4 Sent Done Done false
31121 8887 Acked Flush 0 Sent Done Done false
31122 8888 Acked Read 5 Sent Done Done false
31123 8889 Acked Read 1 Sent Done Done false
31124 8890 Acked Read 3 Sent Done Done false
31125 8891 Acked Write 4 Sent Done Done false
31126 8892 Acked Read 8 Sent Done Done false
31127 8893 Acked Write 5 Sent Done Done false
31128 8894 Acked Flush 0 Sent Done Done false
31129 8895 Acked Write 7 Sent Done Done false
31130 8896 Acked Read 7 Sent Done Done false
31131 8897 Acked Write 9 Sent Done Done false
31132 8898 Acked Flush 0 Sent Done Done false
31133 8899 Acked Read 9 Sent Done Done false
31134 8900 Acked Flush 0 Sent Done Done false
31135 8901 Acked Write 3 Sent Done Done false
31136 8902 Acked Write 8 Sent Done Done false
31137 8903 Acked Write 6 Sent Done Done false
31138 8904 Acked Read 10 Sent Done Done false
31139 8905 Acked Write 4 Sent Done Done false
31140 8906 Acked Write 6 Sent Done Done false
31141 8907 Acked Write 1 Sent Done Done false
31142 8908 Acked Read 1 Sent Done Done false
31143 8909 Acked Write 6 Sent Done Done false
31144 8910 Acked Read 6 Sent Done Done false
31145 8911 Acked Read 1 Sent Done Done false
31146 8912 Acked Flush 0 Sent Done Done false
31147 8913 Acked Read 5 Sent Done Done false
31148 8914 Acked Flush 0 Sent Done Done false
31149 8915 Acked Read 3 Sent Done Done false
31150 8916 Acked Read 8 Sent Done Done false
31151 8917 Acked Read 2 Sent Done Done false
31152 8918 Acked Write 8 Sent Done Done false
31153 8919 Acked Read 7 Sent Done Done false
31154 8920 Acked Read 6 Sent Done Done false
31155 8921 Acked Read 9 Sent Done Done false
31156 8922 Acked Read 2 Sent Done Done false
31157 8923 Acked Read 7 Sent Done Done false
31158 8924 Acked Read 8 Sent Done Done false
31159 8925 Acked Write 5 Sent Done Done false
31160 8926 Acked Read 9 Sent Done Done false
31161 8927 Acked Write 2 Sent Done Done false
31162 8928 Acked Write 4 Sent Done Done false
31163 8929 Acked Read 6 Sent Done Done false
31164 8930 Acked Read 7 Sent Done Done false
31165 8931 Acked Read 3 Sent Done Done false
31166 8932 Acked Write 7 Sent Done Done false
31167 8933 Acked Read 2 Sent Done Done false
31168 8934 Acked Write 7 Sent Done Done false
31169 8935 Acked Flush 0 Sent Done Done false
31170 8936 Acked Read 10 Sent Done Done false
31171 8937 Acked Write 3 Sent Done Done false
31172 8938 Acked Write 3 Sent Done Done false
31173 8939 Acked Write 5 Sent Done Done false
31174 8940 Acked Write 3 Sent Done Done false
31175 8941 Acked Read 7 Sent Done Done false
31176 8942 Acked Read 3 Sent Done Done false
31177 8943 Acked Read 3 Sent Done Done false
31178 8944 Acked Read 7 Sent Done Done false
31179 8945 Acked Write 2 Sent Done Done false
31180 8946 Acked Read 3 Sent Done Done false
31181 8947 Acked Read 2 Sent Done Done false
31182 8948 Acked Read 5 Sent Done Done false
31183 8949 Acked Read 7 Sent Done Done false
31184 8950 Acked Write 10 Sent Done Done false
31185 8951 Acked Read 5 Sent Done Done false
31186 8952 Acked Read 1 Sent Done Done false
31187 8953 Acked Write 3 Sent Done Done false
31188 8954 Acked Write 1 Sent Done Done false
31189 8955 Acked Flush 0 Sent Done Done false
31190 8956 Acked Read 4 Sent Done Done false
31191 8957 Acked Read 8 Sent Done Done false
31192 8958 Acked Flush 0 Sent Done Done false
31193 8959 Acked Flush 0 Sent Done Done false
31194 8960 Acked Read 10 Sent Done Done false
31195 8961 Acked Read 4 Sent Done Done false
31196 8962 Acked Write 8 Sent Done Done false
31197 8963 Acked Read 10 Sent Done Done false
31198 8964 Acked Flush 0 Sent Done Done false
31199 8965 Acked Read 6 Sent Done Done false
31200 8966 Acked Write 9 Sent Done Done false
31201 8967 Acked Read 4 Sent Done Done false
31202 8968 Acked Flush 0 Sent Done Done false
31203 8969 Acked Flush 0 Sent Done Done false
31204 8970 Acked Flush 0 Sent Done Done false
31205 8971 Acked Read 7 Sent Done Done false
31206 8972 Acked Write 8 Sent Done Done false
31207 8973 Acked Write 7 Sent Done Done false
31208 8974 Acked Write 1 Sent Done Done false
31209 8975 Acked Write 1 Sent Done Done false
31210 8976 Acked Write 2 Sent Done Done false
31211 8977 Acked Read 5 Sent Done Done false
31212 8978 Acked Write 8 Sent Done Done false
31213 8979 Acked Read 7 Sent Done Done false
31214 8980 Acked Write 6 Sent Done Done false
31215 8981 Acked Read 6 Sent Done Done false
31216 8982 Acked Read 1 Sent Done Done false
31217 8983 Acked Write 3 Sent Done Done false
31218 8984 Acked Write 10 Sent Done Done false
31219 8985 Acked Write 5 Sent Done Done false
31220 8986 Acked Read 5 Sent Done Done false
31221 8987 Acked Read 3 Sent Done Done false
31222 8988 Acked Write 7 Sent Done Done false
31223 8989 Acked Write 5 Sent Done Done false
31224 8990 Acked Read 2 Sent Done Done false
31225 8991 Acked Read 9 Sent Done Done false
31226 8992 Acked Read 10 Sent Done Done false
31227 8993 Acked Read 1 Sent Done Done false
31228 8994 Acked Read 2 Sent Done Done false
31229 8995 Acked Write 3 Sent Done Done false
31230 8996 Acked Write 7 Sent Done Done false
31231 8997 Acked Read 10 Sent Done Done false
31232 8998 Acked Write 4 Sent Done Done false
31233 8999 Acked Write 9 Sent Done Done false
31234 9000 Acked Read 10 Sent Done Done false
31235 9001 Acked Write 9 Sent Done Done false
31236 9002 Acked Read 1 Sent Done Done false
31237 9003 Acked Read 2 Sent Done Done false
31238 9004 Acked Read 8 Sent Done Done false
31239 9005 Acked Flush 0 Sent Done Done false
31240 9006 Acked Write 4 Sent Done Done false
31241 9007 Acked Flush 0 Sent Done Done false
31242 9008 Acked Read 6 Sent Done Done false
31243 9009 Acked Read 7 Sent Done Done false
31244 9010 Acked Read 5 Sent Done Done false
31245 9011 Acked Read 10 Sent Done Done false
31246 9012 Acked Read 7 Sent Done Done false
31247 9013 Acked Read 1 Sent Done Done false
31248 9014 Acked Read 10 Sent Done Done false
31249 9015 Acked Flush 0 Sent Done Done false
31250 9016 Acked Read 6 Sent Done Done false
31251 9017 Acked Write 2 Sent Done Done false
31252 9018 Acked Read 10 Sent Done Done false
31253 9019 Acked Read 2 Sent Done Done false
31254 9020 Acked Read 7 Sent Done Done false
31255 9021 Acked Write 3 Sent Done Done false
31256 9022 Acked Write 8 Sent Done Done false
31257 9023 Acked Read 8 Sent Done Done false
31258 9024 Acked Write 8 Sent Done Done false
31259 9025 Acked Read 9 Sent Done Done false
31260 9026 Acked Read 6 Sent Done Done false
31261 9027 Acked Write 2 Sent Done Done false
31262 9028 Acked Read 9 Sent Done Done false
31263 9029 Acked Read 5 Sent Done Done false
31264 9030 Acked Flush 0 Sent Done Done false
31265 9031 Acked Write 4 Sent Done Done false
31266 9032 Acked Read 3 Sent Done Done false
31267 9033 Acked Write 1 Sent Done Done false
31268 9034 Acked Read 5 Sent Done Done false
31269 9035 Acked Flush 0 Sent Done Done false
31270 9036 Acked Write 3 Sent Done Done false
31271 9037 Acked Write 3 Sent Done Done false
31272 9038 Acked Read 2 Sent Done Done false
31273 9039 Acked Write 9 Sent Done Done false
31274 9040 Acked Read 9 Sent Done Done false
31275 9041 Acked Read 7 Sent Done Done false
31276 9042 Acked Flush 0 Sent Done Done false
31277 9043 Acked Write 7 Sent Done Done false
31278 9044 Acked Flush 0 Sent Done Done false
31279 9045 Acked Read 9 Sent Done Done false
31280 9046 Acked Flush 0 Sent Done Done false
31281 9047 Acked Write 2 Sent Done Done false
31282 9048 Acked Read 5 Sent Done Done false
31283 9049 Acked Write 8 Sent Done Done false
31284 9050 Acked Read 10 Sent Done Done false
31285 9051 Acked Read 3 Sent Done Done false
31286 9052 Acked Write 2 Sent Done Done false
31287 9053 Acked Read 4 Sent Done Done false
31288 9054 Acked Read 10 Sent Done Done false
31289 9055 Acked Read 8 Sent Done Done false
31290 9056 Acked Read 9 Sent Done Done false
31291 9057 Acked Flush 0 Sent Done Done false
31292 9058 Acked Read 2 Sent Done Done false
31293 9059 Acked Read 9 Sent Done Done false
31294 9060 Acked Read 3 Sent Done Done false
31295 9061 Acked Read 1 Sent Done Done false
31296 9062 Acked Read 1 Sent Done Done false
31297 9063 Acked Read 7 Sent Done Done false
31298 9064 Acked Flush 0 Sent Done Done false
31299 9065 Acked Write 6 Sent Done Done false
31300 9066 Acked Flush 0 Sent Done Done false
31301 9067 Acked Read 6 Sent Done Done false
31302 9068 Acked Read 10 Sent Done Done false
31303 9069 Acked Write 7 Sent Done Done false
31304 9070 Acked Write 10 Sent Done Done false
31305 9071 Acked Read 4 Sent Done Done false
31306 9072 Acked Read 3 Sent Done Done false
31307 9073 Acked Write 5 Sent Done Done false
31308 9074 Acked Read 9 Sent Done Done false
31309 9075 Acked Read 10 Sent Done Done false
31310 9076 Acked Read 9 Sent Done Done false
31311 9077 Acked Read 9 Sent Done Done false
31312 9078 Acked Flush 0 Sent Done Done false
31313 9079 Acked Read 9 Sent Done Done false
31314 9080 Acked Write 10 Sent Done Done false
31315 9081 Acked Flush 0 Sent Done Done false
31316 9082 Acked Read 10 Sent Done Done false
31317 9083 Acked Write 6 Sent Done Done false
31318 9084 Acked Read 3 Sent Done Done false
31319 9085 Acked Read 3 Sent Done Done false
31320 9086 Acked Flush 0 Sent Done Done false
31321 9087 Acked Read 6 Sent Done Done false
31322 9088 Acked Read 10 Sent Done Done false
31323 9089 Acked Read 1 Sent Done Done false
31324 9090 Acked Read 2 Sent Done Done false
31325 9091 Acked Write 8 Sent Done Done false
31326 9092 Acked Read 10 Sent Done Done false
31327 9093 Acked Write 10 Sent Done Done false
31328 9094 Acked Read 7 Sent Done Done false
31329 9095 Acked Read 4 Sent Done Done false
31330 9096 Acked Read 3 Sent Done Done false
31331 9097 Acked Read 3 Sent Done Done false
31332 9098 Acked Read 3 Sent Done Done false
31333 9099 Acked Read 6 Sent Done Done false
31334 9100 Acked Read 4 Sent Done Done false
31335 9101 Acked Write 7 Sent Done Done false
31336 9102 Acked Write 8 Sent Done Done false
31337 9103 Acked Read 10 Sent Done Done false
31338 9104 Acked Read 10 Sent Done Done false
31339 9105 Acked Write 1 Sent Done Done false
31340 9106 Acked Read 6 Sent Done Done false
31341 9107 Acked Write 9 Sent Done Done false
31342 9108 Acked Write 6 Sent Done Done false
31343 9109 Acked Write 7 Sent Done Done false
31344 9110 Acked Read 9 Sent Done Done false
31345 9111 Acked Write 5 Sent Done Done false
31346 9112 Acked Read 7 Sent Done Done false
31347 9113 Acked Write 4 Sent Done Done false
31348 9114 Acked Write 1 Sent Done Done false
31349 9115 Acked Write 1 Sent Done Done false
31350 9116 Acked Read 9 Sent Done Done false
31351 9117 Acked Read 10 Sent Done Done false
31352 9118 Acked Write 5 Sent Done Done false
31353 9119 Acked Read 3 Sent Done Done false
31354 9120 Acked Read 7 Sent Done Done false
31355 9121 Acked Write 3 Sent Done Done false
31356 9122 Acked Write 3 Sent Done Done false
31357 9123 Acked Write 9 Sent Done Done false
31358 9124 Acked Write 3 Sent Done Done false
31359 9125 Acked Read 8 Sent Done Done false
31360 9126 Acked Read 10 Sent Done Done false
31361 9127 Acked Read 4 Sent Done Done false
31362 9128 Acked Read 1 Sent Done Done false
31363 9129 Acked Read 4 Sent Done Done false
31364 9130 Acked Read 6 Sent Done Done false
31365 9131 Acked Read 7 Sent Done Done false
31366 9132 Acked Write 2 Sent Done Done false
31367 9133 Acked Write 6 Sent Done Done false
31368 9134 Acked Read 10 Sent Done Done false
31369 9135 Acked Flush 0 Sent Done Done false
31370 9136 Acked Read 7 Sent Done Done false
31371 9137 Acked Read 9 Sent Done Done false
31372 9138 Acked Write 4 Sent Done Done false
31373 9139 Acked Read 4 Sent Done Done false
31374 9140 Acked Write 7 Sent Done Done false
31375 9141 Acked Flush 0 Sent Done Done false
31376 9142 Acked Read 8 Sent Done Done false
31377 9143 Acked Write 10 Sent Done Done false
31378 9144 Acked Read 4 Sent Done Done false
31379 9145 Acked Read 1 Sent Done Done false
31380 9146 Acked Flush 0 Sent Done Done false
31381 9147 Acked Write 3 Sent Done Done false
31382 STATES DS:0 DS:1 DS:2 TOTAL
31383 Sent 301 0 0 301
31384 Done 0 301 301 602
31385 Skipped 0 0 0 0
31386 Error 0 0 0 0
31387 Last Flush: Some(JobId(8846)) Some(JobId(9146)) Some(JobId(9146))
31388 Downstairs last five completed: 8846 8845 8844 8843 8842
31389 Upstairs last five completed: 9147 9146 9145 9144 9143
31390 CLIENT: Up:0 ds:301 act:2
313912025-02-25T20:53:24.097ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313922025-02-25T20:53:24.709ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313932025-02-25T20:53:25.329ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313942025-02-25T20:53:25.330ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53667) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313952025-02-25T20:53:25.331ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313962025-02-25T20:53:25.331ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313972025-02-25T20:53:25.331ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313982025-02-25T20:53:25.331ZINFOcrucible: send last flush ID to this DS: Some(JobId(8846)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
313992025-02-25T20:53:25.331ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
314002025-02-25T20:53:25.331ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8846)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
314012025-02-25T20:53:25.331ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8846)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31402 ----------------------------------------------------------------
31403 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31404 Downstairs last five completed: 9148 9147 9146 9145 9144
31405 Upstairs last five completed: 9148 9147 9146 9145 9144
31406 CLIENT: Up:0 ds:0 act:3
31407 Replay: All jobs finished
31408 Main task has work to do, go find it
31409 got dsc Stop(0)
31410 stop 0
31411 Feb 25 20:53:27.599 INFO request completed, latency_us: [8810] Got stop action so:false kr:false
31412 411, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c9619fc0-9bb4-4bca-ae50-d1e8b489e66c, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
314132025-02-25T20:53:27.599ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
314142025-02-25T20:53:27.599ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
314152025-02-25T20:53:27.599ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
314162025-02-25T20:53:27.600ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
31417 Feb 25 20:53:27.600 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8c9d6574-81c3-489a-b4c1-821ba91133de, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
31418 [8810] Exited with: ExitStatus(unix_wait_status(9))
31419 [8810][0] reports Exit
31420 Feb 25 20:53:31.602 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2b3b9b5f-df66-41e1-b7b0-ff64ed629a34, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
31421 001/300 Write block 3054 len 32768 data: 5 5 4 4 4 5 5 4
31422 002/300 Write block 285 len 32768 data: 4 4 3 4 4 3 2 2
31423 003/300 Read block 3328 len 16384
31424 004/300 Read block 4467 len 16384
31425 005/300 Write block 2803 len 20480 data: 7 8 7 5 5
31426 006/300 Write block 4590 len 8192 data: 6 5
31427 007/300 Read block 179 len 4096
31428 008/300 Read block 448 len 24576
31429 009/300 Read block 1123 len 24576
31430 010/300 Read block 4979 len 20480
31431 011/300 Read block 170 len 36864
31432 012/300 Read block 9 len 32768
31433 013/300 Write block 2116 len 12288 data: 5 4 4
31434 014/300 Flush
31435 015/300 Write block 4873 len 40960 data: 5 6 6 7 7 5 5 4 7 5
31436 016/300 Flush
31437 017/300 Read block 2232 len 12288
31438 018/300 Flush
31439 019/300 Write block 4670 len 8192 data: 6 4
31440 020/300 Read block 4255 len 16384
31441 021/300 Read block 4282 len 28672
31442 022/300 Read block 2918 len 20480
31443 023/300 Read block 4477 len 4096
31444 024/300 Write block 875 len 32768 data: 5 5 5 5 5 4 2 5
31445 025/300 Read block 3543 len 36864
31446 026/300 Write block 1439 len 40960 data: 3 2 2 2 2 4 4 5 5 4
31447 027/300 Write block 4926 len 28672 data: 5 4 4 4 4 4 6
31448 028/300 Write block 2303 len 36864 data: 6 5 6 6 7 5 6 6 6
31449 029/300 Read block 4982 len 40960
31450 030/300 Flush
31451 031/300 Read block 4236 len 16384
31452 032/300 Read block 638 len 32768
31453 033/300 Read block 677 len 32768
31454 034/300 Read block 3484 len 40960
31455 035/300 Read block 2833 len 4096
31456 036/300 Read block 626 len 24576
31457 037/300 Read block 4594 len 4096
31458 038/300 Read block 605 len 40960
31459 039/300 Read block 4723 len 32768
31460 040/300 Write block 2248 len 16384 data: 5 6 6 6
31461 041/300 Flush
31462 042/300 Read block 4085 len 16384
31463 043/300 Write block 4277 len 12288 data: 4 6 8
31464 044/300 Flush
31465 045/300 Write block 2250 len 12288 data: 7 7 7
31466 046/300 Write block 1584 len 40960 data: 10 11 10 8 5 5 4 4 4 4
31467 047/300 Write block 2887 len 12288 data: 4 4 7
31468 048/300 Flush
31469 049/300 Read block 3983 len 12288
31470 050/300 Write block 719 len 28672 data: 4 6 8 8 6 7 9
31471 051/300 Read block 2301 len 20480
31472 052/300 Read block 3479 len 16384
31473 053/300 Read block 4097 len 4096
31474 054/300 Write block 2948 len 36864 data: 4 3 4 6 6 7 6 6 8
31475 055/300 Write block 1350 len 16384 data: 2 3 2 2
31476 056/300 Read block 2472 len 28672
31477 057/300 Write block 712 len 28672 data: 6 8 6 6 5 4 4
31478 058/300 Write block 1825 len 8192 data: 7 7
31479 059/300 Read block 2783 len 40960
31480 060/300 Flush
31481 061/300 Flush
31482 062/300 Write block 365 len 16384 data: 2 4 4 5
31483 063/300 Read block 3276 len 24576
31484 064/300 Read block 3956 len 32768
31485 065/300 Read block 3803 len 36864
31486 066/300 Flush
31487 067/300 Write block 2170 len 16384 data: 7 6 5 5
31488 068/300 Flush
31489 069/300 Read block 3423 len 32768
31490 070/300 Write block 1714 len 12288 data: 5 6 5
31491 071/300 Write block 1494 len 12288 data: 5 5 4
31492 072/300 Read block 2251 len 4096
31493 073/300 Write block 2114 len 24576 data: 5 5 6 5 5 4
31494 074/300 Flush
31495 075/300 Read block 2301 len 12288
31496 076/300 Read block 1596 len 24576
31497 077/300 Write block 4380 len 28672 data: 6 6 6 7 6 5 6
31498 078/300 Write block 2203 len 36864 data: 4 4 3 3 4 7 6 5 3
31499 079/300 Write block 2469 len 24576 data: 6 7 10 9 8 7
31500 080/300 Write block 1250 len 8192 data: 4 4
31501 081/300 Read block 720 len 32768
31502 082/300 Write block 4205 len 4096 data: 5
31503 083/300 Write block 3692 len 36864 data: 5 4 2 2 4 5 6 6 6
31504 084/300 Read block 270 len 4096
31505 085/300 Read block 3127 len 16384
31506 086/300 Write block 2796 len 36864 data: 5 6 5 6 9 7 6 8 9
31507 087/300 Read block 1123 len 16384
31508 088/300 Read block 1368 len 40960
31509 089/300 Write block 1716 len 12288 data: 6 5 5
31510 090/300 Write block 4762 len 24576 data: 6 6 5 4 4 3
31511 091/300 Read block 3216 len 20480
31512 092/300 Write block 3273 len 40960 data: 5 4 4 6 6 7 5 6 7 5
31513 093/300 Write block 440 len 12288 data: 7 7 7
31514 094/300 Write block 2937 len 24576 data: 2 4 4 4 4 5
31515 095/300 Read block 3143 len 4096
31516 096/300 Read block 3976 len 4096
31517 097/300 Read block 1731 len 28672
31518 098/300 Write block 2701 len 16384 data: 6 6 4 4
31519 099/300 Write block 1161 len 4096 data: 5
31520 100/300 Write block 4851 len 32768 data: 6 5 4 4 4 3 2 2
31521 101/300 Write block 2869 len 4096 data: 7
31522 102/300 Write block 1342 len 12288 data: 5 5 5
31523 103/300 Write block 1208 len 20480 data: 7 7 8 8 8
31524 104/300 Read block 4429 len 16384
31525 105/300 Flush
31526 106/300 Write block 2430 len 12288 data: 7 7 6
31527 107/300 Read block 2737 len 24576
31528 108/300 Read block 4598 len 16384
31529 109/300 Read block 3851 len 32768
31530 110/300 Write block 623 len 16384 data: 5 7 7 6
31531 111/300 Write block 2362 len 40960 data: 6 8 7 6 8 7 5 4 3 3
31532 112/300 Read block 3319 len 12288
31533 113/300 Write block 724 len 16384 data: 8 10 9 8
31534 114/300 Write block 65 len 36864 data: 7 5 4 4 5 6 5 5 6
31535 115/300 Write block 2883 len 16384 data: 8 5 4 3
31536 116/300 Read block 143 len 20480
31537 117/300 Write block 2395 len 8192 data: 4 4
31538 118/300 Write block 841 len 36864 data: 6 8 8 9 8 7 7 7 4
31539 119/300 Read block 1958 len 16384
31540 120/300 Write block 1418 len 28672 data: 6 6 7 7 7 5 6
31541 121/300 Flush
31542 122/300 Write block 4257 len 28672 data: 5 5 6 5 4 4 5
31543 123/300 Write block 2260 len 40960 data: 7 8 6 6 4 4 3 2 3 3
31544 124/300 Read block 408 len 8192
31545 125/300 Write block 95 len 36864 data: 7 5 5 5 5 4 5 4 6
31546 126/300 Write block 1872 len 8192 data: 6 4
31547 127/300 Read block 3686 len 24576
31548 128/300 Write block 1380 len 24576 data: 4 3 5 5 5 7
31549 129/300 Read block 1435 len 4096
31550 130/300 Write block 2100 len 8192 data: 5 6
31551 131/300 Read block 4201 len 24576
31552 132/300 Write block 1904 len 36864 data: 7 7 8 6 4 5 5 5 5
31553 133/300 Write block 2878 len 4096 data: 4
31554 134/300 Flush
31555 135/300 Read block 1443 len 12288
31556 136/300 Write block 2147 len 28672 data: 6 8 7 7 8 8 7
31557 137/300 Read block 2392 len 32768
31558 138/300 Write block 2336 len 4096 data: 7
31559 139/300 Read block 1779 len 32768
31560 140/300 Write block 3586 len 12288 data: 6 6 6
31561 141/300 Write block 1311 len 40960 data: 5 4 4 4 4 4 6 7 8 11
31562 142/300 Write block 522 len 20480 data: 4 3 4 5 4
31563 143/300 Read block 561 len 40960
31564 144/300 Read block 3127 len 4096
31565 145/300 Write block 1341 len 20480 data: 4 6 6 6 6
31566 146/300 Write block 3973 len 40960 data: 6 7 8 8 10 8 9 7 7 7
31567 147/300 Flush
31568 148/300 Read block 514 len 32768
31569 149/300 Write block 4611 len 20480 data: 6 7 7 7 7
31570 150/300 Read block 857 len 36864
31571 151/300 Read block 3877 len 40960
31572 152/300 Write block 4429 len 8192 data: 7 6
31573 153/300 Read block 1066 len 40960
31574 154/300 Read block 1117 len 20480
31575 155/300 Write block 2839 len 32768 data: 6 6 5 4 6 7 9 7
31576 156/300 Flush
31577 157/300 Write block 2780 len 40960 data: 4 6 6 7 5 4 3 4 3 3
31578 158/300 Read block 1250 len 4096
31579 159/300 Read block 837 len 8192
31580 160/300 Read block 3337 len 20480
31581 161/300 Read block 2622 len 32768
31582 162/300 Read block 4220 len 40960
31583 163/300 Read block 4921 len 24576
31584 164/300 Read block 4858 len 28672
31585 165/300 Read block 1212 len 4096
31586 166/300 Flush
31587 167/300 Read block 3725 len 4096
31588 168/300 Read block 2347 len 12288
31589 169/300 Read block 2533 len 8192
31590 170/300 Read block 221 len 12288
31591 171/300 Write block 962 len 36864 data: 7 7 7 5 7 7 7 5 5
31592 172/300 Read block 521 len 36864
31593 173/300 Flush
31594 174/300 Write block 1176 len 28672 data: 7 6 6 5 4 5 7
31595 175/300 Read block 3304 len 20480
31596 176/300 Write block 3046 len 20480 data: 5 6 6 8 6
31597 177/300 Read block 2475 len 20480
31598 178/300 Flush
31599 179/300 Read block 7 len 12288
31600 180/300 Write block 4239 len 28672 data: 3 2 2 4 5 5 7
31601 181/300 Write block 1591 len 40960 data: 5 5 5 3 5 5 5 6 6 7
31602 182/300 Write block 3166 len 16384 data: 7 4 5 5
31603 183/300 Write block 3495 len 32768 data: 5 5 6 4 4 3 4 4
31604 184/300 Read block 2242 len 32768
31605 185/300 Write block 2042 len 16384 data: 5 5 5 6
31606 186/300 Read block 3418 len 32768
31607 187/300 Write block 1963 len 40960 data: 5 6 6 6 6 6 5 6 4 5
31608 188/300 Read block 2005 len 32768
31609 189/300 Flush
31610 190/300 Read block 501 len 28672
31611 191/300 Read block 2944 len 12288
31612 192/300 Write block 2089 len 8192 data: 6 6
31613 193/300 Write block 3061 len 28672 data: 5 6 7 7 6 6 7
31614 194/300 Flush
31615 195/300 Read block 1587 len 28672
31616 196/300 Read block 4508 len 4096
31617 197/300 Flush
31618 198/300 Read block 1960 len 16384
31619 199/300 Write block 915 len 20480 data: 3 4 6 8 9
31620 200/300 Write block 594 len 40960 data: 5 5 5 5 3 2 4 4 5 5
31621 201/300 Write block 1237 len 24576 data: 7 8 6 6 4 5
31622 202/300 Write block 4825 len 40960 data: 6 5 3 3 3 2 4 5 5 5
31623 203/300 Read block 4436 len 4096
31624 204/300 Write block 2462 len 40960 data: 4 4 7 5 4 3 5 7 8 11
31625 205/300 Write block 2544 len 32768 data: 6 8 8 6 7 7 5 5
31626 206/300 Read block 154 len 36864
31627 207/300 Write block 567 len 40960 data: 4 4 4 2 3 3 4 3 4 6
31628 208/300 Read block 2689 len 12288
31629 209/300 Write block 1209 len 20480 data: 8 9 9 9 7
31630 210/300 Write block 3054 len 12288 data: 6 6 5
31631 211/300 Write block 3144 len 28672 data: 8 8 5 5 4 5 5
31632 212/300 Read block 233 len 28672
31633 213/300 Write block 3348 len 28672 data: 7 9 9 6 8 7 5
31634 214/300 Read block 1550 len 24576
31635 215/300 Write block 2360 len 40960 data: 4 4 7 9 8 7 9 8 6 5
31636 216/300 Read block 4573 len 20480
31637 217/300 Write block 4154 len 16384 data: 7 6 4 3
31638 218/300 Write block 1118 len 16384 data: 8 7 7 9
31639 219/300 Write block 4321 len 32768 data: 4 4 5 5 5 6 6 5
31640 220/300 Write block 491 len 28672 data: 5 5 6 7 7 6 6
31641 221/300 Write block 1949 len 20480 data: 8 8 7 6 8
31642 222/300 Write block 1911 len 16384 data: 6 6 5 5
31643 223/300 Flush
31644 224/300 Read block 2798 len 20480
31645 225/300 Read block 2532 len 24576
31646 226/300 Read block 653 len 8192
31647 227/300 Flush
31648 228/300 Write block 2466 len 28672 data: 5 4 6 8 9 12 10
31649 229/300 Write block 146 len 16384 data: 5 5 5 6
31650 230/300 Read block 4725 len 12288
31651 231/300 Write block 1629 len 36864 data: 6 6 6 7 5 6 6 6 5
31652 232/300 Read block 106 len 20480
31653 233/300 Flush
31654 234/300 Write block 4223 len 24576 data: 6 5 4 7 9 9
31655 235/300 Write block 4156 len 8192 data: 5 4
31656 236/300 Flush
31657 237/300 Read block 4348 len 4096
31658 238/300 Read block 4652 len 4096
31659 239/300 Read block 3581 len 12288
31660 240/300 Flush
31661 241/300 Write block 4738 len 16384 data: 4 4 6 6
31662 242/300 Read block 3624 len 16384
31663 243/300 Flush
31664 244/300 Read block 3449 len 4096
31665 245/300 Write block 566 len 20480 data: 4 5 5 5 3
31666 246/300 Write block 1303 len 40960 data: 5 6 6 6 5 5 5 5 6 5
31667 247/300 Write block 2505 len 28672 data: 2 2 5 5 4 4 5
31668 248/300 Write block 2618 len 12288 data: 7 8 10
31669 249/300 Read block 1105 len 36864
31670 250/300 Write block 726 len 24576 data: 10 9 11 11 9 8
31671 251/300 Flush
31672 252/300 Read block 4858 len 36864
31673 253/300 Read block 1951 len 12288
31674 254/300 Flush
31675 255/300 Flush
31676 256/300 Write block 4119 len 12288 data: 6 7 7
31677 257/300 Read block 1751 len 20480
31678 258/300 Read block 2150 len 16384
31679 259/300 Read block 1946 len 4096
31680 260/300 Read block 2140 len 8192
31681 261/300 Read block 4194 len 8192
31682 262/300 Write block 3478 len 4096 data: 7
31683 263/300 Read block 1601 len 12288
31684 264/300 Read block 4538 len 12288
31685 265/300 Read block 1132 len 32768
31686 266/300 Read block 2644 len 32768
31687 267/300 Read block 3116 len 8192
31688 268/300 Write block 3879 len 4096 data: 7
31689 269/300 Read block 27 len 16384
31690 270/300 Read block 1522 len 16384
31691 271/300 Write block 528 len 16384 data: 5 6 6 7
31692 272/300 Write block 3294 len 28672 data: 7 8 7 7 7 9 8
31693 273/300 Read block 4301 len 28672
31694 274/300 Read block 730 len 24576
31695 275/300 Read block 4403 len 24576
31696 276/300 Read block 825 len 12288
31697 277/300 Read block 3570 len 8192
31698 278/300 Read block 3209 len 28672
31699 279/300 Write block 1065 len 36864 data: 7 5 6 6 5 5 6 7 8
31700 280/300 Write block 2972 len 16384 data: 4 3 6 5
31701 281/300 Write block 3517 len 40960 data: 6 6 5 5 4 4 4 4 5 5
31702 282/300 Read block 3456 len 8192
31703 283/300 Read block 1946 len 24576
31704 284/300 Write block 4216 len 28672 data: 5 7 6 5 6 6 5
31705 285/300 Write block 3218 len 12288 data: 5 5 5
31706 286/300 Read block 2066 len 8192
31707 287/300 Read block 1247 len 20480
31708 288/300 Write block 1464 len 32768 data: 3 3 4 4 4 4 5 6
31709 289/300 Read block 1414 len 12288
31710 290/300 Flush
31711 291/300 Write block 521 len 28672 data: 3 5 4 5 6 5 5
31712 292/300 Write block 419 len 36864 data: 5 5 9 9 8 8 9 9 8
31713 293/300 Write block 1225 len 4096 data: 4
31714 294/300 Write block 2685 len 8192 data: 6 3
31715 295/300 Read block 4307 len 28672
31716 296/300 Read block 3138 len 20480
31717 297/300 Read block 3317 len 8192
31718 298/300 Flush
31719 299/300 Read block 788 len 8192
31720 300/300 Read block 4285 len 40960
31721 301/300 Flush
31722 Main task has work to do, go find it
31723 got dsc Start(0)
31724 start 0
31725 Feb 25 20:53:31.839 INFO request completed, latency_us: 289, response_code: 204, uri: /start/cid/0, method: POST, req_id: 85800916-692a-4436-b17e-4405e26bbf13, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
31726 Replay: started 0, returned:Ok(())
31727 ----------------------------------------------------------------
31728 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31729 States: Offline Active Active
31730 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31731 9149 Acked Write 8 Sent Done Done false
31732 9150 Acked Write 8 Sent Done Done false
31733 9151 Acked Read 4 Sent Done Done false
31734 9152 Acked Read 4 Sent Done Done false
31735 9153 Acked Write 5 Sent Done Done false
31736 9154 Acked Write 2 Sent Done Done false
31737 9155 Acked Read 1 Sent Done Done false
31738 9156 Acked Read 6 Sent Done Done false
31739 9157 Acked Read 6 Sent Done Done false
31740 9158 Acked Read 5 Sent Done Done false
31741 9159 Acked Read 9 Sent Done Done false
31742 9160 Acked Read 8 Sent Done Done false
31743 9161 Acked Write 3 Sent Done Done false
31744 9162 Acked Flush 0 Sent Done Done false
31745 9163 Acked Write 10 Sent Done Done false
31746 9164 Acked Flush 0 Sent Done Done false
31747 9165 Acked Read 3 Sent Done Done false
31748 9166 Acked Flush 0 Sent Done Done false
31749 9167 Acked Write 2 Sent Done Done false
31750 9168 Acked Read 4 Sent Done Done false
31751 9169 Acked Read 7 Sent Done Done false
31752 9170 Acked Read 5 Sent Done Done false
31753 9171 Acked Read 1 Sent Done Done false
31754 9172 Acked Write 8 Sent Done Done false
31755 9173 Acked Read 9 Sent Done Done false
31756 9174 Acked Write 10 Sent Done Done false
31757 9175 Acked Write 7 Sent Done Done false
31758 9176 Acked Write 9 Sent Done Done false
31759 9177 Acked Read 10 Sent Done Done false
31760 9178 Acked Flush 0 Sent Done Done false
31761 9179 Acked Read 4 Sent Done Done false
31762 9180 Acked Read 8 Sent Done Done false
31763 9181 Acked Read 8 Sent Done Done false
31764 9182 Acked Read 10 Sent Done Done false
31765 9183 Acked Read 1 Sent Done Done false
31766 9184 Acked Read 6 Sent Done Done false
31767 9185 Acked Read 1 Sent Done Done false
31768 9186 Acked Read 10 Sent Done Done false
31769 9187 Acked Read 8 Sent Done Done false
31770 9188 Acked Write 4 Sent Done Done false
31771 9189 Acked Flush 0 Sent Done Done false
31772 9190 Acked Read 4 Sent Done Done false
31773 9191 Acked Write 3 Sent Done Done false
31774 9192 Acked Flush 0 Sent Done Done false
31775 9193 Acked Write 3 Sent Done Done false
31776 9194 Acked Write 10 Sent Done Done false
31777 9195 Acked Write 3 Sent Done Done false
31778 9196 Acked Flush 0 Sent Done Done false
31779 9197 Acked Read 3 Sent Done Done false
31780 9198 Acked Write 7 Sent Done Done false
31781 9199 Acked Read 5 Sent Done Done false
31782 9200 Acked Read 4 Sent Done Done false
31783 9201 Acked Read 1 Sent Done Done false
31784 9202 Acked Write 9 Sent Done Done false
31785 9203 Acked Write 4 Sent Done Done false
31786 9204 Acked Read 7 Sent Done Done false
31787 9205 Acked Write 7 Sent Done Done false
31788 9206 Acked Write 2 Sent Done Done false
31789 9207 Acked Read 10 Sent Done Done false
31790 9208 Acked Flush 0 Sent Done Done false
31791 9209 Acked Flush 0 Sent Done Done false
31792 9210 Acked Write 4 Sent Done Done false
31793 9211 Acked Read 6 Sent Done Done false
31794 9212 Acked Read 8 Sent Done Done false
31795 9213 Acked Read 9 Sent Done Done false
31796 9214 Acked Flush 0 Sent Done Done false
31797 9215 Acked Write 4 Sent Done Done false
31798 9216 Acked Flush 0 Sent Done Done false
31799 9217 Acked Read 8 Sent Done Done false
31800 9218 Acked Write 3 Sent Done Done false
31801 9219 Acked Write 3 Sent Done Done false
31802 9220 Acked Read 1 Sent Done Done false
31803 9221 Acked Write 6 Sent Done Done false
31804 9222 Acked Flush 0 Sent Done Done false
31805 9223 Acked Read 3 Sent Done Done false
31806 9224 Acked Read 6 Sent Done Done false
31807 9225 Acked Write 7 Sent Done Done false
31808 9226 Acked Write 9 Sent Done Done false
31809 9227 Acked Write 6 Sent Done Done false
31810 9228 Acked Write 2 Sent Done Done false
31811 9229 Acked Read 8 Sent Done Done false
31812 9230 Acked Write 1 Sent Done Done false
31813 9231 Acked Write 9 Sent Done Done false
31814 9232 Acked Read 1 Sent Done Done false
31815 9233 Acked Read 4 Sent Done Done false
31816 9234 Acked Write 9 Sent Done Done false
31817 9235 Acked Read 4 Sent Done Done false
31818 9236 Acked Read 10 Sent Done Done false
31819 9237 Acked Write 3 Sent Done Done false
31820 9238 Acked Write 6 Sent Done Done false
31821 9239 Acked Read 5 Sent Done Done false
31822 9240 Acked Write 10 Sent Done Done false
31823 9241 Acked Write 3 Sent Done Done false
31824 9242 Acked Write 6 Sent Done Done false
31825 9243 Acked Read 1 Sent Done Done false
31826 9244 Acked Read 1 Sent Done Done false
31827 9245 Acked Read 7 Sent Done Done false
31828 9246 Acked Write 4 Sent Done Done false
31829 9247 Acked Write 1 Sent Done Done false
31830 9248 Acked Write 8 Sent Done Done false
31831 9249 Acked Write 1 Sent Done Done false
31832 9250 Acked Write 3 Sent Done Done false
31833 9251 Acked Write 5 Sent Done Done false
31834 9252 Acked Read 4 Sent Done Done false
31835 9253 Acked Flush 0 Sent Done Done false
31836 9254 Acked Write 3 Sent Done Done false
31837 9255 Acked Read 6 Sent Done Done false
31838 9256 Acked Read 4 Sent Done Done false
31839 9257 Acked Read 8 Sent Done Done false
31840 9258 Acked Write 4 Sent Done Done false
31841 9259 Acked Write 10 Sent Done Done false
31842 9260 Acked Read 3 Sent Done Done false
31843 9261 Acked Write 4 Sent Done Done false
31844 9262 Acked Write 9 Sent Done Done false
31845 9263 Acked Write 4 Sent Done Done false
31846 9264 Acked Read 5 Sent Done Done false
31847 9265 Acked Write 2 Sent Done Done false
31848 9266 Acked Write 9 Sent Done Done false
31849 9267 Acked Read 4 Sent Done Done false
31850 9268 Acked Write 7 Sent Done Done false
31851 9269 Acked Flush 0 Sent Done Done false
31852 9270 Acked Write 7 Sent Done Done false
31853 9271 Acked Write 10 Sent Done Done false
31854 9272 Acked Read 2 Sent Done Done false
31855 9273 Acked Write 9 Sent Done Done false
31856 9274 Acked Write 2 Sent Done Done false
31857 9275 Acked Read 6 Sent Done Done false
31858 9276 Acked Write 6 Sent Done Done false
31859 9277 Acked Read 1 Sent Done Done false
31860 9278 Acked Write 2 Sent Done Done false
31861 9279 Acked Read 6 Sent Done Done false
31862 9280 Acked Write 9 Sent Done Done false
31863 9281 Acked Write 1 Sent Done Done false
31864 9282 Acked Flush 0 Sent Done Done false
31865 9283 Acked Read 3 Sent Done Done false
31866 9284 Acked Write 7 Sent Done Done false
31867 9285 Acked Read 8 Sent Done Done false
31868 9286 Acked Write 1 Sent Done Done false
31869 9287 Acked Read 8 Sent Done Done false
31870 9288 Acked Write 3 Sent Done Done false
31871 9289 Acked Write 10 Sent Done Done false
31872 9290 Acked Write 5 Sent Done Done false
31873 9291 Acked Read 10 Sent Done Done false
31874 9292 Acked Read 1 Sent Done Done false
31875 9293 Acked Write 5 Sent Done Done false
31876 9294 Acked Write 10 Sent Done Done false
31877 9295 Acked Flush 0 Sent Done Done false
31878 9296 Acked Read 8 Sent Done Done false
31879 9297 Acked Write 5 Sent Done Done false
31880 9298 Acked Read 9 Sent Done Done false
31881 9299 Acked Read 10 Sent Done Done false
31882 9300 Acked Write 2 Sent Done Done false
31883 9301 Acked Read 10 Sent Done Done false
31884 9302 Acked Read 5 Sent Done Done false
31885 9303 Acked Write 8 Sent Done Done false
31886 9304 Acked Flush 0 Sent Done Done false
31887 9305 Acked Write 10 Sent Done Done false
31888 9306 Acked Read 1 Sent Done Done false
31889 9307 Acked Read 2 Sent Done Done false
31890 9308 Acked Read 5 Sent Done Done false
31891 9309 Acked Read 8 Sent Done Done false
31892 9310 Acked Read 10 Sent Done Done false
31893 9311 Acked Read 6 Sent Done Done false
31894 9312 Acked Read 7 Sent Done Done false
31895 9313 Acked Read 1 Sent Done Done false
31896 9314 Acked Flush 0 Sent Done Done false
31897 9315 Acked Read 1 Sent Done Done false
31898 9316 Acked Read 3 Sent Done Done false
31899 9317 Acked Read 2 Sent Done Done false
31900 9318 Acked Read 3 Sent Done Done false
31901 9319 Acked Write 9 Sent Done Done false
31902 9320 Acked Read 9 Sent Done Done false
31903 9321 Acked Flush 0 Sent Done Done false
31904 9322 Acked Write 7 Sent Done Done false
31905 9323 Acked Read 5 Sent Done Done false
31906 9324 Acked Write 5 Sent Done Done false
31907 9325 Acked Read 5 Sent Done Done false
31908 9326 Acked Flush 0 Sent Done Done false
31909 9327 Acked Read 3 Sent Done Done false
31910 9328 Acked Write 7 Sent Done Done false
31911 9329 Acked Write 10 Sent Done Done false
31912 9330 Acked Write 4 Sent Done Done false
31913 9331 Acked Write 8 Sent Done Done false
31914 9332 Acked Read 8 Sent Done Done false
31915 9333 Acked Write 4 Sent Done Done false
31916 9334 Acked Read 8 Sent Done Done false
31917 9335 Acked Write 10 Sent Done Done false
31918 9336 Acked Read 8 Sent Done Done false
31919 9337 Acked Flush 0 Sent Done Done false
31920 9338 Acked Read 7 Sent Done Done false
31921 9339 Acked Read 3 Sent Done Done false
31922 9340 Acked Write 2 Sent Done Done false
31923 9341 Acked Write 7 Sent Done Done false
31924 9342 Acked Flush 0 Sent Done Done false
31925 9343 Acked Read 7 Sent Done Done false
31926 9344 Acked Read 1 Sent Done Done false
31927 9345 Acked Flush 0 Sent Done Done false
31928 9346 Acked Read 4 Sent Done Done false
31929 9347 Acked Write 5 Sent Done Done false
31930 9348 Acked Write 10 Sent Done Done false
31931 9349 Acked Write 6 Sent Done Done false
31932 9350 Acked Write 10 Sent Done Done false
31933 9351 Acked Read 1 Sent Done Done false
31934 9352 Acked Write 10 Sent Done Done false
31935 9353 Acked Write 8 Sent Done Done false
31936 9354 Acked Read 9 Sent Done Done false
31937 9355 Acked Write 10 Sent Done Done false
31938 9356 Acked Read 3 Sent Done Done false
31939 9357 Acked Write 5 Sent Done Done false
31940 9358 Acked Write 3 Sent Done Done false
31941 9359 Acked Write 7 Sent Done Done false
31942 9360 Acked Read 7 Sent Done Done false
31943 9361 Acked Write 7 Sent Done Done false
31944 9362 Acked Read 6 Sent Done Done false
31945 9363 Acked Write 10 Sent Done Done false
31946 9364 Acked Read 5 Sent Done Done false
31947 9365 Acked Write 4 Sent Done Done false
31948 9366 Acked Write 4 Sent Done Done false
31949 9367 Acked Write 8 Sent Done Done false
31950 9368 Acked Write 7 Sent Done Done false
31951 9369 Acked Write 5 Sent Done Done false
31952 9370 Acked Write 4 Sent Done Done false
31953 9371 Acked Flush 0 Sent Done Done false
31954 9372 Acked Read 5 Sent Done Done false
31955 9373 Acked Read 6 Sent Done Done false
31956 9374 Acked Read 2 Sent Done Done false
31957 9375 Acked Flush 0 Sent Done Done false
31958 9376 Acked Write 7 Sent Done Done false
31959 9377 Acked Write 4 Sent Done Done false
31960 9378 Acked Read 3 Sent Done Done false
31961 9379 Acked Write 9 Sent Done Done false
31962 9380 Acked Read 5 Sent Done Done false
31963 9381 Acked Flush 0 Sent Done Done false
31964 9382 Acked Write 6 Sent Done Done false
31965 9383 Acked Write 2 Sent Done Done false
31966 9384 Acked Flush 0 Sent Done Done false
31967 9385 Acked Read 1 Sent Done Done false
31968 9386 Acked Read 1 Sent Done Done false
31969 9387 Acked Read 3 Sent Done Done false
31970 9388 Acked Flush 0 Sent Done Done false
31971 9389 Acked Write 4 Sent Done Done false
31972 9390 Acked Read 4 Sent Done Done false
31973 9391 Acked Flush 0 Sent Done Done false
31974 9392 Acked Read 1 Sent Done Done false
31975 9393 Acked Write 5 Sent Done Done false
31976 9394 Acked Write 10 Sent Done Done false
31977 9395 Acked Write 7 Sent Done Done false
31978 9396 Acked Write 3 Sent Done Done false
31979 9397 Acked Read 9 Sent Done Done false
31980 9398 Acked Write 6 Sent Done Done false
31981 9399 Acked Flush 0 Sent Done Done false
31982 9400 Acked Read 9 Sent Done Done false
31983 9401 Acked Read 3 Sent Done Done false
31984 9402 Acked Flush 0 Sent Done Done false
31985 9403 Acked Flush 0 Sent Done Done false
31986 9404 Acked Write 3 Sent Done Done false
31987 9405 Acked Read 5 Sent Done Done false
31988 9406 Acked Read 4 Sent Done Done false
31989 9407 Acked Read 1 Sent Done Done false
31990 9408 Acked Read 2 Sent Done Done false
31991 9409 Acked Read 2 Sent Done Done false
31992 9410 Acked Write 1 Sent Done Done false
31993 9411 Acked Read 3 Sent Done Done false
31994 9412 Acked Read 3 Sent Done Done false
31995 9413 Acked Read 8 Sent Done Done false
31996 9414 Acked Read 8 Sent Done Done false
31997 9415 Acked Read 2 Sent Done Done false
31998 9416 Acked Write 1 Sent Done Done false
31999 9417 Acked Read 4 Sent Done Done false
32000 9418 Acked Read 4 Sent Done Done false
32001 9419 Acked Write 4 Sent Done Done false
32002 9420 Acked Write 7 Sent Done Done false
32003 9421 Acked Read 7 Sent Done Done false
32004 9422 Acked Read 6 Sent Done Done false
32005 9423 Acked Read 6 Sent Done Done false
32006 9424 Acked Read 3 Sent Done Done false
32007 9425 Acked Read 2 Sent Done Done false
32008 9426 Acked Read 7 Sent Done Done false
32009 9427 Acked Write 9 Sent Done Done false
32010 9428 Acked Write 4 Sent Done Done false
32011 9429 Acked Write 10 Sent Done Done false
32012 9430 Acked Read 2 Sent Done Done false
32013 9431 Acked Read 6 Sent Done Done false
32014 9432 Acked Write 7 Sent Done Done false
32015 9433 Acked Write 3 Sent Done Done false
32016 9434 Acked Read 2 Sent Done Done false
32017 9435 Acked Read 5 Sent Done Done false
32018 9436 Acked Write 8 Sent Done Done false
32019 9437 Acked Read 3 Sent Done Done false
32020 9438 Acked Flush 0 Sent Done Done false
32021 9439 Acked Write 7 Sent Done Done false
32022 9440 Acked Write 9 Sent Done Done false
32023 9441 Acked Write 1 Sent Done Done false
32024 9442 Acked Write 2 Sent Done Done false
32025 9443 Acked Read 7 Sent Done Done false
32026 9444 Acked Read 5 Sent Done Done false
32027 9445 Acked Read 2 Sent Done Done false
32028 9446 Acked Flush 0 Sent Done Done false
32029 9447 Acked Read 2 Sent Done Done false
32030 9448 Acked Read 10 Sent Done Done false
32031 9449 Acked Flush 0 Sent Done Done false
32032 STATES DS:0 DS:1 DS:2 TOTAL
32033 Sent 301 0 0 301
32034 Done 0 301 301 602
32035 Skipped 0 0 0 0
32036 Error 0 0 0 0
32037 Last Flush: Some(JobId(9148)) Some(JobId(9449)) Some(JobId(9449))
32038 Downstairs last five completed: 9148 9147 9146 9145 9144
32039 Upstairs last five completed: 9449 9448 9447 9446 9445
32040 CLIENT: Up:0 ds:301 act:2
320412025-02-25T20:53:32.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
320422025-02-25T20:53:33.809ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
320432025-02-25T20:53:34.725ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
320442025-02-25T20:53:35.728ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
32045 ----------------------------------------------------------------
32046 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32047 States: Offline Active Active
32048 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32049 9149 Acked Write 8 Sent Done Done false
32050 9150 Acked Write 8 Sent Done Done false
32051 9151 Acked Read 4 Sent Done Done false
32052 9152 Acked Read 4 Sent Done Done false
32053 9153 Acked Write 5 Sent Done Done false
32054 9154 Acked Write 2 Sent Done Done false
32055 9155 Acked Read 1 Sent Done Done false
32056 9156 Acked Read 6 Sent Done Done false
32057 9157 Acked Read 6 Sent Done Done false
32058 9158 Acked Read 5 Sent Done Done false
32059 9159 Acked Read 9 Sent Done Done false
32060 9160 Acked Read 8 Sent Done Done false
32061 9161 Acked Write 3 Sent Done Done false
32062 9162 Acked Flush 0 Sent Done Done false
32063 9163 Acked Write 10 Sent Done Done false
32064 9164 Acked Flush 0 Sent Done Done false
32065 9165 Acked Read 3 Sent Done Done false
32066 9166 Acked Flush 0 Sent Done Done false
32067 9167 Acked Write 2 Sent Done Done false
32068 9168 Acked Read 4 Sent Done Done false
32069 9169 Acked Read 7 Sent Done Done false
32070 9170 Acked Read 5 Sent Done Done false
32071 9171 Acked Read 1 Sent Done Done false
32072 9172 Acked Write 8 Sent Done Done false
32073 9173 Acked Read 9 Sent Done Done false
32074 9174 Acked Write 10 Sent Done Done false
32075 9175 Acked Write 7 Sent Done Done false
32076 9176 Acked Write 9 Sent Done Done false
32077 9177 Acked Read 10 Sent Done Done false
32078 9178 Acked Flush 0 Sent Done Done false
32079 9179 Acked Read 4 Sent Done Done false
32080 9180 Acked Read 8 Sent Done Done false
32081 9181 Acked Read 8 Sent Done Done false
32082 9182 Acked Read 10 Sent Done Done false
32083 9183 Acked Read 1 Sent Done Done false
32084 9184 Acked Read 6 Sent Done Done false
32085 9185 Acked Read 1 Sent Done Done false
32086 9186 Acked Read 10 Sent Done Done false
32087 9187 Acked Read 8 Sent Done Done false
32088 9188 Acked Write 4 Sent Done Done false
32089 9189 Acked Flush 0 Sent Done Done false
32090 9190 Acked Read 4 Sent Done Done false
32091 9191 Acked Write 3 Sent Done Done false
32092 9192 Acked Flush 0 Sent Done Done false
32093 9193 Acked Write 3 Sent Done Done false
32094 9194 Acked Write 10 Sent Done Done false
32095 9195 Acked Write 3 Sent Done Done false
32096 9196 Acked Flush 0 Sent Done Done false
32097 9197 Acked Read 3 Sent Done Done false
32098 9198 Acked Write 7 Sent Done Done false
32099 9199 Acked Read 5 Sent Done Done false
32100 9200 Acked Read 4 Sent Done Done false
32101 9201 Acked Read 1 Sent Done Done false
32102 9202 Acked Write 9 Sent Done Done false
32103 9203 Acked Write 4 Sent Done Done false
32104 9204 Acked Read 7 Sent Done Done false
32105 9205 Acked Write 7 Sent Done Done false
32106 9206 Acked Write 2 Sent Done Done false
32107 9207 Acked Read 10 Sent Done Done false
32108 9208 Acked Flush 0 Sent Done Done false
32109 9209 Acked Flush 0 Sent Done Done false
32110 9210 Acked Write 4 Sent Done Done false
32111 9211 Acked Read 6 Sent Done Done false
32112 9212 Acked Read 8 Sent Done Done false
32113 9213 Acked Read 9 Sent Done Done false
32114 9214 Acked Flush 0 Sent Done Done false
32115 9215 Acked Write 4 Sent Done Done false
32116 9216 Acked Flush 0 Sent Done Done false
32117 9217 Acked Read 8 Sent Done Done false
32118 9218 Acked Write 3 Sent Done Done false
32119 9219 Acked Write 3 Sent Done Done false
32120 9220 Acked Read 1 Sent Done Done false
32121 9221 Acked Write 6 Sent Done Done false
32122 9222 Acked Flush 0 Sent Done Done false
32123 9223 Acked Read 3 Sent Done Done false
32124 9224 Acked Read 6 Sent Done Done false
32125 9225 Acked Write 7 Sent Done Done false
32126 9226 Acked Write 9 Sent Done Done false
32127 9227 Acked Write 6 Sent Done Done false
32128 9228 Acked Write 2 Sent Done Done false
32129 9229 Acked Read 8 Sent Done Done false
32130 9230 Acked Write 1 Sent Done Done false
32131 9231 Acked Write 9 Sent Done Done false
32132 9232 Acked Read 1 Sent Done Done false
32133 9233 Acked Read 4 Sent Done Done false
32134 9234 Acked Write 9 Sent Done Done false
32135 9235 Acked Read 4 Sent Done Done false
32136 9236 Acked Read 10 Sent Done Done false
32137 9237 Acked Write 3 Sent Done Done false
32138 9238 Acked Write 6 Sent Done Done false
32139 9239 Acked Read 5 Sent Done Done false
32140 9240 Acked Write 10 Sent Done Done false
32141 9241 Acked Write 3 Sent Done Done false
32142 9242 Acked Write 6 Sent Done Done false
32143 9243 Acked Read 1 Sent Done Done false
32144 9244 Acked Read 1 Sent Done Done false
32145 9245 Acked Read 7 Sent Done Done false
32146 9246 Acked Write 4 Sent Done Done false
32147 9247 Acked Write 1 Sent Done Done false
32148 9248 Acked Write 8 Sent Done Done false
32149 9249 Acked Write 1 Sent Done Done false
32150 9250 Acked Write 3 Sent Done Done false
32151 9251 Acked Write 5 Sent Done Done false
32152 9252 Acked Read 4 Sent Done Done false
32153 9253 Acked Flush 0 Sent Done Done false
32154 9254 Acked Write 3 Sent Done Done false
32155 9255 Acked Read 6 Sent Done Done false
32156 9256 Acked Read 4 Sent Done Done false
32157 9257 Acked Read 8 Sent Done Done false
32158 9258 Acked Write 4 Sent Done Done false
32159 9259 Acked Write 10 Sent Done Done false
32160 9260 Acked Read 3 Sent Done Done false
32161 9261 Acked Write 4 Sent Done Done false
32162 9262 Acked Write 9 Sent Done Done false
32163 9263 Acked Write 4 Sent Done Done false
32164 9264 Acked Read 5 Sent Done Done false
32165 9265 Acked Write 2 Sent Done Done false
32166 9266 Acked Write 9 Sent Done Done false
32167 9267 Acked Read 4 Sent Done Done false
32168 9268 Acked Write 7 Sent Done Done false
32169 9269 Acked Flush 0 Sent Done Done false
32170 9270 Acked Write 7 Sent Done Done false
32171 9271 Acked Write 10 Sent Done Done false
32172 9272 Acked Read 2 Sent Done Done false
32173 9273 Acked Write 9 Sent Done Done false
32174 9274 Acked Write 2 Sent Done Done false
32175 9275 Acked Read 6 Sent Done Done false
32176 9276 Acked Write 6 Sent Done Done false
32177 9277 Acked Read 1 Sent Done Done false
32178 9278 Acked Write 2 Sent Done Done false
32179 9279 Acked Read 6 Sent Done Done false
32180 9280 Acked Write 9 Sent Done Done false
32181 9281 Acked Write 1 Sent Done Done false
32182 9282 Acked Flush 0 Sent Done Done false
32183 9283 Acked Read 3 Sent Done Done false
32184 9284 Acked Write 7 Sent Done Done false
32185 9285 Acked Read 8 Sent Done Done false
32186 9286 Acked Write 1 Sent Done Done false
32187 9287 Acked Read 8 Sent Done Done false
32188 9288 Acked Write 3 Sent Done Done false
32189 9289 Acked Write 10 Sent Done Done false
32190 9290 Acked Write 5 Sent Done Done false
32191 9291 Acked Read 10 Sent Done Done false
32192 9292 Acked Read 1 Sent Done Done false
32193 9293 Acked Write 5 Sent Done Done false
32194 9294 Acked Write 10 Sent Done Done false
32195 9295 Acked Flush 0 Sent Done Done false
32196 9296 Acked Read 8 Sent Done Done false
32197 9297 Acked Write 5 Sent Done Done false
32198 9298 Acked Read 9 Sent Done Done false
32199 9299 Acked Read 10 Sent Done Done false
32200 9300 Acked Write 2 Sent Done Done false
32201 9301 Acked Read 10 Sent Done Done false
32202 9302 Acked Read 5 Sent Done Done false
32203 9303 Acked Write 8 Sent Done Done false
32204 9304 Acked Flush 0 Sent Done Done false
32205 9305 Acked Write 10 Sent Done Done false
32206 9306 Acked Read 1 Sent Done Done false
32207 9307 Acked Read 2 Sent Done Done false
32208 9308 Acked Read 5 Sent Done Done false
32209 9309 Acked Read 8 Sent Done Done false
32210 9310 Acked Read 10 Sent Done Done false
32211 9311 Acked Read 6 Sent Done Done false
32212 9312 Acked Read 7 Sent Done Done false
32213 9313 Acked Read 1 Sent Done Done false
32214 9314 Acked Flush 0 Sent Done Done false
32215 9315 Acked Read 1 Sent Done Done false
32216 9316 Acked Read 3 Sent Done Done false
32217 9317 Acked Read 2 Sent Done Done false
32218 9318 Acked Read 3 Sent Done Done false
32219 9319 Acked Write 9 Sent Done Done false
32220 9320 Acked Read 9 Sent Done Done false
32221 9321 Acked Flush 0 Sent Done Done false
32222 9322 Acked Write 7 Sent Done Done false
32223 9323 Acked Read 5 Sent Done Done false
32224 9324 Acked Write 5 Sent Done Done false
32225 9325 Acked Read 5 Sent Done Done false
32226 9326 Acked Flush 0 Sent Done Done false
32227 9327 Acked Read 3 Sent Done Done false
32228 9328 Acked Write 7 Sent Done Done false
32229 9329 Acked Write 10 Sent Done Done false
32230 9330 Acked Write 4 Sent Done Done false
32231 9331 Acked Write 8 Sent Done Done false
32232 9332 Acked Read 8 Sent Done Done false
32233 9333 Acked Write 4 Sent Done Done false
32234 9334 Acked Read 8 Sent Done Done false
32235 9335 Acked Write 10 Sent Done Done false
32236 9336 Acked Read 8 Sent Done Done false
32237 9337 Acked Flush 0 Sent Done Done false
32238 9338 Acked Read 7 Sent Done Done false
32239 9339 Acked Read 3 Sent Done Done false
32240 9340 Acked Write 2 Sent Done Done false
32241 9341 Acked Write 7 Sent Done Done false
32242 9342 Acked Flush 0 Sent Done Done false
32243 9343 Acked Read 7 Sent Done Done false
32244 9344 Acked Read 1 Sent Done Done false
32245 9345 Acked Flush 0 Sent Done Done false
32246 9346 Acked Read 4 Sent Done Done false
32247 9347 Acked Write 5 Sent Done Done false
32248 9348 Acked Write 10 Sent Done Done false
32249 9349 Acked Write 6 Sent Done Done false
32250 9350 Acked Write 10 Sent Done Done false
32251 9351 Acked Read 1 Sent Done Done false
32252 9352 Acked Write 10 Sent Done Done false
32253 9353 Acked Write 8 Sent Done Done false
32254 9354 Acked Read 9 Sent Done Done false
32255 9355 Acked Write 10 Sent Done Done false
32256 9356 Acked Read 3 Sent Done Done false
32257 9357 Acked Write 5 Sent Done Done false
32258 9358 Acked Write 3 Sent Done Done false
32259 9359 Acked Write 7 Sent Done Done false
32260 9360 Acked Read 7 Sent Done Done false
32261 9361 Acked Write 7 Sent Done Done false
32262 9362 Acked Read 6 Sent Done Done false
32263 9363 Acked Write 10 Sent Done Done false
32264 9364 Acked Read 5 Sent Done Done false
32265 9365 Acked Write 4 Sent Done Done false
32266 9366 Acked Write 4 Sent Done Done false
32267 9367 Acked Write 8 Sent Done Done false
32268 9368 Acked Write 7 Sent Done Done false
32269 9369 Acked Write 5 Sent Done Done false
32270 9370 Acked Write 4 Sent Done Done false
32271 9371 Acked Flush 0 Sent Done Done false
32272 9372 Acked Read 5 Sent Done Done false
32273 9373 Acked Read 6 Sent Done Done false
32274 9374 Acked Read 2 Sent Done Done false
32275 9375 Acked Flush 0 Sent Done Done false
32276 9376 Acked Write 7 Sent Done Done false
32277 9377 Acked Write 4 Sent Done Done false
32278 9378 Acked Read 3 Sent Done Done false
32279 9379 Acked Write 9 Sent Done Done false
32280 9380 Acked Read 5 Sent Done Done false
32281 9381 Acked Flush 0 Sent Done Done false
32282 9382 Acked Write 6 Sent Done Done false
32283 9383 Acked Write 2 Sent Done Done false
32284 9384 Acked Flush 0 Sent Done Done false
32285 9385 Acked Read 1 Sent Done Done false
32286 9386 Acked Read 1 Sent Done Done false
32287 9387 Acked Read 3 Sent Done Done false
32288 9388 Acked Flush 0 Sent Done Done false
32289 9389 Acked Write 4 Sent Done Done false
32290 9390 Acked Read 4 Sent Done Done false
32291 9391 Acked Flush 0 Sent Done Done false
32292 9392 Acked Read 1 Sent Done Done false
32293 9393 Acked Write 5 Sent Done Done false
32294 9394 Acked Write 10 Sent Done Done false
32295 9395 Acked Write 7 Sent Done Done false
32296 9396 Acked Write 3 Sent Done Done false
32297 9397 Acked Read 9 Sent Done Done false
32298 9398 Acked Write 6 Sent Done Done false
32299 9399 Acked Flush 0 Sent Done Done false
32300 9400 Acked Read 9 Sent Done Done false
32301 9401 Acked Read 3 Sent Done Done false
32302 9402 Acked Flush 0 Sent Done Done false
32303 9403 Acked Flush 0 Sent Done Done false
32304 9404 Acked Write 3 Sent Done Done false
32305 9405 Acked Read 5 Sent Done Done false
32306 9406 Acked Read 4 Sent Done Done false
32307 9407 Acked Read 1 Sent Done Done false
32308 9408 Acked Read 2 Sent Done Done false
32309 9409 Acked Read 2 Sent Done Done false
32310 9410 Acked Write 1 Sent Done Done false
32311 9411 Acked Read 3 Sent Done Done false
32312 9412 Acked Read 3 Sent Done Done false
32313 9413 Acked Read 8 Sent Done Done false
32314 9414 Acked Read 8 Sent Done Done false
32315 9415 Acked Read 2 Sent Done Done false
32316 9416 Acked Write 1 Sent Done Done false
32317 9417 Acked Read 4 Sent Done Done false
32318 9418 Acked Read 4 Sent Done Done false
32319 9419 Acked Write 4 Sent Done Done false
32320 9420 Acked Write 7 Sent Done Done false
32321 9421 Acked Read 7 Sent Done Done false
32322 9422 Acked Read 6 Sent Done Done false
32323 9423 Acked Read 6 Sent Done Done false
32324 9424 Acked Read 3 Sent Done Done false
32325 9425 Acked Read 2 Sent Done Done false
32326 9426 Acked Read 7 Sent Done Done false
32327 9427 Acked Write 9 Sent Done Done false
32328 9428 Acked Write 4 Sent Done Done false
32329 9429 Acked Write 10 Sent Done Done false
32330 9430 Acked Read 2 Sent Done Done false
32331 9431 Acked Read 6 Sent Done Done false
32332 9432 Acked Write 7 Sent Done Done false
32333 9433 Acked Write 3 Sent Done Done false
32334 9434 Acked Read 2 Sent Done Done false
32335 9435 Acked Read 5 Sent Done Done false
32336 9436 Acked Write 8 Sent Done Done false
32337 9437 Acked Read 3 Sent Done Done false
32338 9438 Acked Flush 0 Sent Done Done false
32339 9439 Acked Write 7 Sent Done Done false
32340 9440 Acked Write 9 Sent Done Done false
32341 9441 Acked Write 1 Sent Done Done false
32342 9442 Acked Write 2 Sent Done Done false
32343 9443 Acked Read 7 Sent Done Done false
32344 9444 Acked Read 5 Sent Done Done false
32345 9445 Acked Read 2 Sent Done Done false
32346 9446 Acked Flush 0 Sent Done Done false
32347 9447 Acked Read 2 Sent Done Done false
32348 9448 Acked Read 10 Sent Done Done false
32349 9449 Acked Flush 0 Sent Done Done false
32350 STATES DS:0 DS:1 DS:2 TOTAL
32351 Sent 301 0 0 301
32352 Done 0 301 301 602
32353 Skipped 0 0 0 0
32354 Error 0 0 0 0
32355 Last Flush: Some(JobId(9148)) Some(JobId(9449)) Some(JobId(9449))
32356 Downstairs last five completed: 9148 9147 9146 9145 9144
32357 Upstairs last five completed: 9449 9448 9447 9446 9445
32358 CLIENT: Up:0 ds:301 act:2
323592025-02-25T20:53:36.729ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323602025-02-25T20:53:37.600ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323612025-02-25T20:53:37.600ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323622025-02-25T20:53:37.600ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323632025-02-25T20:53:37.601ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323642025-02-25T20:53:37.601ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323652025-02-25T20:53:38.810ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
323662025-02-25T20:53:39.733ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
32367 ----------------------------------------------------------------
32368 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32369 States: Offline Active Active
32370 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32371 9149 Acked Write 8 Sent Done Done false
32372 9150 Acked Write 8 Sent Done Done false
32373 9151 Acked Read 4 Sent Done Done false
32374 9152 Acked Read 4 Sent Done Done false
32375 9153 Acked Write 5 Sent Done Done false
32376 9154 Acked Write 2 Sent Done Done false
32377 9155 Acked Read 1 Sent Done Done false
32378 9156 Acked Read 6 Sent Done Done false
32379 9157 Acked Read 6 Sent Done Done false
32380 9158 Acked Read 5 Sent Done Done false
32381 9159 Acked Read 9 Sent Done Done false
32382 9160 Acked Read 8 Sent Done Done false
32383 9161 Acked Write 3 Sent Done Done false
32384 9162 Acked Flush 0 Sent Done Done false
32385 9163 Acked Write 10 Sent Done Done false
32386 9164 Acked Flush 0 Sent Done Done false
32387 9165 Acked Read 3 Sent Done Done false
32388 9166 Acked Flush 0 Sent Done Done false
32389 9167 Acked Write 2 Sent Done Done false
32390 9168 Acked Read 4 Sent Done Done false
32391 9169 Acked Read 7 Sent Done Done false
32392 9170 Acked Read 5 Sent Done Done false
32393 9171 Acked Read 1 Sent Done Done false
32394 9172 Acked Write 8 Sent Done Done false
32395 9173 Acked Read 9 Sent Done Done false
32396 9174 Acked Write 10 Sent Done Done false
32397 9175 Acked Write 7 Sent Done Done false
32398 9176 Acked Write 9 Sent Done Done false
32399 9177 Acked Read 10 Sent Done Done false
32400 9178 Acked Flush 0 Sent Done Done false
32401 9179 Acked Read 4 Sent Done Done false
32402 9180 Acked Read 8 Sent Done Done false
32403 9181 Acked Read 8 Sent Done Done false
32404 9182 Acked Read 10 Sent Done Done false
32405 9183 Acked Read 1 Sent Done Done false
32406 9184 Acked Read 6 Sent Done Done false
32407 9185 Acked Read 1 Sent Done Done false
32408 9186 Acked Read 10 Sent Done Done false
32409 9187 Acked Read 8 Sent Done Done false
32410 9188 Acked Write 4 Sent Done Done false
32411 9189 Acked Flush 0 Sent Done Done false
32412 9190 Acked Read 4 Sent Done Done false
32413 9191 Acked Write 3 Sent Done Done false
32414 9192 Acked Flush 0 Sent Done Done false
32415 9193 Acked Write 3 Sent Done Done false
32416 9194 Acked Write 10 Sent Done Done false
32417 9195 Acked Write 3 Sent Done Done false
32418 9196 Acked Flush 0 Sent Done Done false
32419 9197 Acked Read 3 Sent Done Done false
32420 9198 Acked Write 7 Sent Done Done false
32421 9199 Acked Read 5 Sent Done Done false
32422 9200 Acked Read 4 Sent Done Done false
32423 9201 Acked Read 1 Sent Done Done false
32424 9202 Acked Write 9 Sent Done Done false
32425 9203 Acked Write 4 Sent Done Done false
32426 9204 Acked Read 7 Sent Done Done false
32427 9205 Acked Write 7 Sent Done Done false
32428 9206 Acked Write 2 Sent Done Done false
32429 9207 Acked Read 10 Sent Done Done false
32430 9208 Acked Flush 0 Sent Done Done false
32431 9209 Acked Flush 0 Sent Done Done false
32432 9210 Acked Write 4 Sent Done Done false
32433 9211 Acked Read 6 Sent Done Done false
32434 9212 Acked Read 8 Sent Done Done false
32435 9213 Acked Read 9 Sent Done Done false
32436 9214 Acked Flush 0 Sent Done Done false
32437 9215 Acked Write 4 Sent Done Done false
32438 9216 Acked Flush 0 Sent Done Done false
32439 9217 Acked Read 8 Sent Done Done false
32440 9218 Acked Write 3 Sent Done Done false
32441 9219 Acked Write 3 Sent Done Done false
32442 9220 Acked Read 1 Sent Done Done false
32443 9221 Acked Write 6 Sent Done Done false
32444 9222 Acked Flush 0 Sent Done Done false
32445 9223 Acked Read 3 Sent Done Done false
32446 9224 Acked Read 6 Sent Done Done false
32447 9225 Acked Write 7 Sent Done Done false
32448 9226 Acked Write 9 Sent Done Done false
32449 9227 Acked Write 6 Sent Done Done false
32450 9228 Acked Write 2 Sent Done Done false
32451 9229 Acked Read 8 Sent Done Done false
32452 9230 Acked Write 1 Sent Done Done false
32453 9231 Acked Write 9 Sent Done Done false
32454 9232 Acked Read 1 Sent Done Done false
32455 9233 Acked Read 4 Sent Done Done false
32456 9234 Acked Write 9 Sent Done Done false
32457 9235 Acked Read 4 Sent Done Done false
32458 9236 Acked Read 10 Sent Done Done false
32459 9237 Acked Write 3 Sent Done Done false
32460 9238 Acked Write 6 Sent Done Done false
32461 9239 Acked Read 5 Sent Done Done false
32462 9240 Acked Write 10 Sent Done Done false
32463 9241 Acked Write 3 Sent Done Done false
32464 9242 Acked Write 6 Sent Done Done false
32465 9243 Acked Read 1 Sent Done Done false
32466 9244 Acked Read 1 Sent Done Done false
32467 9245 Acked Read 7 Sent Done Done false
32468 9246 Acked Write 4 Sent Done Done false
32469 9247 Acked Write 1 Sent Done Done false
32470 9248 Acked Write 8 Sent Done Done false
32471 9249 Acked Write 1 Sent Done Done false
32472 9250 Acked Write 3 Sent Done Done false
32473 9251 Acked Write 5 Sent Done Done false
32474 9252 Acked Read 4 Sent Done Done false
32475 9253 Acked Flush 0 Sent Done Done false
32476 9254 Acked Write 3 Sent Done Done false
32477 9255 Acked Read 6 Sent Done Done false
32478 9256 Acked Read 4 Sent Done Done false
32479 9257 Acked Read 8 Sent Done Done false
32480 9258 Acked Write 4 Sent Done Done false
32481 9259 Acked Write 10 Sent Done Done false
32482 9260 Acked Read 3 Sent Done Done false
32483 9261 Acked Write 4 Sent Done Done false
32484 9262 Acked Write 9 Sent Done Done false
32485 9263 Acked Write 4 Sent Done Done false
32486 9264 Acked Read 5 Sent Done Done false
32487 9265 Acked Write 2 Sent Done Done false
32488 9266 Acked Write 9 Sent Done Done false
32489 9267 Acked Read 4 Sent Done Done false
32490 9268 Acked Write 7 Sent Done Done false
32491 9269 Acked Flush 0 Sent Done Done false
32492 9270 Acked Write 7 Sent Done Done false
32493 9271 Acked Write 10 Sent Done Done false
32494 9272 Acked Read 2 Sent Done Done false
32495 9273 Acked Write 9 Sent Done Done false
32496 9274 Acked Write 2 Sent Done Done false
32497 9275 Acked Read 6 Sent Done Done false
32498 9276 Acked Write 6 Sent Done Done false
32499 9277 Acked Read 1 Sent Done Done false
32500 9278 Acked Write 2 Sent Done Done false
32501 9279 Acked Read 6 Sent Done Done false
32502 9280 Acked Write 9 Sent Done Done false
32503 9281 Acked Write 1 Sent Done Done false
32504 9282 Acked Flush 0 Sent Done Done false
32505 9283 Acked Read 3 Sent Done Done false
32506 9284 Acked Write 7 Sent Done Done false
32507 9285 Acked Read 8 Sent Done Done false
32508 9286 Acked Write 1 Sent Done Done false
32509 9287 Acked Read 8 Sent Done Done false
32510 9288 Acked Write 3 Sent Done Done false
32511 9289 Acked Write 10 Sent Done Done false
32512 9290 Acked Write 5 Sent Done Done false
32513 9291 Acked Read 10 Sent Done Done false
32514 9292 Acked Read 1 Sent Done Done false
32515 9293 Acked Write 5 Sent Done Done false
32516 9294 Acked Write 10 Sent Done Done false
32517 9295 Acked Flush 0 Sent Done Done false
32518 9296 Acked Read 8 Sent Done Done false
32519 9297 Acked Write 5 Sent Done Done false
32520 9298 Acked Read 9 Sent Done Done false
32521 9299 Acked Read 10 Sent Done Done false
32522 9300 Acked Write 2 Sent Done Done false
32523 9301 Acked Read 10 Sent Done Done false
32524 9302 Acked Read 5 Sent Done Done false
32525 9303 Acked Write 8 Sent Done Done false
32526 9304 Acked Flush 0 Sent Done Done false
32527 9305 Acked Write 10 Sent Done Done false
32528 9306 Acked Read 1 Sent Done Done false
32529 9307 Acked Read 2 Sent Done Done false
32530 9308 Acked Read 5 Sent Done Done false
32531 9309 Acked Read 8 Sent Done Done false
32532 9310 Acked Read 10 Sent Done Done false
32533 9311 Acked Read 6 Sent Done Done false
32534 9312 Acked Read 7 Sent Done Done false
32535 9313 Acked Read 1 Sent Done Done false
32536 9314 Acked Flush 0 Sent Done Done false
32537 9315 Acked Read 1 Sent Done Done false
32538 9316 Acked Read 3 Sent Done Done false
32539 9317 Acked Read 2 Sent Done Done false
32540 9318 Acked Read 3 Sent Done Done false
32541 9319 Acked Write 9 Sent Done Done false
32542 9320 Acked Read 9 Sent Done Done false
32543 9321 Acked Flush 0 Sent Done Done false
32544 9322 Acked Write 7 Sent Done Done false
32545 9323 Acked Read 5 Sent Done Done false
32546 9324 Acked Write 5 Sent Done Done false
32547 9325 Acked Read 5 Sent Done Done false
32548 9326 Acked Flush 0 Sent Done Done false
32549 9327 Acked Read 3 Sent Done Done false
32550 9328 Acked Write 7 Sent Done Done false
32551 9329 Acked Write 10 Sent Done Done false
32552 9330 Acked Write 4 Sent Done Done false
32553 9331 Acked Write 8 Sent Done Done false
32554 9332 Acked Read 8 Sent Done Done false
32555 9333 Acked Write 4 Sent Done Done false
32556 9334 Acked Read 8 Sent Done Done false
32557 9335 Acked Write 10 Sent Done Done false
32558 9336 Acked Read 8 Sent Done Done false
32559 9337 Acked Flush 0 Sent Done Done false
32560 9338 Acked Read 7 Sent Done Done false
32561 9339 Acked Read 3 Sent Done Done false
32562 9340 Acked Write 2 Sent Done Done false
32563 9341 Acked Write 7 Sent Done Done false
32564 9342 Acked Flush 0 Sent Done Done false
32565 9343 Acked Read 7 Sent Done Done false
32566 9344 Acked Read 1 Sent Done Done false
32567 9345 Acked Flush 0 Sent Done Done false
32568 9346 Acked Read 4 Sent Done Done false
32569 9347 Acked Write 5 Sent Done Done false
32570 9348 Acked Write 10 Sent Done Done false
32571 9349 Acked Write 6 Sent Done Done false
32572 9350 Acked Write 10 Sent Done Done false
32573 9351 Acked Read 1 Sent Done Done false
32574 9352 Acked Write 10 Sent Done Done false
32575 9353 Acked Write 8 Sent Done Done false
32576 9354 Acked Read 9 Sent Done Done false
32577 9355 Acked Write 10 Sent Done Done false
32578 9356 Acked Read 3 Sent Done Done false
32579 9357 Acked Write 5 Sent Done Done false
32580 9358 Acked Write 3 Sent Done Done false
32581 9359 Acked Write 7 Sent Done Done false
32582 9360 Acked Read 7 Sent Done Done false
32583 9361 Acked Write 7 Sent Done Done false
32584 9362 Acked Read 6 Sent Done Done false
32585 9363 Acked Write 10 Sent Done Done false
32586 9364 Acked Read 5 Sent Done Done false
32587 9365 Acked Write 4 Sent Done Done false
32588 9366 Acked Write 4 Sent Done Done false
32589 9367 Acked Write 8 Sent Done Done false
32590 9368 Acked Write 7 Sent Done Done false
32591 9369 Acked Write 5 Sent Done Done false
32592 9370 Acked Write 4 Sent Done Done false
32593 9371 Acked Flush 0 Sent Done Done false
32594 9372 Acked Read 5 Sent Done Done false
32595 9373 Acked Read 6 Sent Done Done false
32596 9374 Acked Read 2 Sent Done Done false
32597 9375 Acked Flush 0 Sent Done Done false
32598 9376 Acked Write 7 Sent Done Done false
32599 9377 Acked Write 4 Sent Done Done false
32600 9378 Acked Read 3 Sent Done Done false
32601 9379 Acked Write 9 Sent Done Done false
32602 9380 Acked Read 5 Sent Done Done false
32603 9381 Acked Flush 0 Sent Done Done false
32604 9382 Acked Write 6 Sent Done Done false
32605 9383 Acked Write 2 Sent Done Done false
32606 9384 Acked Flush 0 Sent Done Done false
32607 9385 Acked Read 1 Sent Done Done false
32608 9386 Acked Read 1 Sent Done Done false
32609 9387 Acked Read 3 Sent Done Done false
32610 9388 Acked Flush 0 Sent Done Done false
32611 9389 Acked Write 4 Sent Done Done false
32612 9390 Acked Read 4 Sent Done Done false
32613 9391 Acked Flush 0 Sent Done Done false
32614 9392 Acked Read 1 Sent Done Done false
32615 9393 Acked Write 5 Sent Done Done false
32616 9394 Acked Write 10 Sent Done Done false
32617 9395 Acked Write 7 Sent Done Done false
32618 9396 Acked Write 3 Sent Done Done false
32619 9397 Acked Read 9 Sent Done Done false
32620 9398 Acked Write 6 Sent Done Done false
32621 9399 Acked Flush 0 Sent Done Done false
32622 9400 Acked Read 9 Sent Done Done false
32623 9401 Acked Read 3 Sent Done Done false
32624 9402 Acked Flush 0 Sent Done Done false
32625 9403 Acked Flush 0 Sent Done Done false
32626 9404 Acked Write 3 Sent Done Done false
32627 9405 Acked Read 5 Sent Done Done false
32628 9406 Acked Read 4 Sent Done Done false
32629 9407 Acked Read 1 Sent Done Done false
32630 9408 Acked Read 2 Sent Done Done false
32631 9409 Acked Read 2 Sent Done Done false
32632 9410 Acked Write 1 Sent Done Done false
32633 9411 Acked Read 3 Sent Done Done false
32634 9412 Acked Read 3 Sent Done Done false
32635 9413 Acked Read 8 Sent Done Done false
32636 9414 Acked Read 8 Sent Done Done false
32637 9415 Acked Read 2 Sent Done Done false
32638 9416 Acked Write 1 Sent Done Done false
32639 9417 Acked Read 4 Sent Done Done false
32640 9418 Acked Read 4 Sent Done Done false
32641 9419 Acked Write 4 Sent Done Done false
32642 9420 Acked Write 7 Sent Done Done false
32643 9421 Acked Read 7 Sent Done Done false
32644 9422 Acked Read 6 Sent Done Done false
32645 9423 Acked Read 6 Sent Done Done false
32646 9424 Acked Read 3 Sent Done Done false
32647 9425 Acked Read 2 Sent Done Done false
32648 9426 Acked Read 7 Sent Done Done false
32649 9427 Acked Write 9 Sent Done Done false
32650 9428 Acked Write 4 Sent Done Done false
32651 9429 Acked Write 10 Sent Done Done false
32652 9430 Acked Read 2 Sent Done Done false
32653 9431 Acked Read 6 Sent Done Done false
32654 9432 Acked Write 7 Sent Done Done false
32655 9433 Acked Write 3 Sent Done Done false
32656 9434 Acked Read 2 Sent Done Done false
32657 9435 Acked Read 5 Sent Done Done false
32658 9436 Acked Write 8 Sent Done Done false
32659 9437 Acked Read 3 Sent Done Done false
32660 9438 Acked Flush 0 Sent Done Done false
32661 9439 Acked Write 7 Sent Done Done false
32662 9440 Acked Write 9 Sent Done Done false
32663 9441 Acked Write 1 Sent Done Done false
32664 9442 Acked Write 2 Sent Done Done false
32665 9443 Acked Read 7 Sent Done Done false
32666 9444 Acked Read 5 Sent Done Done false
32667 9445 Acked Read 2 Sent Done Done false
32668 9446 Acked Flush 0 Sent Done Done false
32669 9447 Acked Read 2 Sent Done Done false
32670 9448 Acked Read 10 Sent Done Done false
32671 9449 Acked Flush 0 Sent Done Done false
32672 STATES DS:0 DS:1 DS:2 TOTAL
32673 Sent 301 0 0 301
32674 Done 0 301 301 602
32675 Skipped 0 0 0 0
32676 Error 0 0 0 0
32677 Last Flush: Some(JobId(9148)) Some(JobId(9449)) Some(JobId(9449))
32678 Downstairs last five completed: 9148 9147 9146 9145 9144
32679 Upstairs last five completed: 9449 9448 9447 9446 9445
32680 CLIENT: Up:0 ds:301 act:2
326812025-02-25T20:53:40.736ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
32682 [8810] got start action
32683 [8810] I am going to restart
32684 Starting downstairs at port 8810
32685 Make output file at "/tmp/dsc/downstairs-8810.txt"
32686 [8810][0] reports Starting
32687 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
32688 [8810][0] reports Running
326892025-02-25T20:53:41.737ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
326902025-02-25T20:53:42.738ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
326912025-02-25T20:53:43.811ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
32692 ----------------------------------------------------------------
32693 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32694 States: Offline Active Active
32695 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32696 9149 Acked Write 8 Sent Done Done false
32697 9150 Acked Write 8 Sent Done Done false
32698 9151 Acked Read 4 Sent Done Done false
32699 9152 Acked Read 4 Sent Done Done false
32700 9153 Acked Write 5 Sent Done Done false
32701 9154 Acked Write 2 Sent Done Done false
32702 9155 Acked Read 1 Sent Done Done false
32703 9156 Acked Read 6 Sent Done Done false
32704 9157 Acked Read 6 Sent Done Done false
32705 9158 Acked Read 5 Sent Done Done false
32706 9159 Acked Read 9 Sent Done Done false
32707 9160 Acked Read 8 Sent Done Done false
32708 9161 Acked Write 3 Sent Done Done false
32709 9162 Acked Flush 0 Sent Done Done false
32710 9163 Acked Write 10 Sent Done Done false
32711 9164 Acked Flush 0 Sent Done Done false
32712 9165 Acked Read 3 Sent Done Done false
32713 9166 Acked Flush 0 Sent Done Done false
32714 9167 Acked Write 2 Sent Done Done false
32715 9168 Acked Read 4 Sent Done Done false
32716 9169 Acked Read 7 Sent Done Done false
32717 9170 Acked Read 5 Sent Done Done false
32718 9171 Acked Read 1 Sent Done Done false
32719 9172 Acked Write 8 Sent Done Done false
32720 9173 Acked Read 9 Sent Done Done false
32721 9174 Acked Write 10 Sent Done Done false
32722 9175 Acked Write 7 Sent Done Done false
32723 9176 Acked Write 9 Sent Done Done false
32724 9177 Acked Read 10 Sent Done Done false
32725 9178 Acked Flush 0 Sent Done Done false
32726 9179 Acked Read 4 Sent Done Done false
32727 9180 Acked Read 8 Sent Done Done false
32728 9181 Acked Read 8 Sent Done Done false
32729 9182 Acked Read 10 Sent Done Done false
32730 9183 Acked Read 1 Sent Done Done false
32731 9184 Acked Read 6 Sent Done Done false
32732 9185 Acked Read 1 Sent Done Done false
32733 9186 Acked Read 10 Sent Done Done false
32734 9187 Acked Read 8 Sent Done Done false
32735 9188 Acked Write 4 Sent Done Done false
32736 9189 Acked Flush 0 Sent Done Done false
32737 9190 Acked Read 4 Sent Done Done false
32738 9191 Acked Write 3 Sent Done Done false
32739 9192 Acked Flush 0 Sent Done Done false
32740 9193 Acked Write 3 Sent Done Done false
32741 9194 Acked Write 10 Sent Done Done false
32742 9195 Acked Write 3 Sent Done Done false
32743 9196 Acked Flush 0 Sent Done Done false
32744 9197 Acked Read 3 Sent Done Done false
32745 9198 Acked Write 7 Sent Done Done false
32746 9199 Acked Read 5 Sent Done Done false
32747 9200 Acked Read 4 Sent Done Done false
32748 9201 Acked Read 1 Sent Done Done false
32749 9202 Acked Write 9 Sent Done Done false
32750 9203 Acked Write 4 Sent Done Done false
32751 9204 Acked Read 7 Sent Done Done false
32752 9205 Acked Write 7 Sent Done Done false
32753 9206 Acked Write 2 Sent Done Done false
32754 9207 Acked Read 10 Sent Done Done false
32755 9208 Acked Flush 0 Sent Done Done false
32756 9209 Acked Flush 0 Sent Done Done false
32757 9210 Acked Write 4 Sent Done Done false
32758 9211 Acked Read 6 Sent Done Done false
32759 9212 Acked Read 8 Sent Done Done false
32760 9213 Acked Read 9 Sent Done Done false
32761 9214 Acked Flush 0 Sent Done Done false
32762 9215 Acked Write 4 Sent Done Done false
32763 9216 Acked Flush 0 Sent Done Done false
32764 9217 Acked Read 8 Sent Done Done false
32765 9218 Acked Write 3 Sent Done Done false
32766 9219 Acked Write 3 Sent Done Done false
32767 9220 Acked Read 1 Sent Done Done false
32768 9221 Acked Write 6 Sent Done Done false
32769 9222 Acked Flush 0 Sent Done Done false
32770 9223 Acked Read 3 Sent Done Done false
32771 9224 Acked Read 6 Sent Done Done false
32772 9225 Acked Write 7 Sent Done Done false
32773 9226 Acked Write 9 Sent Done Done false
32774 9227 Acked Write 6 Sent Done Done false
32775 9228 Acked Write 2 Sent Done Done false
32776 9229 Acked Read 8 Sent Done Done false
32777 9230 Acked Write 1 Sent Done Done false
32778 9231 Acked Write 9 Sent Done Done false
32779 9232 Acked Read 1 Sent Done Done false
32780 9233 Acked Read 4 Sent Done Done false
32781 9234 Acked Write 9 Sent Done Done false
32782 9235 Acked Read 4 Sent Done Done false
32783 9236 Acked Read 10 Sent Done Done false
32784 9237 Acked Write 3 Sent Done Done false
32785 9238 Acked Write 6 Sent Done Done false
32786 9239 Acked Read 5 Sent Done Done false
32787 9240 Acked Write 10 Sent Done Done false
32788 9241 Acked Write 3 Sent Done Done false
32789 9242 Acked Write 6 Sent Done Done false
32790 9243 Acked Read 1 Sent Done Done false
32791 9244 Acked Read 1 Sent Done Done false
32792 9245 Acked Read 7 Sent Done Done false
32793 9246 Acked Write 4 Sent Done Done false
32794 9247 Acked Write 1 Sent Done Done false
32795 9248 Acked Write 8 Sent Done Done false
32796 9249 Acked Write 1 Sent Done Done false
32797 9250 Acked Write 3 Sent Done Done false
32798 9251 Acked Write 5 Sent Done Done false
32799 9252 Acked Read 4 Sent Done Done false
32800 9253 Acked Flush 0 Sent Done Done false
32801 9254 Acked Write 3 Sent Done Done false
32802 9255 Acked Read 6 Sent Done Done false
32803 9256 Acked Read 4 Sent Done Done false
32804 9257 Acked Read 8 Sent Done Done false
32805 9258 Acked Write 4 Sent Done Done false
32806 9259 Acked Write 10 Sent Done Done false
32807 9260 Acked Read 3 Sent Done Done false
32808 9261 Acked Write 4 Sent Done Done false
32809 9262 Acked Write 9 Sent Done Done false
32810 9263 Acked Write 4 Sent Done Done false
32811 9264 Acked Read 5 Sent Done Done false
32812 9265 Acked Write 2 Sent Done Done false
32813 9266 Acked Write 9 Sent Done Done false
32814 9267 Acked Read 4 Sent Done Done false
32815 9268 Acked Write 7 Sent Done Done false
32816 9269 Acked Flush 0 Sent Done Done false
32817 9270 Acked Write 7 Sent Done Done false
32818 9271 Acked Write 10 Sent Done Done false
32819 9272 Acked Read 2 Sent Done Done false
32820 9273 Acked Write 9 Sent Done Done false
32821 9274 Acked Write 2 Sent Done Done false
32822 9275 Acked Read 6 Sent Done Done false
32823 9276 Acked Write 6 Sent Done Done false
32824 9277 Acked Read 1 Sent Done Done false
32825 9278 Acked Write 2 Sent Done Done false
32826 9279 Acked Read 6 Sent Done Done false
32827 9280 Acked Write 9 Sent Done Done false
32828 9281 Acked Write 1 Sent Done Done false
32829 9282 Acked Flush 0 Sent Done Done false
32830 9283 Acked Read 3 Sent Done Done false
32831 9284 Acked Write 7 Sent Done Done false
32832 9285 Acked Read 8 Sent Done Done false
32833 9286 Acked Write 1 Sent Done Done false
32834 9287 Acked Read 8 Sent Done Done false
32835 9288 Acked Write 3 Sent Done Done false
32836 9289 Acked Write 10 Sent Done Done false
32837 9290 Acked Write 5 Sent Done Done false
32838 9291 Acked Read 10 Sent Done Done false
32839 9292 Acked Read 1 Sent Done Done false
32840 9293 Acked Write 5 Sent Done Done false
32841 9294 Acked Write 10 Sent Done Done false
32842 9295 Acked Flush 0 Sent Done Done false
32843 9296 Acked Read 8 Sent Done Done false
32844 9297 Acked Write 5 Sent Done Done false
32845 9298 Acked Read 9 Sent Done Done false
32846 9299 Acked Read 10 Sent Done Done false
32847 9300 Acked Write 2 Sent Done Done false
32848 9301 Acked Read 10 Sent Done Done false
32849 9302 Acked Read 5 Sent Done Done false
32850 9303 Acked Write 8 Sent Done Done false
32851 9304 Acked Flush 0 Sent Done Done false
32852 9305 Acked Write 10 Sent Done Done false
32853 9306 Acked Read 1 Sent Done Done false
32854 9307 Acked Read 2 Sent Done Done false
32855 9308 Acked Read 5 Sent Done Done false
32856 9309 Acked Read 8 Sent Done Done false
32857 9310 Acked Read 10 Sent Done Done false
32858 9311 Acked Read 6 Sent Done Done false
32859 9312 Acked Read 7 Sent Done Done false
32860 9313 Acked Read 1 Sent Done Done false
32861 9314 Acked Flush 0 Sent Done Done false
32862 9315 Acked Read 1 Sent Done Done false
32863 9316 Acked Read 3 Sent Done Done false
32864 9317 Acked Read 2 Sent Done Done false
32865 9318 Acked Read 3 Sent Done Done false
32866 9319 Acked Write 9 Sent Done Done false
32867 9320 Acked Read 9 Sent Done Done false
32868 9321 Acked Flush 0 Sent Done Done false
32869 9322 Acked Write 7 Sent Done Done false
32870 9323 Acked Read 5 Sent Done Done false
32871 9324 Acked Write 5 Sent Done Done false
32872 9325 Acked Read 5 Sent Done Done false
32873 9326 Acked Flush 0 Sent Done Done false
32874 9327 Acked Read 3 Sent Done Done false
32875 9328 Acked Write 7 Sent Done Done false
32876 9329 Acked Write 10 Sent Done Done false
32877 9330 Acked Write 4 Sent Done Done false
32878 9331 Acked Write 8 Sent Done Done false
32879 9332 Acked Read 8 Sent Done Done false
32880 9333 Acked Write 4 Sent Done Done false
32881 9334 Acked Read 8 Sent Done Done false
32882 9335 Acked Write 10 Sent Done Done false
32883 9336 Acked Read 8 Sent Done Done false
32884 9337 Acked Flush 0 Sent Done Done false
32885 9338 Acked Read 7 Sent Done Done false
32886 9339 Acked Read 3 Sent Done Done false
32887 9340 Acked Write 2 Sent Done Done false
32888 9341 Acked Write 7 Sent Done Done false
32889 9342 Acked Flush 0 Sent Done Done false
32890 9343 Acked Read 7 Sent Done Done false
32891 9344 Acked Read 1 Sent Done Done false
32892 9345 Acked Flush 0 Sent Done Done false
32893 9346 Acked Read 4 Sent Done Done false
32894 9347 Acked Write 5 Sent Done Done false
32895 9348 Acked Write 10 Sent Done Done false
32896 9349 Acked Write 6 Sent Done Done false
32897 9350 Acked Write 10 Sent Done Done false
32898 9351 Acked Read 1 Sent Done Done false
32899 9352 Acked Write 10 Sent Done Done false
32900 9353 Acked Write 8 Sent Done Done false
32901 9354 Acked Read 9 Sent Done Done false
32902 9355 Acked Write 10 Sent Done Done false
32903 9356 Acked Read 3 Sent Done Done false
32904 9357 Acked Write 5 Sent Done Done false
32905 9358 Acked Write 3 Sent Done Done false
32906 9359 Acked Write 7 Sent Done Done false
32907 9360 Acked Read 7 Sent Done Done false
32908 9361 Acked Write 7 Sent Done Done false
32909 9362 Acked Read 6 Sent Done Done false
32910 9363 Acked Write 10 Sent Done Done false
32911 9364 Acked Read 5 Sent Done Done false
32912 9365 Acked Write 4 Sent Done Done false
32913 9366 Acked Write 4 Sent Done Done false
32914 9367 Acked Write 8 Sent Done Done false
32915 9368 Acked Write 7 Sent Done Done false
32916 9369 Acked Write 5 Sent Done Done false
32917 9370 Acked Write 4 Sent Done Done false
32918 9371 Acked Flush 0 Sent Done Done false
32919 9372 Acked Read 5 Sent Done Done false
32920 9373 Acked Read 6 Sent Done Done false
32921 9374 Acked Read 2 Sent Done Done false
32922 9375 Acked Flush 0 Sent Done Done false
32923 9376 Acked Write 7 Sent Done Done false
32924 9377 Acked Write 4 Sent Done Done false
32925 9378 Acked Read 3 Sent Done Done false
32926 9379 Acked Write 9 Sent Done Done false
32927 9380 Acked Read 5 Sent Done Done false
32928 9381 Acked Flush 0 Sent Done Done false
32929 9382 Acked Write 6 Sent Done Done false
32930 9383 Acked Write 2 Sent Done Done false
32931 9384 Acked Flush 0 Sent Done Done false
32932 9385 Acked Read 1 Sent Done Done false
32933 9386 Acked Read 1 Sent Done Done false
32934 9387 Acked Read 3 Sent Done Done false
32935 9388 Acked Flush 0 Sent Done Done false
32936 9389 Acked Write 4 Sent Done Done false
32937 9390 Acked Read 4 Sent Done Done false
32938 9391 Acked Flush 0 Sent Done Done false
32939 9392 Acked Read 1 Sent Done Done false
32940 9393 Acked Write 5 Sent Done Done false
32941 9394 Acked Write 10 Sent Done Done false
32942 9395 Acked Write 7 Sent Done Done false
32943 9396 Acked Write 3 Sent Done Done false
32944 9397 Acked Read 9 Sent Done Done false
32945 9398 Acked Write 6 Sent Done Done false
32946 9399 Acked Flush 0 Sent Done Done false
32947 9400 Acked Read 9 Sent Done Done false
32948 9401 Acked Read 3 Sent Done Done false
32949 9402 Acked Flush 0 Sent Done Done false
32950 9403 Acked Flush 0 Sent Done Done false
32951 9404 Acked Write 3 Sent Done Done false
32952 9405 Acked Read 5 Sent Done Done false
32953 9406 Acked Read 4 Sent Done Done false
32954 9407 Acked Read 1 Sent Done Done false
32955 9408 Acked Read 2 Sent Done Done false
32956 9409 Acked Read 2 Sent Done Done false
32957 9410 Acked Write 1 Sent Done Done false
32958 9411 Acked Read 3 Sent Done Done false
32959 9412 Acked Read 3 Sent Done Done false
32960 9413 Acked Read 8 Sent Done Done false
32961 9414 Acked Read 8 Sent Done Done false
32962 9415 Acked Read 2 Sent Done Done false
32963 9416 Acked Write 1 Sent Done Done false
32964 9417 Acked Read 4 Sent Done Done false
32965 9418 Acked Read 4 Sent Done Done false
32966 9419 Acked Write 4 Sent Done Done false
32967 9420 Acked Write 7 Sent Done Done false
32968 9421 Acked Read 7 Sent Done Done false
32969 9422 Acked Read 6 Sent Done Done false
32970 9423 Acked Read 6 Sent Done Done false
32971 9424 Acked Read 3 Sent Done Done false
32972 9425 Acked Read 2 Sent Done Done false
32973 9426 Acked Read 7 Sent Done Done false
32974 9427 Acked Write 9 Sent Done Done false
32975 9428 Acked Write 4 Sent Done Done false
32976 9429 Acked Write 10 Sent Done Done false
32977 9430 Acked Read 2 Sent Done Done false
32978 9431 Acked Read 6 Sent Done Done false
32979 9432 Acked Write 7 Sent Done Done false
32980 9433 Acked Write 3 Sent Done Done false
32981 9434 Acked Read 2 Sent Done Done false
32982 9435 Acked Read 5 Sent Done Done false
32983 9436 Acked Write 8 Sent Done Done false
32984 9437 Acked Read 3 Sent Done Done false
32985 9438 Acked Flush 0 Sent Done Done false
32986 9439 Acked Write 7 Sent Done Done false
32987 9440 Acked Write 9 Sent Done Done false
32988 9441 Acked Write 1 Sent Done Done false
32989 9442 Acked Write 2 Sent Done Done false
32990 9443 Acked Read 7 Sent Done Done false
32991 9444 Acked Read 5 Sent Done Done false
32992 9445 Acked Read 2 Sent Done Done false
32993 9446 Acked Flush 0 Sent Done Done false
32994 9447 Acked Read 2 Sent Done Done false
32995 9448 Acked Read 10 Sent Done Done false
32996 9449 Acked Flush 0 Sent Done Done false
32997 STATES DS:0 DS:1 DS:2 TOTAL
32998 Sent 301 0 0 301
32999 Done 0 301 301 602
33000 Skipped 0 0 0 0
33001 Error 0 0 0 0
33002 Last Flush: Some(JobId(9148)) Some(JobId(9449)) Some(JobId(9449))
33003 Downstairs last five completed: 9148 9147 9146 9145 9144
33004 Upstairs last five completed: 9449 9448 9447 9446 9445
33005 CLIENT: Up:0 ds:301 act:2
330062025-02-25T20:53:44.739ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330072025-02-25T20:53:45.741ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330082025-02-25T20:53:46.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330092025-02-25T20:53:47.602ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330102025-02-25T20:53:47.602ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38507) = io task client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330112025-02-25T20:53:47.603ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330122025-02-25T20:53:47.604ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330132025-02-25T20:53:47.604ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330142025-02-25T20:53:47.604ZINFOcrucible: send last flush ID to this DS: Some(JobId(9148)) = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330152025-02-25T20:53:47.604ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330162025-02-25T20:53:47.604ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9148)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330172025-02-25T20:53:47.604ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9148)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
33018 ----------------------------------------------------------------
33019 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33020 Downstairs last five completed: 9449 9448 9447 9446 9445
33021 Upstairs last five completed: 9449 9448 9447 9446 9445
33022 CLIENT: Up:0 ds:0 act:3
33023 Replay: All jobs finished
33024 Main task has work to do, go find it
33025 got dsc Stop(1)
33026 stop 1
33027 Feb 25 20:53:47.851 INFO request completed, latency_us: 413, response_code: 204, uri: /stop/cid/1, method: POST, req_id: bb270c15-b40c-47fd-a7ec-65518ff1b868, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:[8820] Got stop action so:false kr:false
33028 9998
33029 Feb 25 20:53:47.852 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4633bb15-9714-41c9-9b39-e8d87c84f505, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
330302025-02-25T20:53:47.852ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330312025-02-25T20:53:47.852ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330322025-02-25T20:53:47.852ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
330332025-02-25T20:53:47.852ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
33034 [8820] Exited with: ExitStatus(unix_wait_status(9))
33035 [8820][1] reports Exit
33036 Feb 25 20:53:51.855 INFO request completed, latency_us: 330, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8e135976-aeaf-416c-9c78-90c65803f081, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
33037 001/300 Write block 3223 len 32768 data: 4 4 3 3 3 3 4 3
33038 002/300 Read block 3971 len 16384
33039 003/300 Read block 2601 len 4096
33040 004/300 Read block 1882 len 28672
33041 005/300 Write block 1584 len 28672 data: 11 12 11 9 6 6 5
33042 006/300 Read block 2303 len 36864
33043 007/300 Write block 1758 len 40960 data: 3 2 4 6 6 6 6 5 4 4
33044 008/300 Write block 101 len 12288 data: 6 5 7
33045 009/300 Write block 664 len 12288 data: 9 8 10
33046 010/300 Read block 2984 len 36864
33047 011/300 Read block 538 len 8192
33048 012/300 Read block 1502 len 24576
33049 013/300 Read block 3885 len 28672
33050 014/300 Write block 4808 len 20480 data: 7 6 4 4 4
33051 015/300 Read block 2156 len 4096
33052 016/300 Read block 3001 len 8192
33053 017/300 Read block 2485 len 24576
33054 018/300 Write block 4376 len 12288 data: 6 7 8
33055 019/300 Write block 152 len 12288 data: 6 6 5
33056 020/300 Read block 4179 len 32768
33057 021/300 Read block 1822 len 4096
33058 022/300 Read block 2502 len 20480
33059 023/300 Read block 3882 len 28672
33060 024/300 Read block 2176 len 4096
33061 025/300 Write block 280 len 24576 data: 6 7 7 5 5 5
33062 026/300 Read block 1513 len 32768
33063 027/300 Read block 3644 len 16384
33064 028/300 Write block 1619 len 24576 data: 7 6 7 4 5 5
33065 029/300 Read block 428 len 36864
33066 030/300 Write block 3992 len 36864 data: 7 6 6 6 5 4 5 5 5
33067 031/300 Read block 3577 len 24576
33068 032/300 Write block 1598 len 32768 data: 7 7 8 5 5 6 6 7
33069 033/300 Write block 50 len 4096 data: 4
33070 034/300 Flush
33071 035/300 Read block 2241 len 20480
33072 036/300 Write block 4395 len 8192 data: 8 7
33073 037/300 Write block 4566 len 16384 data: 3 4 6 6
33074 038/300 Write block 482 len 32768 data: 6 6 7 8 8 5 7 8
33075 039/300 Read block 1745 len 16384
33076 040/300 Read block 545 len 20480
33077 041/300 Read block 97 len 36864
33078 042/300 Write block 3900 len 16384 data: 5 7 6 7
33079 043/300 Write block 2291 len 28672 data: 4 3 4 4 5 4 3
33080 044/300 Read block 1468 len 24576
33081 045/300 Write block 4320 len 4096 data: 3
33082 046/300 Read block 4510 len 8192
33083 047/300 Read block 440 len 32768
33084 048/300 Read block 525 len 12288
33085 049/300 Read block 233 len 20480
33086 050/300 Flush
33087 051/300 Write block 4095 len 16384 data: 7 6 4 5
33088 052/300 Read block 1833 len 20480
33089 053/300 Write block 3750 len 12288 data: 5 5 4
33090 054/300 Write block 1611 len 24576 data: 5 6 7 8 7 8
33091 055/300 Write block 2249 len 4096 data: 7
33092 056/300 Read block 3343 len 4096
33093 057/300 Read block 2401 len 32768
33094 058/300 Write block 1896 len 32768 data: 5 5 6 6 6 6 7 8
33095 059/300 Read block 3651 len 20480
33096 060/300 Write block 1907 len 24576 data: 7 5 6 6 7 7
33097 061/300 Read block 2818 len 20480
33098 062/300 Read block 594 len 28672
33099 063/300 Flush
33100 064/300 Write block 1691 len 36864 data: 3 4 4 5 5 4 4 4 5
33101 065/300 Read block 1086 len 40960
33102 066/300 Read block 1689 len 8192
33103 067/300 Write block 1208 len 40960 data: 8 9 10 10 10 8 7 5 4 5
33104 068/300 Read block 1194 len 36864
33105 069/300 Write block 563 len 16384 data: 4 4 4 5
33106 070/300 Read block 1316 len 28672
33107 071/300 Flush
33108 072/300 Read block 3973 len 32768
33109 073/300 Write block 89 len 32768 data: 6 6 8 9 9 9 8 6
33110 074/300 Flush
33111 075/300 Read block 2385 len 32768
33112 076/300 Read block 2005 len 20480
33113 077/300 Write block 212 len 32768 data: 5 6 6 8 8 6 7 8
33114 078/300 Read block 1250 len 16384
33115 079/300 Write block 3859 len 12288 data: 3 3 3
33116 080/300 Write block 1904 len 20480 data: 8 8 9 8 6
33117 081/300 Flush
33118 082/300 Write block 1836 len 12288 data: 2 3 2
33119 083/300 Write block 1960 len 36864 data: 7 6 6 6 7 7 7 7 7
33120 084/300 Flush
33121 085/300 Write block 240 len 16384 data: 6 6 5 5
33122 086/300 Write block 995 len 12288 data: 8 9 6
33123 087/300 Write block 2707 len 4096 data: 7
33124 088/300 Write block 4821 len 8192 data: 8 7
33125 089/300 Write block 468 len 12288 data: 3 3 3
33126 090/300 Write block 2301 len 4096 data: 6
33127 091/300 Flush
33128 092/300 Write block 2888 len 40960 data: 5 8 7 4 4 7 7 10 9 9
33129 093/300 Write block 2913 len 40960 data: 6 7 7 6 6 6 7 8 9 9
33130 094/300 Flush
33131 095/300 Read block 1816 len 4096
33132 096/300 Read block 4143 len 28672
33133 097/300 Flush
33134 098/300 Read block 480 len 12288
33135 099/300 Write block 539 len 28672 data: 3 3 4 3 3 3 4
33136 100/300 Write block 4900 len 16384 data: 6 5 5 6
33137 101/300 Read block 1953 len 24576
33138 102/300 Read block 3176 len 20480
33139 103/300 Read block 922 len 20480
33140 104/300 Write block 2339 len 36864 data: 10 10 11 12 12 7 8 7 6
33141 105/300 Read block 3041 len 12288
33142 106/300 Read block 4072 len 20480
33143 107/300 Flush
33144 108/300 Read block 1968 len 20480
33145 109/300 Read block 1331 len 24576
33146 110/300 Read block 2370 len 36864
33147 111/300 Write block 1675 len 20480 data: 5 5 5 5 4
33148 112/300 Write block 1502 len 36864 data: 4 4 4 6 7 6 4 4 4
33149 113/300 Flush
33150 114/300 Write block 2298 len 40960 data: 4 4 5 7 7 7 6 7 7 8
33151 115/300 Read block 2859 len 16384
33152 116/300 Flush
33153 117/300 Read block 4858 len 36864
33154 118/300 Write block 3125 len 36864 data: 8 8 7 6 7 6 6 5 5
33155 119/300 Write block 563 len 20480 data: 5 5 5 6 6
33156 120/300 Write block 676 len 8192 data: 10 7
33157 121/300 Read block 3064 len 4096
33158 122/300 Write block 1310 len 40960 data: 6 7 6 5 5 5 5 7 8 9
33159 123/300 Write block 1519 len 28672 data: 3 3 5 8 8 9 8
33160 124/300 Write block 149 len 16384 data: 7 6 6 7
33161 125/300 Write block 435 len 12288 data: 7 6 9
33162 126/300 Read block 139 len 24576
33163 127/300 Read block 1023 len 24576
33164 128/300 Read block 1606 len 20480
33165 129/300 Read block 4617 len 24576
33166 130/300 Read block 3768 len 24576
33167 131/300 Write block 2042 len 36864 data: 6 6 6 7 4 5 6 6 6
33168 132/300 Write block 4913 len 40960 data: 3 3 4 4 5 7 7 8 9 8
33169 133/300 Write block 2629 len 40960 data: 5 5 3 3 4 3 3 4 4 7
33170 134/300 Read block 4457 len 36864
33171 135/300 Write block 349 len 36864 data: 6 5 5 5 6 6 4 4 6
33172 136/300 Flush
33173 137/300 Read block 1226 len 32768
33174 138/300 Write block 3465 len 28672 data: 6 6 6 6 7 5 5
33175 139/300 Read block 3713 len 20480
33176 140/300 Read block 1226 len 36864
33177 141/300 Write block 3447 len 8192 data: 7 6
33178 142/300 Read block 3437 len 12288
33179 143/300 Read block 2627 len 24576
33180 144/300 Write block 4054 len 12288 data: 2 3 4
33181 145/300 Read block 4851 len 28672
33182 146/300 Read block 4594 len 16384
33183 147/300 Flush
33184 148/300 Read block 3755 len 16384
33185 149/300 Write block 1858 len 32768 data: 4 3 4 4 7 7 6 5
33186 150/300 Write block 1386 len 20480 data: 6 9 10 9 9
33187 151/300 Read block 1248 len 12288
33188 152/300 Read block 2689 len 24576
33189 153/300 Read block 2328 len 24576
33190 154/300 Read block 2920 len 8192
33191 155/300 Read block 1742 len 12288
33192 156/300 Write block 3042 len 8192 data: 7 7
33193 157/300 Write block 4003 len 28672 data: 6 4 4 4 4 4 4
33194 158/300 Write block 534 len 4096 data: 6
33195 159/300 Flush
33196 160/300 Read block 3934 len 20480
33197 161/300 Write block 1168 len 32768 data: 9 9 6 5 4 5 6 7
33198 162/300 Write block 714 len 4096 data: 7
33199 163/300 Flush
33200 164/300 Read block 3529 len 40960
33201 165/300 Flush
33202 166/300 Write block 4873 len 36864 data: 6 7 7 8 8 6 6 5 8
33203 167/300 Flush
33204 168/300 Flush
33205 169/300 Read block 4738 len 36864
33206 170/300 Read block 4798 len 12288
33207 171/300 Read block 2159 len 8192
33208 172/300 Read block 3215 len 36864
33209 173/300 Write block 1014 len 20480 data: 8 7 7 5 6
33210 174/300 Write block 3999 len 24576 data: 6 6 5 6 7 5
33211 175/300 Write block 2698 len 40960 data: 4 5 6 7 7 5 5 5 6 8
33212 176/300 Write block 4364 len 4096 data: 6
33213 177/300 Read block 1761 len 36864
33214 178/300 Read block 1784 len 40960
33215 179/300 Write block 3410 len 32768 data: 4 3 3 4 4 4 6 8
33216 180/300 Write block 1898 len 8192 data: 7 7
33217 181/300 Write block 2342 len 16384 data: 13 13 8 9
33218 182/300 Read block 3686 len 28672
33219 183/300 Read block 79 len 36864
33220 184/300 Write block 2468 len 8192 data: 7 9
33221 185/300 Read block 2536 len 28672
33222 186/300 Write block 4710 len 24576 data: 6 6 5 6 5 6
33223 187/300 Read block 371 len 20480
33224 188/300 Read block 1013 len 20480
33225 189/300 Read block 3070 len 16384
33226 190/300 Read block 1870 len 8192
33227 191/300 Write block 2129 len 20480 data: 5 4 5 4 6
33228 192/300 Write block 3303 len 16384 data: 11 10 9 7
33229 193/300 Write block 3586 len 8192 data: 7 7
33230 194/300 Read block 3027 len 28672
33231 195/300 Read block 2159 len 20480
33232 196/300 Write block 2597 len 28672 data: 5 5 5 5 3 3 3
33233 197/300 Write block 332 len 20480 data: 8 7 8 8 8
33234 198/300 Write block 395 len 20480 data: 4 4 6 2 2
33235 199/300 Read block 4634 len 40960
33236 200/300 Read block 2855 len 12288
33237 201/300 Flush
33238 202/300 Read block 1427 len 28672
33239 203/300 Read block 155 len 12288
33240 204/300 Read block 4820 len 40960
33241 205/300 Write block 4335 len 20480 data: 6 6 6 5 6
33242 206/300 Write block 3199 len 20480 data: 6 7 7 5 4
33243 207/300 Write block 911 len 8192 data: 3 3
33244 208/300 Write block 529 len 40960 data: 7 7 8 9 8 7 4 4 3 3
33245 209/300 Read block 184 len 20480
33246 210/300 Read block 1902 len 36864
33247 211/300 Write block 4974 len 12288 data: 8 8 7
33248 212/300 Flush
33249 213/300 Write block 1351 len 32768 data: 4 3 3 3 3 3 3 3
33250 214/300 Read block 243 len 8192
33251 215/300 Write block 529 len 16384 data: 8 8 9 10
33252 216/300 Read block 3834 len 16384
33253 217/300 Read block 1662 len 16384
33254 218/300 Read block 2847 len 16384
33255 219/300 Write block 3964 len 24576 data: 5 4 4 4 6 6
33256 220/300 Read block 1439 len 12288
33257 221/300 Read block 2239 len 20480
33258 222/300 Write block 4036 len 20480 data: 5 6 6 5 5
33259 223/300 Flush
33260 224/300 Read block 754 len 24576
33261 225/300 Write block 1813 len 24576 data: 8 6 5 3 3 3
33262 226/300 Write block 3552 len 12288 data: 4 3 4
33263 227/300 Read block 4707 len 28672
33264 228/300 Write block 213 len 12288 data: 7 7 9
33265 229/300 Read block 687 len 8192
33266 230/300 Read block 4438 len 20480
33267 231/300 Read block 1495 len 36864
33268 232/300 Write block 2171 len 20480 data: 7 6 6 5 4
33269 233/300 Read block 2847 len 24576
33270 234/300 Read block 3230 len 16384
33271 235/300 Flush
33272 236/300 Write block 680 len 28672 data: 4 4 4 4 5 5 5
33273 237/300 Write block 4712 len 8192 data: 6 7
33274 238/300 Read block 2664 len 24576
33275 239/300 Read block 1366 len 40960
33276 240/300 Read block 3002 len 12288
33277 241/300 Read block 1118 len 12288
33278 242/300 Flush
33279 243/300 Flush
33280 244/300 Read block 1907 len 12288
33281 245/300 Write block 3654 len 8192 data: 6 6
33282 246/300 Write block 1554 len 40960 data: 5 7 4 6 7 8 9 10 11 7
33283 247/300 Read block 1493 len 36864
33284 248/300 Write block 3885 len 28672 data: 9 9 7 7 7 6 8
33285 249/300 Write block 2078 len 8192 data: 5 5
33286 250/300 Read block 1265 len 24576
33287 251/300 Write block 4239 len 36864 data: 4 3 3 5 6 6 8 9 6
33288 252/300 Flush
33289 253/300 Write block 1755 len 24576 data: 3 3 3 4 3 5
33290 254/300 Read block 2894 len 12288
33291 255/300 Read block 4916 len 32768
33292 256/300 Read block 3671 len 28672
33293 257/300 Read block 654 len 32768
33294 258/300 Write block 4377 len 8192 data: 8 9
33295 259/300 Read block 1878 len 12288
33296 260/300 Write block 1236 len 28672 data: 7 8 9 7 7 5 6
33297 261/300 Write block 191 len 12288 data: 6 5 5
33298 262/300 Write block 4312 len 12288 data: 3 3 4
33299 263/300 Read block 4648 len 16384
33300 264/300 Write block 1963 len 36864 data: 7 8 8 8 8 8 6 7 5
33301 265/300 Write block 2211 len 28672 data: 4 4 4 3 5 5 6
33302 266/300 Write block 4703 len 20480 data: 5 6 5 4 5
33303 267/300 Flush
33304 268/300 Write block 4780 len 32768 data: 4 5 5 5 4 5 5 5
33305 269/300 Read block 4078 len 12288
33306 270/300 Write block 1019 len 32768 data: 7 6 7 8 8 10 10 8
33307 271/300 Read block 2566 len 28672
33308 272/300 Read block 569 len 36864
33309 273/300 Write block 1712 len 4096 data: 4
33310 274/300 Read block 2050 len 4096
33311 275/300 Write block 2495 len 4096 data: 6
33312 276/300 Read block 2235 len 12288
33313 277/300 Read block 4482 len 36864
33314 278/300 Read block 4861 len 4096
33315 279/300 Read block 997 len 24576
33316 280/300 Read block 2159 len 40960
33317 281/300 Flush
33318 282/300 Read block 465 len 32768
33319 283/300 Read block 2821 len 12288
33320 284/300 Write block 673 len 40960 data: 9 11 13 11 8 5 5 5 5 5
33321 285/300 Write block 2774 len 40960 data: 7 4 3 3 3 3 5 7 7 8
33322 286/300 Read block 1982 len 4096
33323 287/300 Write block 1185 len 36864 data: 5 6 4 4 4 5 4 6 6
33324 288/300 Read block 586 len 16384
33325 289/300 Write block 1383 len 20480 data: 6 6 8 7 10
33326 290/300 Read block 2161 len 40960
33327 291/300 Read block 4055 len 8192
33328 292/300 Read block 2459 len 24576
33329 293/300 Write block 4530 len 4096 data: 7
33330 294/300 Write block 3763 len 28672 data: 6 7 7 8 6 4 6
33331 295/300 Read block 1487 len 16384
33332 296/300 Read block 962 len 16384
33333 297/300 Read block 4588 len 32768
33334 298/300 Flush
33335 299/300 Read block 2001 len 16384
33336 300/300 Write block 1191 len 20480 data: 5 7 7 5 4
33337 301/300 Write block 1277 len 40960 data: 2 2 3 3 3 3 2 4 4 4
33338 Main task has work to do, go find it
33339 got dsc Start(1)
33340 start 1
33341 Feb 25 20:53:52.104 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8bc86253-77c3-478d-8f94-f027ab337674, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
33342 Replay: started 1, returned:Ok(())
33343 ----------------------------------------------------------------
33344 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33345 States: Active Offline Active
33346 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33347 9450 Acked Write 8 Done Sent Done false
33348 9451 Acked Read 4 Done Sent Done false
33349 9452 Acked Read 1 Done Sent Done false
33350 9453 Acked Read 7 Done Sent Done false
33351 9454 Acked Write 7 Done Sent Done false
33352 9455 Acked Read 9 Done Sent Done false
33353 9456 Acked Write 10 Done Sent Done false
33354 9457 Acked Write 3 Done Sent Done false
33355 9458 Acked Write 3 Done Sent Done false
33356 9459 Acked Read 9 Done Sent Done false
33357 9460 Acked Read 2 Done Sent Done false
33358 9461 Acked Read 6 Done Sent Done false
33359 9462 Acked Read 7 Done Sent Done false
33360 9463 Acked Write 5 Done Sent Done false
33361 9464 Acked Read 1 Done Sent Done false
33362 9465 Acked Read 2 Done Sent Done false
33363 9466 Acked Read 6 Done Sent Done false
33364 9467 Acked Write 3 Done Sent Done false
33365 9468 Acked Write 3 Done Sent Done false
33366 9469 Acked Read 8 Done Sent Done false
33367 9470 Acked Read 1 Done Sent Done false
33368 9471 Acked Read 5 Done Sent Done false
33369 9472 Acked Read 7 Done Sent Done false
33370 9473 Acked Read 1 Done Sent Done false
33371 9474 Acked Write 6 Done Sent Done false
33372 9475 Acked Read 8 Done Sent Done false
33373 9476 Acked Read 4 Done Sent Done false
33374 9477 Acked Write 6 Done Sent Done false
33375 9478 Acked Read 9 Done Sent Done false
33376 9479 Acked Write 9 Done Sent Done false
33377 9480 Acked Read 6 Done Sent Done false
33378 9481 Acked Write 8 Done Sent Done false
33379 9482 Acked Write 1 Done Sent Done false
33380 9483 Acked Flush 0 Done Sent Done false
33381 9484 Acked Read 5 Done Sent Done false
33382 9485 Acked Write 2 Done Sent Done false
33383 9486 Acked Write 4 Done Sent Done false
33384 9487 Acked Write 8 Done Sent Done false
33385 9488 Acked Read 4 Done Sent Done false
33386 9489 Acked Read 5 Done Sent Done false
33387 9490 Acked Read 9 Done Sent Done false
33388 9491 Acked Write 4 Done Sent Done false
33389 9492 Acked Write 7 Done Sent Done false
33390 9493 Acked Read 6 Done Sent Done false
33391 9494 Acked Write 1 Done Sent Done false
33392 9495 Acked Read 2 Done Sent Done false
33393 9496 Acked Read 8 Done Sent Done false
33394 9497 Acked Read 3 Done Sent Done false
33395 9498 Acked Read 5 Done Sent Done false
33396 9499 Acked Flush 0 Done Sent Done false
33397 9500 Acked Write 4 Done Sent Done false
33398 9501 Acked Read 5 Done Sent Done false
33399 9502 Acked Write 3 Done Sent Done false
33400 9503 Acked Write 6 Done Sent Done false
33401 9504 Acked Write 1 Done Sent Done false
33402 9505 Acked Read 1 Done Sent Done false
33403 9506 Acked Read 8 Done Sent Done false
33404 9507 Acked Write 8 Done Sent Done false
33405 9508 Acked Read 5 Done Sent Done false
33406 9509 Acked Write 6 Done Sent Done false
33407 9510 Acked Read 5 Done Sent Done false
33408 9511 Acked Read 7 Done Sent Done false
33409 9512 Acked Flush 0 Done Sent Done false
33410 9513 Acked Write 9 Done Sent Done false
33411 9514 Acked Read 10 Done Sent Done false
33412 9515 Acked Read 2 Done Sent Done false
33413 9516 Acked Write 10 Done Sent Done false
33414 9517 Acked Read 9 Done Sent Done false
33415 9518 Acked Write 4 Done Sent Done false
33416 9519 Acked Read 7 Done Sent Done false
33417 9520 Acked Flush 0 Done Sent Done false
33418 9521 Acked Read 8 Done Sent Done false
33419 9522 Acked Write 8 Done Sent Done false
33420 9523 Acked Flush 0 Done Sent Done false
33421 9524 Acked Read 8 Done Sent Done false
33422 9525 Acked Read 5 Done Sent Done false
33423 9526 Acked Write 8 Done Sent Done false
33424 9527 Acked Read 4 Done Sent Done false
33425 9528 Acked Write 3 Done Sent Done false
33426 9529 Acked Write 5 Done Sent Done false
33427 9530 Acked Flush 0 Done Sent Done false
33428 9531 Acked Write 3 Done Sent Done false
33429 9532 Acked Write 9 Done Sent Done false
33430 9533 Acked Flush 0 Done Sent Done false
33431 9534 Acked Write 4 Done Sent Done false
33432 9535 Acked Write 3 Done Sent Done false
33433 9536 Acked Write 1 Done Sent Done false
33434 9537 Acked Write 2 Done Sent Done false
33435 9538 Acked Write 3 Done Sent Done false
33436 9539 Acked Write 1 Done Sent Done false
33437 9540 Acked Flush 0 Done Sent Done false
33438 9541 Acked Write 10 Done Sent Done false
33439 9542 Acked Write 10 Done Sent Done false
33440 9543 Acked Flush 0 Done Sent Done false
33441 9544 Acked Read 1 Done Sent Done false
33442 9545 Acked Read 7 Done Sent Done false
33443 9546 Acked Flush 0 Done Sent Done false
33444 9547 Acked Read 3 Done Sent Done false
33445 9548 Acked Write 7 Done Sent Done false
33446 9549 Acked Write 4 Done Sent Done false
33447 9550 Acked Read 6 Done Sent Done false
33448 9551 Acked Read 5 Done Sent Done false
33449 9552 Acked Read 5 Done Sent Done false
33450 9553 Acked Write 9 Done Sent Done false
33451 9554 Acked Read 3 Done Sent Done false
33452 9555 Acked Read 5 Done Sent Done false
33453 9556 Acked Flush 0 Done Sent Done false
33454 9557 Acked Read 5 Done Sent Done false
33455 9558 Acked Read 6 Done Sent Done false
33456 9559 Acked Read 9 Done Sent Done false
33457 9560 Acked Write 5 Done Sent Done false
33458 9561 Acked Write 9 Done Sent Done false
33459 9562 Acked Flush 0 Done Sent Done false
33460 9563 Acked Write 10 Done Sent Done false
33461 9564 Acked Read 4 Done Sent Done false
33462 9565 Acked Flush 0 Done Sent Done false
33463 9566 Acked Read 9 Done Sent Done false
33464 9567 Acked Write 9 Done Sent Done false
33465 9568 Acked Write 5 Done Sent Done false
33466 9569 Acked Write 2 Done Sent Done false
33467 9570 Acked Read 1 Done Sent Done false
33468 9571 Acked Write 10 Done Sent Done false
33469 9572 Acked Write 7 Done Sent Done false
33470 9573 Acked Write 4 Done Sent Done false
33471 9574 Acked Write 3 Done Sent Done false
33472 9575 Acked Read 6 Done Sent Done false
33473 9576 Acked Read 6 Done Sent Done false
33474 9577 Acked Read 5 Done Sent Done false
33475 9578 Acked Read 6 Done Sent Done false
33476 9579 Acked Read 6 Done Sent Done false
33477 9580 Acked Write 9 Done Sent Done false
33478 9581 Acked Write 10 Done Sent Done false
33479 9582 Acked Write 10 Done Sent Done false
33480 9583 Acked Read 9 Done Sent Done false
33481 9584 Acked Write 9 Done Sent Done false
33482 9585 Acked Flush 0 Done Sent Done false
33483 9586 Acked Read 8 Done Sent Done false
33484 9587 Acked Write 7 Done Sent Done false
33485 9588 Acked Read 5 Done Sent Done false
33486 9589 Acked Read 9 Done Sent Done false
33487 9590 Acked Write 2 Done Sent Done false
33488 9591 Acked Read 3 Done Sent Done false
33489 9592 Acked Read 6 Done Sent Done false
33490 9593 Acked Write 3 Done Sent Done false
33491 9594 Acked Read 7 Done Sent Done false
33492 9595 Acked Read 4 Done Sent Done false
33493 9596 Acked Flush 0 Done Sent Done false
33494 9597 Acked Read 4 Done Sent Done false
33495 9598 Acked Write 8 Done Sent Done false
33496 9599 Acked Write 5 Done Sent Done false
33497 9600 Acked Read 3 Done Sent Done false
33498 9601 Acked Read 6 Done Sent Done false
33499 9602 Acked Read 6 Done Sent Done false
33500 9603 Acked Read 2 Done Sent Done false
33501 9604 Acked Read 3 Done Sent Done false
33502 9605 Acked Write 2 Done Sent Done false
33503 9606 Acked Write 7 Done Sent Done false
33504 9607 Acked Write 1 Done Sent Done false
33505 9608 Acked Flush 0 Done Sent Done false
33506 9609 Acked Read 5 Done Sent Done false
33507 9610 Acked Write 8 Done Sent Done false
33508 9611 Acked Write 1 Done Sent Done false
33509 9612 Acked Flush 0 Done Sent Done false
33510 9613 Acked Read 10 Done Sent Done false
33511 9614 Acked Flush 0 Done Sent Done false
33512 9615 Acked Write 9 Done Sent Done false
33513 9616 Acked Flush 0 Done Sent Done false
33514 9617 Acked Flush 0 Done Sent Done false
33515 9618 Acked Read 9 Done Sent Done false
33516 9619 Acked Read 3 Done Sent Done false
33517 9620 Acked Read 2 Done Sent Done false
33518 9621 Acked Read 9 Done Sent Done false
33519 9622 Acked Write 5 Done Sent Done false
33520 9623 Acked Write 6 Done Sent Done false
33521 9624 Acked Write 10 Done Sent Done false
33522 9625 Acked Write 1 Done Sent Done false
33523 9626 Acked Read 9 Done Sent Done false
33524 9627 Acked Read 10 Done Sent Done false
33525 9628 Acked Write 8 Done Sent Done false
33526 9629 Acked Write 2 Done Sent Done false
33527 9630 Acked Write 4 Done Sent Done false
33528 9631 Acked Read 7 Done Sent Done false
33529 9632 Acked Read 9 Done Sent Done false
33530 9633 Acked Write 2 Done Sent Done false
33531 9634 Acked Read 7 Done Sent Done false
33532 9635 Acked Write 6 Done Sent Done false
33533 9636 Acked Read 5 Done Sent Done false
33534 9637 Acked Read 5 Done Sent Done false
33535 9638 Acked Read 4 Done Sent Done false
33536 9639 Acked Read 2 Done Sent Done false
33537 9640 Acked Write 5 Done Sent Done false
33538 9641 Acked Write 4 Done Sent Done false
33539 9642 Acked Write 2 Done Sent Done false
33540 9643 Acked Read 7 Done Sent Done false
33541 9644 Acked Read 5 Done Sent Done false
33542 9645 Acked Write 7 Done Sent Done false
33543 9646 Acked Write 5 Done Sent Done false
33544 9647 Acked Write 5 Done Sent Done false
33545 9648 Acked Read 10 Done Sent Done false
33546 9649 Acked Read 3 Done Sent Done false
33547 9650 Acked Flush 0 Done Sent Done false
33548 9651 Acked Read 7 Done Sent Done false
33549 9652 Acked Read 3 Done Sent Done false
33550 9653 Acked Read 10 Done Sent Done false
33551 9654 Acked Write 5 Done Sent Done false
33552 9655 Acked Write 5 Done Sent Done false
33553 9656 Acked Write 2 Done Sent Done false
33554 9657 Acked Write 10 Done Sent Done false
33555 9658 Acked Read 5 Done Sent Done false
33556 9659 Acked Read 9 Done Sent Done false
33557 9660 Acked Write 3 Done Sent Done false
33558 9661 Acked Flush 0 Done Sent Done false
33559 9662 Acked Write 8 Done Sent Done false
33560 9663 Acked Read 2 Done Sent Done false
33561 9664 Acked Write 4 Done Sent Done false
33562 9665 Acked Read 4 Done Sent Done false
33563 9666 Acked Read 4 Done Sent Done false
33564 9667 Acked Read 4 Done Sent Done false
33565 9668 Acked Write 6 Done Sent Done false
33566 9669 Acked Read 3 Done Sent Done false
33567 9670 Acked Read 5 Done Sent Done false
33568 9671 Acked Write 5 Done Sent Done false
33569 9672 Acked Flush 0 Done Sent Done false
33570 9673 Acked Read 6 Done Sent Done false
33571 9674 Acked Write 6 Done Sent Done false
33572 9675 Acked Write 3 Done Sent Done false
33573 9676 Acked Read 7 Done Sent Done false
33574 9677 Acked Write 3 Done Sent Done false
33575 9678 Acked Read 2 Done Sent Done false
33576 9679 Acked Read 5 Done Sent Done false
33577 9680 Acked Read 9 Done Sent Done false
33578 9681 Acked Write 5 Done Sent Done false
33579 9682 Acked Read 6 Done Sent Done false
33580 9683 Acked Read 4 Done Sent Done false
33581 9684 Acked Flush 0 Done Sent Done false
33582 9685 Acked Write 7 Done Sent Done false
33583 9686 Acked Write 2 Done Sent Done false
33584 9687 Acked Read 6 Done Sent Done false
33585 9688 Acked Read 10 Done Sent Done false
33586 9689 Acked Read 3 Done Sent Done false
33587 9690 Acked Read 3 Done Sent Done false
33588 9691 Acked Flush 0 Done Sent Done false
33589 9692 Acked Flush 0 Done Sent Done false
33590 9693 Acked Read 3 Done Sent Done false
33591 9694 Acked Write 2 Done Sent Done false
33592 9695 Acked Write 10 Done Sent Done false
33593 9696 Acked Read 9 Done Sent Done false
33594 9697 Acked Write 7 Done Sent Done false
33595 9698 Acked Write 2 Done Sent Done false
33596 9699 Acked Read 6 Done Sent Done false
33597 9700 Acked Write 9 Done Sent Done false
33598 9701 Acked Flush 0 Done Sent Done false
33599 9702 Acked Write 6 Done Sent Done false
33600 9703 Acked Read 3 Done Sent Done false
33601 9704 Acked Read 8 Done Sent Done false
33602 9705 Acked Read 7 Done Sent Done false
33603 9706 Acked Read 8 Done Sent Done false
33604 9707 Acked Write 2 Done Sent Done false
33605 9708 Acked Read 3 Done Sent Done false
33606 9709 Acked Write 7 Done Sent Done false
33607 9710 Acked Write 3 Done Sent Done false
33608 9711 Acked Write 3 Done Sent Done false
33609 9712 Acked Read 4 Done Sent Done false
33610 9713 Acked Write 9 Done Sent Done false
33611 9714 Acked Write 7 Done Sent Done false
33612 9715 Acked Write 5 Done Sent Done false
33613 9716 Acked Flush 0 Done Sent Done false
33614 9717 Acked Write 8 Done Sent Done false
33615 9718 Acked Read 3 Done Sent Done false
33616 9719 Acked Write 8 Done Sent Done false
33617 9720 Acked Read 7 Done Sent Done false
33618 9721 Acked Read 9 Done Sent Done false
33619 9722 Acked Write 1 Done Sent Done false
33620 9723 Acked Read 1 Done Sent Done false
33621 9724 Acked Write 1 Done Sent Done false
33622 9725 Acked Read 3 Done Sent Done false
33623 9726 Acked Read 9 Done Sent Done false
33624 9727 Acked Read 1 Done Sent Done false
33625 9728 Acked Read 6 Done Sent Done false
33626 9729 Acked Read 10 Done Sent Done false
33627 9730 Acked Flush 0 Done Sent Done false
33628 9731 Acked Read 8 Done Sent Done false
33629 9732 Acked Read 3 Done Sent Done false
33630 9733 Acked Write 10 Done Sent Done false
33631 9734 Acked Write 10 Done Sent Done false
33632 9735 Acked Read 1 Done Sent Done false
33633 9736 Acked Write 9 Done Sent Done false
33634 9737 Acked Read 4 Done Sent Done false
33635 9738 Acked Write 5 Done Sent Done false
33636 9739 Acked Read 10 Done Sent Done false
33637 9740 Acked Read 2 Done Sent Done false
33638 9741 Acked Read 6 Done Sent Done false
33639 9742 Acked Write 1 Done Sent Done false
33640 9743 Acked Write 7 Done Sent Done false
33641 9744 Acked Read 4 Done Sent Done false
33642 9745 Acked Read 4 Done Sent Done false
33643 9746 Acked Read 8 Done Sent Done false
33644 9747 Acked Flush 0 Done Sent Done false
33645 9748 Acked Read 4 Done Sent Done false
33646 9749 Acked Write 5 Done Sent Done false
33647 9750 Acked Write 10 Done Sent Done false
33648 STATES DS:0 DS:1 DS:2 TOTAL
33649 Sent 0 301 0 301
33650 Done 301 0 301 602
33651 Skipped 0 0 0 0
33652 Error 0 0 0 0
33653 Last Flush: Some(JobId(9747)) Some(JobId(9449)) Some(JobId(9747))
33654 Downstairs last five completed: 9449 9448 9447 9446 9445
33655 Upstairs last five completed: 9750 9749 9748 9747 9746
33656 CLIENT: Up:0 ds:301 act:2
336572025-02-25T20:53:53.757ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
336582025-02-25T20:53:54.760ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
336592025-02-25T20:53:55.761ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
33660 [8820] got start action
33661 [8820] I am going to restart
33662 Starting downstairs at port 8820
33663 Make output file at "/tmp/dsc/downstairs-8820.txt"
33664 [8820][1] reports Starting
33665 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1235)
33666 [8820][1] reports Running
33667 ----------------------------------------------------------------
33668 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33669 States: Active Offline Active
33670 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33671 9450 Acked Write 8 Done Sent Done false
33672 9451 Acked Read 4 Done Sent Done false
33673 9452 Acked Read 1 Done Sent Done false
33674 9453 Acked Read 7 Done Sent Done false
33675 9454 Acked Write 7 Done Sent Done false
33676 9455 Acked Read 9 Done Sent Done false
33677 9456 Acked Write 10 Done Sent Done false
33678 9457 Acked Write 3 Done Sent Done false
33679 9458 Acked Write 3 Done Sent Done false
33680 9459 Acked Read 9 Done Sent Done false
33681 9460 Acked Read 2 Done Sent Done false
33682 9461 Acked Read 6 Done Sent Done false
33683 9462 Acked Read 7 Done Sent Done false
33684 9463 Acked Write 5 Done Sent Done false
33685 9464 Acked Read 1 Done Sent Done false
33686 9465 Acked Read 2 Done Sent Done false
33687 9466 Acked Read 6 Done Sent Done false
33688 9467 Acked Write 3 Done Sent Done false
33689 9468 Acked Write 3 Done Sent Done false
33690 9469 Acked Read 8 Done Sent Done false
33691 9470 Acked Read 1 Done Sent Done false
33692 9471 Acked Read 5 Done Sent Done false
33693 9472 Acked Read 7 Done Sent Done false
33694 9473 Acked Read 1 Done Sent Done false
33695 9474 Acked Write 6 Done Sent Done false
33696 9475 Acked Read 8 Done Sent Done false
33697 9476 Acked Read 4 Done Sent Done false
33698 9477 Acked Write 6 Done Sent Done false
33699 9478 Acked Read 9 Done Sent Done false
33700 9479 Acked Write 9 Done Sent Done false
33701 9480 Acked Read 6 Done Sent Done false
33702 9481 Acked Write 8 Done Sent Done false
33703 9482 Acked Write 1 Done Sent Done false
33704 9483 Acked Flush 0 Done Sent Done false
33705 9484 Acked Read 5 Done Sent Done false
33706 9485 Acked Write 2 Done Sent Done false
33707 9486 Acked Write 4 Done Sent Done false
33708 9487 Acked Write 8 Done Sent Done false
33709 9488 Acked Read 4 Done Sent Done false
33710 9489 Acked Read 5 Done Sent Done false
33711 9490 Acked Read 9 Done Sent Done false
33712 9491 Acked Write 4 Done Sent Done false
33713 9492 Acked Write 7 Done Sent Done false
33714 9493 Acked Read 6 Done Sent Done false
33715 9494 Acked Write 1 Done Sent Done false
33716 9495 Acked Read 2 Done Sent Done false
33717 9496 Acked Read 8 Done Sent Done false
33718 9497 Acked Read 3 Done Sent Done false
33719 9498 Acked Read 5 Done Sent Done false
33720 9499 Acked Flush 0 Done Sent Done false
33721 9500 Acked Write 4 Done Sent Done false
33722 9501 Acked Read 5 Done Sent Done false
33723 9502 Acked Write 3 Done Sent Done false
33724 9503 Acked Write 6 Done Sent Done false
33725 9504 Acked Write 1 Done Sent Done false
33726 9505 Acked Read 1 Done Sent Done false
33727 9506 Acked Read 8 Done Sent Done false
33728 9507 Acked Write 8 Done Sent Done false
33729 9508 Acked Read 5 Done Sent Done false
33730 9509 Acked Write 6 Done Sent Done false
33731 9510 Acked Read 5 Done Sent Done false
33732 9511 Acked Read 7 Done Sent Done false
33733 9512 Acked Flush 0 Done Sent Done false
33734 9513 Acked Write 9 Done Sent Done false
33735 9514 Acked Read 10 Done Sent Done false
33736 9515 Acked Read 2 Done Sent Done false
33737 9516 Acked Write 10 Done Sent Done false
33738 9517 Acked Read 9 Done Sent Done false
33739 9518 Acked Write 4 Done Sent Done false
33740 9519 Acked Read 7 Done Sent Done false
33741 9520 Acked Flush 0 Done Sent Done false
33742 9521 Acked Read 8 Done Sent Done false
33743 9522 Acked Write 8 Done Sent Done false
33744 9523 Acked Flush 0 Done Sent Done false
33745 9524 Acked Read 8 Done Sent Done false
33746 9525 Acked Read 5 Done Sent Done false
33747 9526 Acked Write 8 Done Sent Done false
33748 9527 Acked Read 4 Done Sent Done false
33749 9528 Acked Write 3 Done Sent Done false
33750 9529 Acked Write 5 Done Sent Done false
33751 9530 Acked Flush 0 Done Sent Done false
33752 9531 Acked Write 3 Done Sent Done false
33753 9532 Acked Write 9 Done Sent Done false
33754 9533 Acked Flush 0 Done Sent Done false
33755 9534 Acked Write 4 Done Sent Done false
33756 9535 Acked Write 3 Done Sent Done false
33757 9536 Acked Write 1 Done Sent Done false
33758 9537 Acked Write 2 Done Sent Done false
33759 9538 Acked Write 3 Done Sent Done false
33760 9539 Acked Write 1 Done Sent Done false
33761 9540 Acked Flush 0 Done Sent Done false
33762 9541 Acked Write 10 Done Sent Done false
33763 9542 Acked Write 10 Done Sent Done false
33764 9543 Acked Flush 0 Done Sent Done false
33765 9544 Acked Read 1 Done Sent Done false
33766 9545 Acked Read 7 Done Sent Done false
33767 9546 Acked Flush 0 Done Sent Done false
33768 9547 Acked Read 3 Done Sent Done false
33769 9548 Acked Write 7 Done Sent Done false
33770 9549 Acked Write 4 Done Sent Done false
33771 9550 Acked Read 6 Done Sent Done false
33772 9551 Acked Read 5 Done Sent Done false
33773 9552 Acked Read 5 Done Sent Done false
33774 9553 Acked Write 9 Done Sent Done false
33775 9554 Acked Read 3 Done Sent Done false
33776 9555 Acked Read 5 Done Sent Done false
33777 9556 Acked Flush 0 Done Sent Done false
33778 9557 Acked Read 5 Done Sent Done false
33779 9558 Acked Read 6 Done Sent Done false
33780 9559 Acked Read 9 Done Sent Done false
33781 9560 Acked Write 5 Done Sent Done false
33782 9561 Acked Write 9 Done Sent Done false
33783 9562 Acked Flush 0 Done Sent Done false
33784 9563 Acked Write 10 Done Sent Done false
33785 9564 Acked Read 4 Done Sent Done false
33786 9565 Acked Flush 0 Done Sent Done false
33787 9566 Acked Read 9 Done Sent Done false
33788 9567 Acked Write 9 Done Sent Done false
33789 9568 Acked Write 5 Done Sent Done false
33790 9569 Acked Write 2 Done Sent Done false
33791 9570 Acked Read 1 Done Sent Done false
33792 9571 Acked Write 10 Done Sent Done false
33793 9572 Acked Write 7 Done Sent Done false
33794 9573 Acked Write 4 Done Sent Done false
33795 9574 Acked Write 3 Done Sent Done false
33796 9575 Acked Read 6 Done Sent Done false
33797 9576 Acked Read 6 Done Sent Done false
33798 9577 Acked Read 5 Done Sent Done false
33799 9578 Acked Read 6 Done Sent Done false
33800 9579 Acked Read 6 Done Sent Done false
33801 9580 Acked Write 9 Done Sent Done false
33802 9581 Acked Write 10 Done Sent Done false
33803 9582 Acked Write 10 Done Sent Done false
33804 9583 Acked Read 9 Done Sent Done false
33805 9584 Acked Write 9 Done Sent Done false
33806 9585 Acked Flush 0 Done Sent Done false
33807 9586 Acked Read 8 Done Sent Done false
33808 9587 Acked Write 7 Done Sent Done false
33809 9588 Acked Read 5 Done Sent Done false
33810 9589 Acked Read 9 Done Sent Done false
33811 9590 Acked Write 2 Done Sent Done false
33812 9591 Acked Read 3 Done Sent Done false
33813 9592 Acked Read 6 Done Sent Done false
33814 9593 Acked Write 3 Done Sent Done false
33815 9594 Acked Read 7 Done Sent Done false
33816 9595 Acked Read 4 Done Sent Done false
33817 9596 Acked Flush 0 Done Sent Done false
33818 9597 Acked Read 4 Done Sent Done false
33819 9598 Acked Write 8 Done Sent Done false
33820 9599 Acked Write 5 Done Sent Done false
33821 9600 Acked Read 3 Done Sent Done false
33822 9601 Acked Read 6 Done Sent Done false
33823 9602 Acked Read 6 Done Sent Done false
33824 9603 Acked Read 2 Done Sent Done false
33825 9604 Acked Read 3 Done Sent Done false
33826 9605 Acked Write 2 Done Sent Done false
33827 9606 Acked Write 7 Done Sent Done false
33828 9607 Acked Write 1 Done Sent Done false
33829 9608 Acked Flush 0 Done Sent Done false
33830 9609 Acked Read 5 Done Sent Done false
33831 9610 Acked Write 8 Done Sent Done false
33832 9611 Acked Write 1 Done Sent Done false
33833 9612 Acked Flush 0 Done Sent Done false
33834 9613 Acked Read 10 Done Sent Done false
33835 9614 Acked Flush 0 Done Sent Done false
33836 9615 Acked Write 9 Done Sent Done false
33837 9616 Acked Flush 0 Done Sent Done false
33838 9617 Acked Flush 0 Done Sent Done false
33839 9618 Acked Read 9 Done Sent Done false
33840 9619 Acked Read 3 Done Sent Done false
33841 9620 Acked Read 2 Done Sent Done false
33842 9621 Acked Read 9 Done Sent Done false
33843 9622 Acked Write 5 Done Sent Done false
33844 9623 Acked Write 6 Done Sent Done false
33845 9624 Acked Write 10 Done Sent Done false
33846 9625 Acked Write 1 Done Sent Done false
33847 9626 Acked Read 9 Done Sent Done false
33848 9627 Acked Read 10 Done Sent Done false
33849 9628 Acked Write 8 Done Sent Done false
33850 9629 Acked Write 2 Done Sent Done false
33851 9630 Acked Write 4 Done Sent Done false
33852 9631 Acked Read 7 Done Sent Done false
33853 9632 Acked Read 9 Done Sent Done false
33854 9633 Acked Write 2 Done Sent Done false
33855 9634 Acked Read 7 Done Sent Done false
33856 9635 Acked Write 6 Done Sent Done false
33857 9636 Acked Read 5 Done Sent Done false
33858 9637 Acked Read 5 Done Sent Done false
33859 9638 Acked Read 4 Done Sent Done false
33860 9639 Acked Read 2 Done Sent Done false
33861 9640 Acked Write 5 Done Sent Done false
33862 9641 Acked Write 4 Done Sent Done false
33863 9642 Acked Write 2 Done Sent Done false
33864 9643 Acked Read 7 Done Sent Done false
33865 9644 Acked Read 5 Done Sent Done false
33866 9645 Acked Write 7 Done Sent Done false
33867 9646 Acked Write 5 Done Sent Done false
33868 9647 Acked Write 5 Done Sent Done false
33869 9648 Acked Read 10 Done Sent Done false
33870 9649 Acked Read 3 Done Sent Done false
33871 9650 Acked Flush 0 Done Sent Done false
33872 9651 Acked Read 7 Done Sent Done false
33873 9652 Acked Read 3 Done Sent Done false
33874 9653 Acked Read 10 Done Sent Done false
33875 9654 Acked Write 5 Done Sent Done false
33876 9655 Acked Write 5 Done Sent Done false
33877 9656 Acked Write 2 Done Sent Done false
33878 9657 Acked Write 10 Done Sent Done false
33879 9658 Acked Read 5 Done Sent Done false
33880 9659 Acked Read 9 Done Sent Done false
33881 9660 Acked Write 3 Done Sent Done false
33882 9661 Acked Flush 0 Done Sent Done false
33883 9662 Acked Write 8 Done Sent Done false
33884 9663 Acked Read 2 Done Sent Done false
33885 9664 Acked Write 4 Done Sent Done false
33886 9665 Acked Read 4 Done Sent Done false
33887 9666 Acked Read 4 Done Sent Done false
33888 9667 Acked Read 4 Done Sent Done false
33889 9668 Acked Write 6 Done Sent Done false
33890 9669 Acked Read 3 Done Sent Done false
33891 9670 Acked Read 5 Done Sent Done false
33892 9671 Acked Write 5 Done Sent Done false
33893 9672 Acked Flush 0 Done Sent Done false
33894 9673 Acked Read 6 Done Sent Done false
33895 9674 Acked Write 6 Done Sent Done false
33896 9675 Acked Write 3 Done Sent Done false
33897 9676 Acked Read 7 Done Sent Done false
33898 9677 Acked Write 3 Done Sent Done false
33899 9678 Acked Read 2 Done Sent Done false
33900 9679 Acked Read 5 Done Sent Done false
33901 9680 Acked Read 9 Done Sent Done false
33902 9681 Acked Write 5 Done Sent Done false
33903 9682 Acked Read 6 Done Sent Done false
33904 9683 Acked Read 4 Done Sent Done false
33905 9684 Acked Flush 0 Done Sent Done false
33906 9685 Acked Write 7 Done Sent Done false
33907 9686 Acked Write 2 Done Sent Done false
33908 9687 Acked Read 6 Done Sent Done false
33909 9688 Acked Read 10 Done Sent Done false
33910 9689 Acked Read 3 Done Sent Done false
33911 9690 Acked Read 3 Done Sent Done false
33912 9691 Acked Flush 0 Done Sent Done false
33913 9692 Acked Flush 0 Done Sent Done false
33914 9693 Acked Read 3 Done Sent Done false
33915 9694 Acked Write 2 Done Sent Done false
33916 9695 Acked Write 10 Done Sent Done false
33917 9696 Acked Read 9 Done Sent Done false
33918 9697 Acked Write 7 Done Sent Done false
33919 9698 Acked Write 2 Done Sent Done false
33920 9699 Acked Read 6 Done Sent Done false
33921 9700 Acked Write 9 Done Sent Done false
33922 9701 Acked Flush 0 Done Sent Done false
33923 9702 Acked Write 6 Done Sent Done false
33924 9703 Acked Read 3 Done Sent Done false
33925 9704 Acked Read 8 Done Sent Done false
33926 9705 Acked Read 7 Done Sent Done false
33927 9706 Acked Read 8 Done Sent Done false
33928 9707 Acked Write 2 Done Sent Done false
33929 9708 Acked Read 3 Done Sent Done false
33930 9709 Acked Write 7 Done Sent Done false
33931 9710 Acked Write 3 Done Sent Done false
33932 9711 Acked Write 3 Done Sent Done false
33933 9712 Acked Read 4 Done Sent Done false
33934 9713 Acked Write 9 Done Sent Done false
33935 9714 Acked Write 7 Done Sent Done false
33936 9715 Acked Write 5 Done Sent Done false
33937 9716 Acked Flush 0 Done Sent Done false
33938 9717 Acked Write 8 Done Sent Done false
33939 9718 Acked Read 3 Done Sent Done false
33940 9719 Acked Write 8 Done Sent Done false
33941 9720 Acked Read 7 Done Sent Done false
33942 9721 Acked Read 9 Done Sent Done false
33943 9722 Acked Write 1 Done Sent Done false
33944 9723 Acked Read 1 Done Sent Done false
33945 9724 Acked Write 1 Done Sent Done false
33946 9725 Acked Read 3 Done Sent Done false
33947 9726 Acked Read 9 Done Sent Done false
33948 9727 Acked Read 1 Done Sent Done false
33949 9728 Acked Read 6 Done Sent Done false
33950 9729 Acked Read 10 Done Sent Done false
33951 9730 Acked Flush 0 Done Sent Done false
33952 9731 Acked Read 8 Done Sent Done false
33953 9732 Acked Read 3 Done Sent Done false
33954 9733 Acked Write 10 Done Sent Done false
33955 9734 Acked Write 10 Done Sent Done false
33956 9735 Acked Read 1 Done Sent Done false
33957 9736 Acked Write 9 Done Sent Done false
33958 9737 Acked Read 4 Done Sent Done false
33959 9738 Acked Write 5 Done Sent Done false
33960 9739 Acked Read 10 Done Sent Done false
33961 9740 Acked Read 2 Done Sent Done false
33962 9741 Acked Read 6 Done Sent Done false
33963 9742 Acked Write 1 Done Sent Done false
33964 9743 Acked Write 7 Done Sent Done false
33965 9744 Acked Read 4 Done Sent Done false
33966 9745 Acked Read 4 Done Sent Done false
33967 9746 Acked Read 8 Done Sent Done false
33968 9747 Acked Flush 0 Done Sent Done false
33969 9748 Acked Read 4 Done Sent Done false
33970 9749 Acked Write 5 Done Sent Done false
33971 9750 Acked Write 10 Done Sent Done false
33972 STATES DS:0 DS:1 DS:2 TOTAL
33973 Sent 0 301 0 301
33974 Done 301 0 301 602
33975 Skipped 0 0 0 0
33976 Error 0 0 0 0
33977 Last Flush: Some(JobId(9747)) Some(JobId(9449)) Some(JobId(9747))
33978 Downstairs last five completed: 9449 9448 9447 9446 9445
33979 Upstairs last five completed: 9750 9749 9748 9747 9746
33980 CLIENT: Up:0 ds:301 act:2
339812025-02-25T20:53:56.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339822025-02-25T20:53:57.853ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339832025-02-25T20:53:57.853ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58434) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339842025-02-25T20:53:57.855ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339852025-02-25T20:53:57.856ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339862025-02-25T20:53:57.856ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339872025-02-25T20:53:57.856ZINFOcrucible: send last flush ID to this DS: Some(JobId(9449)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339882025-02-25T20:53:57.856ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339892025-02-25T20:53:57.856ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9449)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
339902025-02-25T20:53:57.856ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9449)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
33991 ----------------------------------------------------------------
33992 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33993 Downstairs last five completed: 9751 9750 9749 9748 9747
33994 Upstairs last five completed: 9751 9750 9749 9748 9747
33995 CLIENT: Up:0 ds:0 act:3
33996 Replay: All jobs finished
33997 Main task has work to do, go find it
33998 got dsc Stop(1)
33999 stop 1
34000 Feb 25 20:54:00.110 INFO request completed, latency_us: 380, response_code: 204[8820] Got stop action so:false kr:false
34001 , uri: /stop/cid/1, method: POST, req_id: d5f88889-51b3-4058-8c2b-1d61e74ad3e3, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
34002 Feb 25 20:54:00.110 INFO request completed, latency_us: 129, response_code: 200, uri: /state/cid/1, method: GET, req_id: c1839f49-9712-4197-9ccf-1eb980c25bc2, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
340032025-02-25T20:54:00.110ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
340042025-02-25T20:54:00.110ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
340052025-02-25T20:54:00.110ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
340062025-02-25T20:54:00.110ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34007 [8820] Exited with: ExitStatus(unix_wait_status(9))
34008 [8820][1] reports Exit
34009 Feb 25 20:54:04.114 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6178af0c-b74b-4bfb-9d89-5b14a7a333d8, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
34010 001/300 Write block 2595 len 20480 data: 6 7 6 6 6
34011 002/300 Flush
34012 003/300 Flush
34013 004/300 Read block 185 len 32768
34014 005/300 Read block 18 len 20480
34015 006/300 Flush
34016 007/300 Write block 3286 len 16384 data: 6 5 5 4
34017 008/300 Flush
34018 009/300 Read block 2186 len 20480
34019 010/300 Read block 97 len 12288
34020 011/300 Write block 3268 len 40960 data: 9 7 8 4 5 6 5 5 7 7
34021 012/300 Write block 124 len 40960 data: 7 8 8 8 8 7 7 6 7 7
34022 013/300 Read block 5 len 32768
34023 014/300 Read block 3818 len 24576
34024 015/300 Read block 4946 len 12288
34025 016/300 Write block 1675 len 24576 data: 6 6 6 6 5 5
34026 017/300 Read block 4619 len 16384
34027 018/300 Write block 4186 len 4096 data: 5
34028 019/300 Write block 2473 len 4096 data: 9
34029 020/300 Read block 284 len 8192
34030 021/300 Write block 4197 len 4096 data: 6
34031 022/300 Write block 2956 len 40960 data: 9 10 12 10 9 10 8 9 8 6
34032 023/300 Read block 3070 len 32768
34033 024/300 Read block 4400 len 36864
34034 025/300 Read block 1636 len 20480
34035 026/300 Read block 3960 len 28672
34036 027/300 Write block 4307 len 8192 data: 2 2
34037 028/300 Read block 176 len 4096
34038 029/300 Write block 3028 len 16384 data: 5 5 6 5
34039 030/300 Write block 2048 len 32768 data: 7 7 7 3 3 6 5 6
34040 031/300 Read block 2730 len 4096
34041 032/300 Read block 745 len 8192
34042 033/300 Write block 3341 len 20480 data: 5 5 6 6 6
34043 034/300 Write block 4279 len 4096 data: 9
34044 035/300 Read block 3376 len 8192
34045 036/300 Read block 1388 len 8192
34046 037/300 Read block 1870 len 8192
34047 038/300 Read block 509 len 16384
34048 039/300 Write block 4134 len 36864 data: 6 4 6 6 6 8 7 7 8
34049 040/300 Read block 904 len 32768
34050 041/300 Read block 3869 len 28672
34051 042/300 Read block 4058 len 32768
34052 043/300 Write block 2713 len 12288 data: 4 3 3
34053 044/300 Write block 3536 len 20480 data: 3 4 4 4 4
34054 045/300 Read block 3526 len 24576
34055 046/300 Write block 2224 len 24576 data: 8 8 7 8 7 7
34056 047/300 Read block 3730 len 4096
34057 048/300 Write block 223 len 12288 data: 7 6 5
34058 049/300 Write block 1230 len 16384 data: 4 4 4 5
34059 050/300 Read block 3663 len 24576
34060 051/300 Read block 2786 len 4096
34061 052/300 Read block 3215 len 40960
34062 053/300 Write block 2776 len 8192 data: 4 4
34063 054/300 Write block 4183 len 20480 data: 5 4 5 6 5
34064 055/300 Read block 97 len 16384
34065 056/300 Read block 3730 len 4096
34066 057/300 Write block 1197 len 24576 data: 4 5 6 6 6 6
34067 058/300 Flush
34068 059/300 Read block 1299 len 20480
34069 060/300 Read block 2224 len 8192
34070 061/300 Read block 3584 len 20480
34071 062/300 Read block 898 len 40960
34072 063/300 Flush
34073 064/300 Read block 159 len 28672
34074 065/300 Read block 1850 len 4096
34075 066/300 Flush
34076 067/300 Read block 2262 len 8192
34077 068/300 Read block 160 len 24576
34078 069/300 Write block 2337 len 16384 data: 8 10 11 11
34079 070/300 Flush
34080 071/300 Read block 1480 len 20480
34081 072/300 Write block 3057 len 8192 data: 5 5
34082 073/300 Write block 2334 len 24576 data: 6 7 8 9 11 12
34083 074/300 Read block 4311 len 4096
34084 075/300 Read block 3943 len 4096
34085 076/300 Write block 4035 len 20480 data: 4 6 7 7 6
34086 077/300 Read block 2945 len 32768
34087 078/300 Read block 3653 len 28672
34088 079/300 Write block 3084 len 8192 data: 7 8
34089 080/300 Read block 4894 len 28672
34090 081/300 Flush
34091 082/300 Write block 1631 len 36864 data: 7 8 6 7 7 7 6 5 6
34092 083/300 Read block 2007 len 12288
34093 084/300 Read block 2095 len 12288
34094 085/300 Flush
34095 086/300 Read block 4732 len 40960
34096 087/300 Read block 3137 len 40960
34097 088/300 Read block 1832 len 16384
34098 089/300 Write block 770 len 16384 data: 4 4 4 3
34099 090/300 Read block 4835 len 36864
34100 091/300 Write block 411 len 16384 data: 6 7 6 8
34101 092/300 Read block 342 len 8192
34102 093/300 Write block 4661 len 32768 data: 5 5 5 5 7 5 5 5
34103 094/300 Flush
34104 095/300 Read block 3462 len 12288
34105 096/300 Write block 326 len 16384 data: 4 4 5 6
34106 097/300 Read block 2136 len 4096
34107 098/300 Write block 458 len 24576 data: 5 5 4 3 3 3
34108 099/300 Write block 1193 len 28672 data: 8 6 5 4 5 6 7
34109 100/300 Write block 1681 len 32768 data: 5 5 5 4 3 3 2 2
34110 101/300 Read block 510 len 36864
34111 102/300 Write block 4819 len 24576 data: 6 7 9 8 7 7
34112 103/300 Write block 4465 len 12288 data: 5 4 4
34113 104/300 Read block 1941 len 8192
34114 105/300 Read block 493 len 32768
34115 106/300 Flush
34116 107/300 Read block 1860 len 32768
34117 108/300 Flush
34118 109/300 Read block 953 len 28672
34119 110/300 Read block 4955 len 12288
34120 111/300 Read block 3518 len 8192
34121 112/300 Read block 3939 len 4096
34122 113/300 Read block 11 len 16384
34123 114/300 Read block 137 len 20480
34124 115/300 Flush
34125 116/300 Write block 3081 len 8192 data: 6 6
34126 117/300 Write block 3410 len 28672 data: 5 4 4 5 5 5 7
34127 118/300 Read block 2874 len 36864
34128 119/300 Write block 476 len 4096 data: 3
34129 120/300 Read block 531 len 36864
34130 121/300 Read block 2187 len 20480
34131 122/300 Read block 2285 len 12288
34132 123/300 Flush
34133 124/300 Flush
34134 125/300 Write block 3696 len 28672 data: 5 6 7 7 7 8 8
34135 126/300 Read block 3612 len 4096
34136 127/300 Write block 212 len 32768 data: 6 8 8 10 9 7 8 9
34137 128/300 Flush
34138 129/300 Read block 1992 len 36864
34139 130/300 Write block 4813 len 12288 data: 3 3 5
34140 131/300 Write block 3175 len 24576 data: 7 6 7 6 7 8
34141 132/300 Write block 4637 len 40960 data: 7 6 7 7 5 5 6 5 6 6
34142 133/300 Read block 3493 len 8192
34143 134/300 Read block 4826 len 24576
34144 135/300 Write block 3763 len 8192 data: 7 8
34145 136/300 Flush
34146 137/300 Read block 2521 len 8192
34147 138/300 Read block 169 len 40960
34148 139/300 Write block 2381 len 4096 data: 5
34149 140/300 Read block 2229 len 4096
34150 141/300 Read block 4220 len 20480
34151 142/300 Write block 4562 len 12288 data: 3 3 4
34152 143/300 Read block 1574 len 12288
34153 144/300 Write block 2346 len 12288 data: 8 7 8
34154 145/300 Read block 3258 len 28672
34155 146/300 Read block 3248 len 16384
34156 147/300 Read block 162 len 36864
34157 148/300 Read block 774 len 20480
34158 149/300 Read block 4655 len 16384
34159 150/300 Read block 3428 len 40960
34160 151/300 Read block 3641 len 36864
34161 152/300 Write block 3424 len 24576 data: 7 6 4 4 4 6
34162 153/300 Read block 875 len 32768
34163 154/300 Flush
34164 155/300 Read block 3299 len 16384
34165 156/300 Read block 1019 len 16384
34166 157/300 Read block 2569 len 4096
34167 158/300 Read block 3209 len 28672
34168 159/300 Read block 622 len 4096
34169 160/300 Flush
34170 161/300 Flush
34171 162/300 Write block 3522 len 16384 data: 5 5 5 6
34172 163/300 Read block 171 len 12288
34173 164/300 Read block 4798 len 8192
34174 165/300 Read block 4744 len 4096
34175 166/300 Write block 2555 len 16384 data: 7 6 6 5
34176 167/300 Flush
34177 168/300 Write block 2615 len 32768 data: 7 5 6 8 9 11 9 8
34178 169/300 Flush
34179 170/300 Read block 4687 len 12288
34180 171/300 Write block 4083 len 32768 data: 7 7 4 2 2 2 4 5
34181 172/300 Write block 3543 len 20480 data: 5 4 4 4 4
34182 173/300 Read block 1251 len 20480
34183 174/300 Read block 1762 len 16384
34184 175/300 Read block 1054 len 4096
34185 176/300 Read block 2657 len 28672
34186 177/300 Read block 2438 len 32768
34187 178/300 Read block 320 len 4096
34188 179/300 Read block 1377 len 12288
34189 180/300 Read block 436 len 16384
34190 181/300 Write block 166 len 20480 data: 7 5 6 8 8
34191 182/300 Read block 502 len 28672
34192 183/300 Write block 2235 len 24576 data: 4 3 4 5 5 5
34193 184/300 Flush
34194 185/300 Read block 2869 len 28672
34195 186/300 Write block 4985 len 36864 data: 5 5 6 6 7 6 7 7 7
34196 187/300 Read block 1459 len 24576
34197 188/300 Write block 2440 len 36864 data: 5 5 5 6 6 8 9 9 8
34198 189/300 Flush
34199 190/300 Write block 804 len 40960 data: 6 6 7 6 4 3 3 4 5 7
34200 191/300 Write block 4402 len 40960 data: 5 5 4 4 5 5 5 4 4 4
34201 192/300 Write block 1944 len 4096 data: 6
34202 193/300 Read block 2766 len 20480
34203 194/300 Read block 3358 len 20480
34204 195/300 Read block 2263 len 32768
34205 196/300 Read block 3224 len 32768
34206 197/300 Read block 848 len 32768
34207 198/300 Read block 16 len 36864
34208 199/300 Flush
34209 200/300 Write block 3115 len 36864 data: 6 7 5 6 6 6 7 6 6
34210 201/300 Write block 4143 len 32768 data: 6 6 7 5 4 4 6 7
34211 202/300 Flush
34212 203/300 Read block 724 len 36864
34213 204/300 Write block 465 len 8192 data: 3 2
34214 205/300 Write block 1646 len 24576 data: 3 3 5 6 5 8
34215 206/300 Read block 322 len 28672
34216 207/300 Read block 2594 len 32768
34217 208/300 Read block 1343 len 24576
34218 209/300 Read block 2852 len 36864
34219 210/300 Flush
34220 211/300 Read block 6 len 24576
34221 212/300 Write block 16 len 16384 data: 6 5 6 5
34222 213/300 Read block 3747 len 28672
34223 214/300 Read block 465 len 36864
34224 215/300 Write block 2467 len 12288 data: 5 8 10
34225 216/300 Read block 3747 len 24576
34226 217/300 Write block 2534 len 12288 data: 8 6 5
34227 218/300 Read block 2881 len 8192
34228 219/300 Read block 3569 len 20480
34229 220/300 Write block 2061 len 16384 data: 4 4 6 6
34230 221/300 Write block 4941 len 20480 data: 7 8 8 7 5
34231 222/300 Write block 401 len 8192 data: 3 4
34232 223/300 Write block 1965 len 40960 data: 9 9 9 9 7 8 6 6 5 6
34233 224/300 Flush
34234 225/300 Write block 559 len 12288 data: 6 7 7
34235 226/300 Write block 3956 len 4096 data: 7
34236 227/300 Read block 2831 len 8192
34237 228/300 Read block 3961 len 16384
34238 229/300 Flush
34239 230/300 Flush
34240 231/300 Flush
34241 232/300 Flush
34242 233/300 Write block 4861 len 12288 data: 4 4 4
34243 234/300 Write block 50 len 24576 data: 5 4 5 4 4 5
34244 235/300 Write block 1488 len 40960 data: 3 4 3 4 3 4 6 6 5 3
34245 236/300 Write block 2283 len 4096 data: 10
34246 237/300 Read block 3343 len 4096
34247 238/300 Read block 4796 len 28672
34248 239/300 Read block 1825 len 8192
34249 240/300 Write block 1367 len 4096 data: 6
34250 241/300 Read block 2302 len 12288
34251 242/300 Write block 3085 len 4096 data: 9
34252 243/300 Write block 4683 len 36864 data: 4 4 5 5 6 6 7 9 9
34253 244/300 Write block 3231 len 8192 data: 3 4
34254 245/300 Read block 4120 len 16384
34255 246/300 Read block 734 len 24576
34256 247/300 Read block 2933 len 40960
34257 248/300 Write block 4244 len 36864 data: 7 9 10 7 5 6 6 6 6
34258 249/300 Read block 4693 len 4096
34259 250/300 Read block 3475 len 4096
34260 251/300 Flush
34261 252/300 Read block 4224 len 16384
34262 253/300 Read block 2613 len 32768
34263 254/300 Write block 2762 len 28672 data: 4 2 2 2 3 4 4
34264 255/300 Read block 1031 len 24576
34265 256/300 Read block 358 len 28672
34266 257/300 Read block 3968 len 32768
34267 258/300 Read block 4139 len 16384
34268 259/300 Read block 862 len 24576
34269 260/300 Read block 4280 len 36864
34270 261/300 Read block 4849 len 12288
34271 262/300 Write block 2092 len 20480 data: 5 4 3 3 3
34272 263/300 Read block 2478 len 32768
34273 264/300 Write block 3693 len 36864 data: 5 3 3 6 7 8 8 8 9
34274 265/300 Read block 516 len 32768
34275 266/300 Read block 3517 len 36864
34276 267/300 Write block 4684 len 8192 data: 5 6
34277 268/300 Write block 4608 len 12288 data: 5 6 5
34278 269/300 Write block 2842 len 16384 data: 5 7 8 10
34279 270/300 Write block 2990 len 8192 data: 10 9
34280 271/300 Write block 205 len 24576 data: 5 6 4 5 6 5
34281 272/300 Flush
34282 273/300 Write block 1094 len 8192 data: 8 10
34283 274/300 Write block 1352 len 24576 data: 4 4 4 4 4 4
34284 275/300 Read block 1780 len 12288
34285 276/300 Write block 1781 len 40960 data: 5 6 5 7 7 9 8 7 8 9
34286 277/300 Read block 1409 len 40960
34287 278/300 Read block 3636 len 16384
34288 279/300 Read block 2809 len 8192
34289 280/300 Read block 3878 len 12288
34290 281/300 Write block 957 len 20480 data: 8 10 9 11 8
34291 282/300 Read block 1903 len 20480
34292 283/300 Write block 2452 len 16384 data: 9 10 8 9
34293 284/300 Read block 3124 len 20480
34294 285/300 Read block 4389 len 12288
34295 286/300 Read block 4495 len 24576
34296 287/300 Read block 3032 len 4096
34297 288/300 Write block 2265 len 12288 data: 5 4 3
34298 289/300 Write block 4801 len 28672 data: 4 3 3 4 5 5 6
34299 290/300 Read block 3756 len 20480
34300 291/300 Read block 3600 len 16384
34301 292/300 Write block 894 len 12288 data: 4 5 6
34302 293/300 Write block 187 len 36864 data: 6 6 5 6 7 6 6 6 6
34303 294/300 Flush
34304 295/300 Write block 107 len 20480 data: 5 6 7 7 6
34305 296/300 Read block 561 len 20480
34306 297/300 Flush
34307 298/300 Read block 80 len 36864
34308 299/300 Flush
34309 300/300 Flush
34310 301/300 Read block 887 len 4096
34311 Main task has work to do, go find it
34312 got dsc Start(1)
34313 start 1
34314 Feb 25 20:54:04.357 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/1, method: POST, req_id: ff03a26c-73cb-4cd6-a468-563cc725a7ed, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
34315 Replay: started 1, returned:Ok(())
34316 ----------------------------------------------------------------
34317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34318 States: Active Offline Active
34319 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34320 9752 Acked Write 5 Done Sent Done false
34321 9753 Acked Flush 0 Done Sent Done false
34322 9754 Acked Flush 0 Done Sent Done false
34323 9755 Acked Read 8 Done Sent Done false
34324 9756 Acked Read 5 Done Sent Done false
34325 9757 Acked Flush 0 Done Sent Done false
34326 9758 Acked Write 4 Done Sent Done false
34327 9759 Acked Flush 0 Done Sent Done false
34328 9760 Acked Read 5 Done Sent Done false
34329 9761 Acked Read 3 Done Sent Done false
34330 9762 Acked Write 10 Done Sent Done false
34331 9763 Acked Write 10 Done Sent Done false
34332 9764 Acked Read 8 Done Sent Done false
34333 9765 Acked Read 6 Done Sent Done false
34334 9766 Acked Read 3 Done Sent Done false
34335 9767 Acked Write 6 Done Sent Done false
34336 9768 Acked Read 4 Done Sent Done false
34337 9769 Acked Write 1 Done Sent Done false
34338 9770 Acked Write 1 Done Sent Done false
34339 9771 Acked Read 2 Done Sent Done false
34340 9772 Acked Write 1 Done Sent Done false
34341 9773 Acked Write 10 Done Sent Done false
34342 9774 Acked Read 8 Done Sent Done false
34343 9775 Acked Read 9 Done Sent Done false
34344 9776 Acked Read 5 Done Sent Done false
34345 9777 Acked Read 7 Done Sent Done false
34346 9778 Acked Write 2 Done Sent Done false
34347 9779 Acked Read 1 Done Sent Done false
34348 9780 Acked Write 4 Done Sent Done false
34349 9781 Acked Write 8 Done Sent Done false
34350 9782 Acked Read 1 Done Sent Done false
34351 9783 Acked Read 2 Done Sent Done false
34352 9784 Acked Write 5 Done Sent Done false
34353 9785 Acked Write 1 Done Sent Done false
34354 9786 Acked Read 2 Done Sent Done false
34355 9787 Acked Read 2 Done Sent Done false
34356 9788 Acked Read 2 Done Sent Done false
34357 9789 Acked Read 4 Done Sent Done false
34358 9790 Acked Write 9 Done Sent Done false
34359 9791 Acked Read 8 Done Sent Done false
34360 9792 Acked Read 7 Done Sent Done false
34361 9793 Acked Read 8 Done Sent Done false
34362 9794 Acked Write 3 Done Sent Done false
34363 9795 Acked Write 5 Done Sent Done false
34364 9796 Acked Read 6 Done Sent Done false
34365 9797 Acked Write 6 Done Sent Done false
34366 9798 Acked Read 1 Done Sent Done false
34367 9799 Acked Write 3 Done Sent Done false
34368 9800 Acked Write 4 Done Sent Done false
34369 9801 Acked Read 6 Done Sent Done false
34370 9802 Acked Read 1 Done Sent Done false
34371 9803 Acked Read 10 Done Sent Done false
34372 9804 Acked Write 2 Done Sent Done false
34373 9805 Acked Write 5 Done Sent Done false
34374 9806 Acked Read 4 Done Sent Done false
34375 9807 Acked Read 1 Done Sent Done false
34376 9808 Acked Write 6 Done Sent Done false
34377 9809 Acked Flush 0 Done Sent Done false
34378 9810 Acked Read 5 Done Sent Done false
34379 9811 Acked Read 2 Done Sent Done false
34380 9812 Acked Read 5 Done Sent Done false
34381 9813 Acked Read 10 Done Sent Done false
34382 9814 Acked Flush 0 Done Sent Done false
34383 9815 Acked Read 7 Done Sent Done false
34384 9816 Acked Read 1 Done Sent Done false
34385 9817 Acked Flush 0 Done Sent Done false
34386 9818 Acked Read 2 Done Sent Done false
34387 9819 Acked Read 6 Done Sent Done false
34388 9820 Acked Write 4 Done Sent Done false
34389 9821 Acked Flush 0 Done Sent Done false
34390 9822 Acked Read 5 Done Sent Done false
34391 9823 Acked Write 2 Done Sent Done false
34392 9824 Acked Write 6 Done Sent Done false
34393 9825 Acked Read 1 Done Sent Done false
34394 9826 Acked Read 1 Done Sent Done false
34395 9827 Acked Write 5 Done Sent Done false
34396 9828 Acked Read 8 Done Sent Done false
34397 9829 Acked Read 7 Done Sent Done false
34398 9830 Acked Write 2 Done Sent Done false
34399 9831 Acked Read 7 Done Sent Done false
34400 9832 Acked Flush 0 Done Sent Done false
34401 9833 Acked Write 9 Done Sent Done false
34402 9834 Acked Read 3 Done Sent Done false
34403 9835 Acked Read 3 Done Sent Done false
34404 9836 Acked Flush 0 Done Sent Done false
34405 9837 Acked Read 10 Done Sent Done false
34406 9838 Acked Read 10 Done Sent Done false
34407 9839 Acked Read 4 Done Sent Done false
34408 9840 Acked Write 4 Done Sent Done false
34409 9841 Acked Read 9 Done Sent Done false
34410 9842 Acked Write 4 Done Sent Done false
34411 9843 Acked Read 2 Done Sent Done false
34412 9844 Acked Write 8 Done Sent Done false
34413 9845 Acked Flush 0 Done Sent Done false
34414 9846 Acked Read 3 Done Sent Done false
34415 9847 Acked Write 4 Done Sent Done false
34416 9848 Acked Read 1 Done Sent Done false
34417 9849 Acked Write 6 Done Sent Done false
34418 9850 Acked Write 7 Done Sent Done false
34419 9851 Acked Write 8 Done Sent Done false
34420 9852 Acked Read 9 Done Sent Done false
34421 9853 Acked Write 6 Done Sent Done false
34422 9854 Acked Write 3 Done Sent Done false
34423 9855 Acked Read 2 Done Sent Done false
34424 9856 Acked Read 8 Done Sent Done false
34425 9857 Acked Flush 0 Done Sent Done false
34426 9858 Acked Read 8 Done Sent Done false
34427 9859 Acked Flush 0 Done Sent Done false
34428 9860 Acked Read 7 Done Sent Done false
34429 9861 Acked Read 3 Done Sent Done false
34430 9862 Acked Read 2 Done Sent Done false
34431 9863 Acked Read 1 Done Sent Done false
34432 9864 Acked Read 4 Done Sent Done false
34433 9865 Acked Read 5 Done Sent Done false
34434 9866 Acked Flush 0 Done Sent Done false
34435 9867 Acked Write 2 Done Sent Done false
34436 9868 Acked Write 7 Done Sent Done false
34437 9869 Acked Read 9 Done Sent Done false
34438 9870 Acked Write 1 Done Sent Done false
34439 9871 Acked Read 9 Done Sent Done false
34440 9872 Acked Read 5 Done Sent Done false
34441 9873 Acked Read 3 Done Sent Done false
34442 9874 Acked Flush 0 Done Sent Done false
34443 9875 Acked Flush 0 Done Sent Done false
34444 9876 Acked Write 7 Done Sent Done false
34445 9877 Acked Read 1 Done Sent Done false
34446 9878 Acked Write 8 Done Sent Done false
34447 9879 Acked Flush 0 Done Sent Done false
34448 9880 Acked Read 9 Done Sent Done false
34449 9881 Acked Write 3 Done Sent Done false
34450 9882 Acked Write 6 Done Sent Done false
34451 9883 Acked Write 10 Done Sent Done false
34452 9884 Acked Read 2 Done Sent Done false
34453 9885 Acked Read 6 Done Sent Done false
34454 9886 Acked Write 2 Done Sent Done false
34455 9887 Acked Flush 0 Done Sent Done false
34456 9888 Acked Read 2 Done Sent Done false
34457 9889 Acked Read 10 Done Sent Done false
34458 9890 Acked Write 1 Done Sent Done false
34459 9891 Acked Read 1 Done Sent Done false
34460 9892 Acked Read 5 Done Sent Done false
34461 9893 Acked Write 3 Done Sent Done false
34462 9894 Acked Read 3 Done Sent Done false
34463 9895 Acked Write 3 Done Sent Done false
34464 9896 Acked Read 7 Done Sent Done false
34465 9897 Acked Read 4 Done Sent Done false
34466 9898 Acked Read 9 Done Sent Done false
34467 9899 Acked Read 5 Done Sent Done false
34468 9900 Acked Read 4 Done Sent Done false
34469 9901 Acked Read 10 Done Sent Done false
34470 9902 Acked Read 9 Done Sent Done false
34471 9903 Acked Write 6 Done Sent Done false
34472 9904 Acked Read 8 Done Sent Done false
34473 9905 Acked Flush 0 Done Sent Done false
34474 9906 Acked Read 4 Done Sent Done false
34475 9907 Acked Read 4 Done Sent Done false
34476 9908 Acked Read 1 Done Sent Done false
34477 9909 Acked Read 7 Done Sent Done false
34478 9910 Acked Read 1 Done Sent Done false
34479 9911 Acked Flush 0 Done Sent Done false
34480 9912 Acked Flush 0 Done Sent Done false
34481 9913 Acked Write 4 Done Sent Done false
34482 9914 Acked Read 3 Done Sent Done false
34483 9915 Acked Read 2 Done Sent Done false
34484 9916 Acked Read 1 Done Sent Done false
34485 9917 Acked Write 4 Done Sent Done false
34486 9918 Acked Flush 0 Done Sent Done false
34487 9919 Acked Write 8 Done Sent Done false
34488 9920 Acked Flush 0 Done Sent Done false
34489 9921 Acked Read 3 Done Sent Done false
34490 9922 Acked Write 8 Done Sent Done false
34491 9923 Acked Write 5 Done Sent Done false
34492 9924 Acked Read 5 Done Sent Done false
34493 9925 Acked Read 4 Done Sent Done false
34494 9926 Acked Read 1 Done Sent Done false
34495 9927 Acked Read 7 Done Sent Done false
34496 9928 Acked Read 8 Done Sent Done false
34497 9929 Acked Read 1 Done Sent Done false
34498 9930 Acked Read 3 Done Sent Done false
34499 9931 Acked Read 4 Done Sent Done false
34500 9932 Acked Write 5 Done Sent Done false
34501 9933 Acked Read 7 Done Sent Done false
34502 9934 Acked Write 6 Done Sent Done false
34503 9935 Acked Flush 0 Done Sent Done false
34504 9936 Acked Read 7 Done Sent Done false
34505 9937 Acked Write 9 Done Sent Done false
34506 9938 Acked Read 6 Done Sent Done false
34507 9939 Acked Write 9 Done Sent Done false
34508 9940 Acked Flush 0 Done Sent Done false
34509 9941 Acked Write 10 Done Sent Done false
34510 9942 Acked Write 10 Done Sent Done false
34511 9943 Acked Write 1 Done Sent Done false
34512 9944 Acked Read 5 Done Sent Done false
34513 9945 Acked Read 5 Done Sent Done false
34514 9946 Acked Read 8 Done Sent Done false
34515 9947 Acked Read 8 Done Sent Done false
34516 9948 Acked Read 8 Done Sent Done false
34517 9949 Acked Read 9 Done Sent Done false
34518 9950 Acked Flush 0 Done Sent Done false
34519 9951 Acked Write 9 Done Sent Done false
34520 9952 Acked Write 8 Done Sent Done false
34521 9953 Acked Flush 0 Done Sent Done false
34522 9954 Acked Read 9 Done Sent Done false
34523 9955 Acked Write 2 Done Sent Done false
34524 9956 Acked Write 6 Done Sent Done false
34525 9957 Acked Read 7 Done Sent Done false
34526 9958 Acked Read 8 Done Sent Done false
34527 9959 Acked Read 6 Done Sent Done false
34528 9960 Acked Read 9 Done Sent Done false
34529 9961 Acked Flush 0 Done Sent Done false
34530 9962 Acked Read 6 Done Sent Done false
34531 9963 Acked Write 4 Done Sent Done false
34532 9964 Acked Read 7 Done Sent Done false
34533 9965 Acked Read 9 Done Sent Done false
34534 9966 Acked Write 3 Done Sent Done false
34535 9967 Acked Read 6 Done Sent Done false
34536 9968 Acked Write 3 Done Sent Done false
34537 9969 Acked Read 2 Done Sent Done false
34538 9970 Acked Read 5 Done Sent Done false
34539 9971 Acked Write 4 Done Sent Done false
34540 9972 Acked Write 5 Done Sent Done false
34541 9973 Acked Write 2 Done Sent Done false
34542 9974 Acked Write 10 Done Sent Done false
34543 9975 Acked Flush 0 Done Sent Done false
34544 9976 Acked Write 3 Done Sent Done false
34545 9977 Acked Write 1 Done Sent Done false
34546 9978 Acked Read 2 Done Sent Done false
34547 9979 Acked Read 4 Done Sent Done false
34548 9980 Acked Flush 0 Done Sent Done false
34549 9981 Acked Flush 0 Done Sent Done false
34550 9982 Acked Flush 0 Done Sent Done false
34551 9983 Acked Flush 0 Done Sent Done false
34552 9984 Acked Write 3 Done Sent Done false
34553 9985 Acked Write 6 Done Sent Done false
34554 9986 Acked Write 10 Done Sent Done false
34555 9987 Acked Write 1 Done Sent Done false
34556 9988 Acked Read 1 Done Sent Done false
34557 9989 Acked Read 7 Done Sent Done false
34558 9990 Acked Read 2 Done Sent Done false
34559 9991 Acked Write 1 Done Sent Done false
34560 9992 Acked Read 3 Done Sent Done false
34561 9993 Acked Write 1 Done Sent Done false
34562 9994 Acked Write 9 Done Sent Done false
34563 9995 Acked Write 2 Done Sent Done false
34564 9996 Acked Read 4 Done Sent Done false
34565 9997 Acked Read 6 Done Sent Done false
34566 9998 Acked Read 10 Done Sent Done false
34567 9999 Acked Write 9 Done Sent Done false
34568 10000 Acked Read 1 Done Sent Done false
34569 10001 Acked Read 1 Done Sent Done false
34570 10002 Acked Flush 0 Done Sent Done false
34571 10003 Acked Read 4 Done Sent Done false
34572 10004 Acked Read 8 Done Sent Done false
34573 10005 Acked Write 7 Done Sent Done false
34574 10006 Acked Read 6 Done Sent Done false
34575 10007 Acked Read 7 Done Sent Done false
34576 10008 Acked Read 8 Done Sent Done false
34577 10009 Acked Read 4 Done Sent Done false
34578 10010 Acked Read 6 Done Sent Done false
34579 10011 Acked Read 9 Done Sent Done false
34580 10012 Acked Read 3 Done Sent Done false
34581 10013 Acked Write 5 Done Sent Done false
34582 10014 Acked Read 8 Done Sent Done false
34583 10015 Acked Write 9 Done Sent Done false
34584 10016 Acked Read 8 Done Sent Done false
34585 10017 Acked Read 9 Done Sent Done false
34586 10018 Acked Write 2 Done Sent Done false
34587 10019 Acked Write 3 Done Sent Done false
34588 10020 Acked Write 4 Done Sent Done false
34589 10021 Acked Write 2 Done Sent Done false
34590 10022 Acked Write 6 Done Sent Done false
34591 10023 Acked Flush 0 Done Sent Done false
34592 10024 Acked Write 2 Done Sent Done false
34593 10025 Acked Write 6 Done Sent Done false
34594 10026 Acked Read 3 Done Sent Done false
34595 10027 Acked Write 10 Done Sent Done false
34596 10028 Acked Read 10 Done Sent Done false
34597 10029 Acked Read 4 Done Sent Done false
34598 10030 Acked Read 2 Done Sent Done false
34599 10031 Acked Read 3 Done Sent Done false
34600 10032 Acked Write 5 Done Sent Done false
34601 10033 Acked Read 5 Done Sent Done false
34602 10034 Acked Write 4 Done Sent Done false
34603 10035 Acked Read 5 Done Sent Done false
34604 10036 Acked Read 3 Done Sent Done false
34605 10037 Acked Read 6 Done Sent Done false
34606 10038 Acked Read 1 Done Sent Done false
34607 10039 Acked Write 3 Done Sent Done false
34608 10040 Acked Write 7 Done Sent Done false
34609 10041 Acked Read 5 Done Sent Done false
34610 10042 Acked Read 4 Done Sent Done false
34611 10043 Acked Write 3 Done Sent Done false
34612 10044 Acked Write 9 Done Sent Done false
34613 10045 Acked Flush 0 Done Sent Done false
34614 10046 Acked Write 5 Done Sent Done false
34615 10047 Acked Read 5 Done Sent Done false
34616 10048 Acked Flush 0 Done Sent Done false
34617 10049 Acked Read 9 Done Sent Done false
34618 10050 Acked Flush 0 Done Sent Done false
34619 10051 Acked Flush 0 Done Sent Done false
34620 10052 Acked Read 1 Done Sent Done false
34621 STATES DS:0 DS:1 DS:2 TOTAL
34622 Sent 0 301 0 301
34623 Done 301 0 301 602
34624 Skipped 0 0 0 0
34625 Error 0 0 0 0
34626 Last Flush: Some(JobId(10051)) Some(JobId(9751)) Some(JobId(10051))
34627 Downstairs last five completed: 9751 9750 9749 9748 9747
34628 Upstairs last five completed: 10052 10051 10050 10049 10048
34629 CLIENT: Up:0 ds:301 act:2
346302025-02-25T20:54:04.858ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
346312025-02-25T20:54:05.777ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34632 [8820] got start action
34633 [8820] I am going to restart
34634 Starting downstairs at port 8820
34635 Make output file at "/tmp/dsc/downstairs-8820.txt"
34636 [8820][1] reports Starting
34637 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
34638 [8820][1] reports Running
346392025-02-25T20:54:06.778ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34640 ----------------------------------------------------------------
34641 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34642 States: Active Offline Active
34643 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34644 9752 Acked Write 5 Done Sent Done false
34645 9753 Acked Flush 0 Done Sent Done false
34646 9754 Acked Flush 0 Done Sent Done false
34647 9755 Acked Read 8 Done Sent Done false
34648 9756 Acked Read 5 Done Sent Done false
34649 9757 Acked Flush 0 Done Sent Done false
34650 9758 Acked Write 4 Done Sent Done false
34651 9759 Acked Flush 0 Done Sent Done false
34652 9760 Acked Read 5 Done Sent Done false
34653 9761 Acked Read 3 Done Sent Done false
34654 9762 Acked Write 10 Done Sent Done false
34655 9763 Acked Write 10 Done Sent Done false
34656 9764 Acked Read 8 Done Sent Done false
34657 9765 Acked Read 6 Done Sent Done false
34658 9766 Acked Read 3 Done Sent Done false
34659 9767 Acked Write 6 Done Sent Done false
34660 9768 Acked Read 4 Done Sent Done false
34661 9769 Acked Write 1 Done Sent Done false
34662 9770 Acked Write 1 Done Sent Done false
34663 9771 Acked Read 2 Done Sent Done false
34664 9772 Acked Write 1 Done Sent Done false
34665 9773 Acked Write 10 Done Sent Done false
34666 9774 Acked Read 8 Done Sent Done false
34667 9775 Acked Read 9 Done Sent Done false
34668 9776 Acked Read 5 Done Sent Done false
34669 9777 Acked Read 7 Done Sent Done false
34670 9778 Acked Write 2 Done Sent Done false
34671 9779 Acked Read 1 Done Sent Done false
34672 9780 Acked Write 4 Done Sent Done false
34673 9781 Acked Write 8 Done Sent Done false
34674 9782 Acked Read 1 Done Sent Done false
34675 9783 Acked Read 2 Done Sent Done false
34676 9784 Acked Write 5 Done Sent Done false
34677 9785 Acked Write 1 Done Sent Done false
34678 9786 Acked Read 2 Done Sent Done false
34679 9787 Acked Read 2 Done Sent Done false
34680 9788 Acked Read 2 Done Sent Done false
34681 9789 Acked Read 4 Done Sent Done false
34682 9790 Acked Write 9 Done Sent Done false
34683 9791 Acked Read 8 Done Sent Done false
34684 9792 Acked Read 7 Done Sent Done false
34685 9793 Acked Read 8 Done Sent Done false
34686 9794 Acked Write 3 Done Sent Done false
34687 9795 Acked Write 5 Done Sent Done false
34688 9796 Acked Read 6 Done Sent Done false
34689 9797 Acked Write 6 Done Sent Done false
34690 9798 Acked Read 1 Done Sent Done false
34691 9799 Acked Write 3 Done Sent Done false
34692 9800 Acked Write 4 Done Sent Done false
34693 9801 Acked Read 6 Done Sent Done false
34694 9802 Acked Read 1 Done Sent Done false
34695 9803 Acked Read 10 Done Sent Done false
34696 9804 Acked Write 2 Done Sent Done false
34697 9805 Acked Write 5 Done Sent Done false
34698 9806 Acked Read 4 Done Sent Done false
34699 9807 Acked Read 1 Done Sent Done false
34700 9808 Acked Write 6 Done Sent Done false
34701 9809 Acked Flush 0 Done Sent Done false
34702 9810 Acked Read 5 Done Sent Done false
34703 9811 Acked Read 2 Done Sent Done false
34704 9812 Acked Read 5 Done Sent Done false
34705 9813 Acked Read 10 Done Sent Done false
34706 9814 Acked Flush 0 Done Sent Done false
34707 9815 Acked Read 7 Done Sent Done false
34708 9816 Acked Read 1 Done Sent Done false
34709 9817 Acked Flush 0 Done Sent Done false
34710 9818 Acked Read 2 Done Sent Done false
34711 9819 Acked Read 6 Done Sent Done false
34712 9820 Acked Write 4 Done Sent Done false
34713 9821 Acked Flush 0 Done Sent Done false
34714 9822 Acked Read 5 Done Sent Done false
34715 9823 Acked Write 2 Done Sent Done false
34716 9824 Acked Write 6 Done Sent Done false
34717 9825 Acked Read 1 Done Sent Done false
34718 9826 Acked Read 1 Done Sent Done false
34719 9827 Acked Write 5 Done Sent Done false
34720 9828 Acked Read 8 Done Sent Done false
34721 9829 Acked Read 7 Done Sent Done false
34722 9830 Acked Write 2 Done Sent Done false
34723 9831 Acked Read 7 Done Sent Done false
34724 9832 Acked Flush 0 Done Sent Done false
34725 9833 Acked Write 9 Done Sent Done false
34726 9834 Acked Read 3 Done Sent Done false
34727 9835 Acked Read 3 Done Sent Done false
34728 9836 Acked Flush 0 Done Sent Done false
34729 9837 Acked Read 10 Done Sent Done false
34730 9838 Acked Read 10 Done Sent Done false
34731 9839 Acked Read 4 Done Sent Done false
34732 9840 Acked Write 4 Done Sent Done false
34733 9841 Acked Read 9 Done Sent Done false
34734 9842 Acked Write 4 Done Sent Done false
34735 9843 Acked Read 2 Done Sent Done false
34736 9844 Acked Write 8 Done Sent Done false
34737 9845 Acked Flush 0 Done Sent Done false
34738 9846 Acked Read 3 Done Sent Done false
34739 9847 Acked Write 4 Done Sent Done false
34740 9848 Acked Read 1 Done Sent Done false
34741 9849 Acked Write 6 Done Sent Done false
34742 9850 Acked Write 7 Done Sent Done false
34743 9851 Acked Write 8 Done Sent Done false
34744 9852 Acked Read 9 Done Sent Done false
34745 9853 Acked Write 6 Done Sent Done false
34746 9854 Acked Write 3 Done Sent Done false
34747 9855 Acked Read 2 Done Sent Done false
34748 9856 Acked Read 8 Done Sent Done false
34749 9857 Acked Flush 0 Done Sent Done false
34750 9858 Acked Read 8 Done Sent Done false
34751 9859 Acked Flush 0 Done Sent Done false
34752 9860 Acked Read 7 Done Sent Done false
34753 9861 Acked Read 3 Done Sent Done false
34754 9862 Acked Read 2 Done Sent Done false
34755 9863 Acked Read 1 Done Sent Done false
34756 9864 Acked Read 4 Done Sent Done false
34757 9865 Acked Read 5 Done Sent Done false
34758 9866 Acked Flush 0 Done Sent Done false
34759 9867 Acked Write 2 Done Sent Done false
34760 9868 Acked Write 7 Done Sent Done false
34761 9869 Acked Read 9 Done Sent Done false
34762 9870 Acked Write 1 Done Sent Done false
34763 9871 Acked Read 9 Done Sent Done false
34764 9872 Acked Read 5 Done Sent Done false
34765 9873 Acked Read 3 Done Sent Done false
34766 9874 Acked Flush 0 Done Sent Done false
34767 9875 Acked Flush 0 Done Sent Done false
34768 9876 Acked Write 7 Done Sent Done false
34769 9877 Acked Read 1 Done Sent Done false
34770 9878 Acked Write 8 Done Sent Done false
34771 9879 Acked Flush 0 Done Sent Done false
34772 9880 Acked Read 9 Done Sent Done false
34773 9881 Acked Write 3 Done Sent Done false
34774 9882 Acked Write 6 Done Sent Done false
34775 9883 Acked Write 10 Done Sent Done false
34776 9884 Acked Read 2 Done Sent Done false
34777 9885 Acked Read 6 Done Sent Done false
34778 9886 Acked Write 2 Done Sent Done false
34779 9887 Acked Flush 0 Done Sent Done false
34780 9888 Acked Read 2 Done Sent Done false
34781 9889 Acked Read 10 Done Sent Done false
34782 9890 Acked Write 1 Done Sent Done false
34783 9891 Acked Read 1 Done Sent Done false
34784 9892 Acked Read 5 Done Sent Done false
34785 9893 Acked Write 3 Done Sent Done false
34786 9894 Acked Read 3 Done Sent Done false
34787 9895 Acked Write 3 Done Sent Done false
34788 9896 Acked Read 7 Done Sent Done false
34789 9897 Acked Read 4 Done Sent Done false
34790 9898 Acked Read 9 Done Sent Done false
34791 9899 Acked Read 5 Done Sent Done false
34792 9900 Acked Read 4 Done Sent Done false
34793 9901 Acked Read 10 Done Sent Done false
34794 9902 Acked Read 9 Done Sent Done false
34795 9903 Acked Write 6 Done Sent Done false
34796 9904 Acked Read 8 Done Sent Done false
34797 9905 Acked Flush 0 Done Sent Done false
34798 9906 Acked Read 4 Done Sent Done false
34799 9907 Acked Read 4 Done Sent Done false
34800 9908 Acked Read 1 Done Sent Done false
34801 9909 Acked Read 7 Done Sent Done false
34802 9910 Acked Read 1 Done Sent Done false
34803 9911 Acked Flush 0 Done Sent Done false
34804 9912 Acked Flush 0 Done Sent Done false
34805 9913 Acked Write 4 Done Sent Done false
34806 9914 Acked Read 3 Done Sent Done false
34807 9915 Acked Read 2 Done Sent Done false
34808 9916 Acked Read 1 Done Sent Done false
34809 9917 Acked Write 4 Done Sent Done false
34810 9918 Acked Flush 0 Done Sent Done false
34811 9919 Acked Write 8 Done Sent Done false
34812 9920 Acked Flush 0 Done Sent Done false
34813 9921 Acked Read 3 Done Sent Done false
34814 9922 Acked Write 8 Done Sent Done false
34815 9923 Acked Write 5 Done Sent Done false
34816 9924 Acked Read 5 Done Sent Done false
34817 9925 Acked Read 4 Done Sent Done false
34818 9926 Acked Read 1 Done Sent Done false
34819 9927 Acked Read 7 Done Sent Done false
34820 9928 Acked Read 8 Done Sent Done false
34821 9929 Acked Read 1 Done Sent Done false
34822 9930 Acked Read 3 Done Sent Done false
34823 9931 Acked Read 4 Done Sent Done false
34824 9932 Acked Write 5 Done Sent Done false
34825 9933 Acked Read 7 Done Sent Done false
34826 9934 Acked Write 6 Done Sent Done false
34827 9935 Acked Flush 0 Done Sent Done false
34828 9936 Acked Read 7 Done Sent Done false
34829 9937 Acked Write 9 Done Sent Done false
34830 9938 Acked Read 6 Done Sent Done false
34831 9939 Acked Write 9 Done Sent Done false
34832 9940 Acked Flush 0 Done Sent Done false
34833 9941 Acked Write 10 Done Sent Done false
34834 9942 Acked Write 10 Done Sent Done false
34835 9943 Acked Write 1 Done Sent Done false
34836 9944 Acked Read 5 Done Sent Done false
34837 9945 Acked Read 5 Done Sent Done false
34838 9946 Acked Read 8 Done Sent Done false
34839 9947 Acked Read 8 Done Sent Done false
34840 9948 Acked Read 8 Done Sent Done false
34841 9949 Acked Read 9 Done Sent Done false
34842 9950 Acked Flush 0 Done Sent Done false
34843 9951 Acked Write 9 Done Sent Done false
34844 9952 Acked Write 8 Done Sent Done false
34845 9953 Acked Flush 0 Done Sent Done false
34846 9954 Acked Read 9 Done Sent Done false
34847 9955 Acked Write 2 Done Sent Done false
34848 9956 Acked Write 6 Done Sent Done false
34849 9957 Acked Read 7 Done Sent Done false
34850 9958 Acked Read 8 Done Sent Done false
34851 9959 Acked Read 6 Done Sent Done false
34852 9960 Acked Read 9 Done Sent Done false
34853 9961 Acked Flush 0 Done Sent Done false
34854 9962 Acked Read 6 Done Sent Done false
34855 9963 Acked Write 4 Done Sent Done false
34856 9964 Acked Read 7 Done Sent Done false
34857 9965 Acked Read 9 Done Sent Done false
34858 9966 Acked Write 3 Done Sent Done false
34859 9967 Acked Read 6 Done Sent Done false
34860 9968 Acked Write 3 Done Sent Done false
34861 9969 Acked Read 2 Done Sent Done false
34862 9970 Acked Read 5 Done Sent Done false
34863 9971 Acked Write 4 Done Sent Done false
34864 9972 Acked Write 5 Done Sent Done false
34865 9973 Acked Write 2 Done Sent Done false
34866 9974 Acked Write 10 Done Sent Done false
34867 9975 Acked Flush 0 Done Sent Done false
34868 9976 Acked Write 3 Done Sent Done false
34869 9977 Acked Write 1 Done Sent Done false
34870 9978 Acked Read 2 Done Sent Done false
34871 9979 Acked Read 4 Done Sent Done false
34872 9980 Acked Flush 0 Done Sent Done false
34873 9981 Acked Flush 0 Done Sent Done false
34874 9982 Acked Flush 0 Done Sent Done false
34875 9983 Acked Flush 0 Done Sent Done false
34876 9984 Acked Write 3 Done Sent Done false
34877 9985 Acked Write 6 Done Sent Done false
34878 9986 Acked Write 10 Done Sent Done false
34879 9987 Acked Write 1 Done Sent Done false
34880 9988 Acked Read 1 Done Sent Done false
34881 9989 Acked Read 7 Done Sent Done false
34882 9990 Acked Read 2 Done Sent Done false
34883 9991 Acked Write 1 Done Sent Done false
34884 9992 Acked Read 3 Done Sent Done false
34885 9993 Acked Write 1 Done Sent Done false
34886 9994 Acked Write 9 Done Sent Done false
34887 9995 Acked Write 2 Done Sent Done false
34888 9996 Acked Read 4 Done Sent Done false
34889 9997 Acked Read 6 Done Sent Done false
34890 9998 Acked Read 10 Done Sent Done false
34891 9999 Acked Write 9 Done Sent Done false
34892 10000 Acked Read 1 Done Sent Done false
34893 10001 Acked Read 1 Done Sent Done false
34894 10002 Acked Flush 0 Done Sent Done false
34895 10003 Acked Read 4 Done Sent Done false
34896 10004 Acked Read 8 Done Sent Done false
34897 10005 Acked Write 7 Done Sent Done false
34898 10006 Acked Read 6 Done Sent Done false
34899 10007 Acked Read 7 Done Sent Done false
34900 10008 Acked Read 8 Done Sent Done false
34901 10009 Acked Read 4 Done Sent Done false
34902 10010 Acked Read 6 Done Sent Done false
34903 10011 Acked Read 9 Done Sent Done false
34904 10012 Acked Read 3 Done Sent Done false
34905 10013 Acked Write 5 Done Sent Done false
34906 10014 Acked Read 8 Done Sent Done false
34907 10015 Acked Write 9 Done Sent Done false
34908 10016 Acked Read 8 Done Sent Done false
34909 10017 Acked Read 9 Done Sent Done false
34910 10018 Acked Write 2 Done Sent Done false
34911 10019 Acked Write 3 Done Sent Done false
34912 10020 Acked Write 4 Done Sent Done false
34913 10021 Acked Write 2 Done Sent Done false
34914 10022 Acked Write 6 Done Sent Done false
34915 10023 Acked Flush 0 Done Sent Done false
34916 10024 Acked Write 2 Done Sent Done false
34917 10025 Acked Write 6 Done Sent Done false
34918 10026 Acked Read 3 Done Sent Done false
34919 10027 Acked Write 10 Done Sent Done false
34920 10028 Acked Read 10 Done Sent Done false
34921 10029 Acked Read 4 Done Sent Done false
34922 10030 Acked Read 2 Done Sent Done false
34923 10031 Acked Read 3 Done Sent Done false
34924 10032 Acked Write 5 Done Sent Done false
34925 10033 Acked Read 5 Done Sent Done false
34926 10034 Acked Write 4 Done Sent Done false
34927 10035 Acked Read 5 Done Sent Done false
34928 10036 Acked Read 3 Done Sent Done false
34929 10037 Acked Read 6 Done Sent Done false
34930 10038 Acked Read 1 Done Sent Done false
34931 10039 Acked Write 3 Done Sent Done false
34932 10040 Acked Write 7 Done Sent Done false
34933 10041 Acked Read 5 Done Sent Done false
34934 10042 Acked Read 4 Done Sent Done false
34935 10043 Acked Write 3 Done Sent Done false
34936 10044 Acked Write 9 Done Sent Done false
34937 10045 Acked Flush 0 Done Sent Done false
34938 10046 Acked Write 5 Done Sent Done false
34939 10047 Acked Read 5 Done Sent Done false
34940 10048 Acked Flush 0 Done Sent Done false
34941 10049 Acked Read 9 Done Sent Done false
34942 10050 Acked Flush 0 Done Sent Done false
34943 10051 Acked Flush 0 Done Sent Done false
34944 10052 Acked Read 1 Done Sent Done false
34945 STATES DS:0 DS:1 DS:2 TOTAL
34946 Sent 0 301 0 301
34947 Done 301 0 301 602
34948 Skipped 0 0 0 0
34949 Error 0 0 0 0
34950 Last Flush: Some(JobId(10051)) Some(JobId(9751)) Some(JobId(10051))
34951 Downstairs last five completed: 9751 9750 9749 9748 9747
34952 Upstairs last five completed: 10052 10051 10050 10049 10048
34953 CLIENT: Up:0 ds:301 act:2
349542025-02-25T20:54:08.861ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349552025-02-25T20:54:09.781ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349562025-02-25T20:54:10.111ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349572025-02-25T20:54:10.111ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50585) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349582025-02-25T20:54:10.113ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349592025-02-25T20:54:10.113ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349602025-02-25T20:54:10.113ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349612025-02-25T20:54:10.113ZINFOcrucible: send last flush ID to this DS: Some(JobId(9751)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349622025-02-25T20:54:10.113ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349632025-02-25T20:54:10.113ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9751)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
349642025-02-25T20:54:10.113ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9751)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34965 ----------------------------------------------------------------
34966 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34967 Downstairs last five completed: 10053 10052 10051 10050 10049
34968 Upstairs last five completed: 10053 10052 10051 10050 10049
34969 CLIENT: Up:0 ds:0 act:3
34970 Replay: All jobs finished
34971 Main task has work to do, go find it
34972 got dsc Stop(1)
34973 stop 1
34974 Feb 25 20:54:12.363[8820] Got stop action so:false kr:false
34975 INFO request completed, latency_us: 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e9a7362d-bb84-4f4b-bf76-2f8767d75069, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
349762025-02-25T20:54:12.364ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
34977 Feb 25 20:54:12.364{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:54:12.364124264Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
34978 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/1, method: GET, req_id: 296f9cae-6ff7-472d-9f1a-da9468070e05, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T20:54:12.364227317Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
34979 remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T20:54:12.364265244Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0093020f-8db1-4b17-8f9d-499d3d7b91d5"}
34980 
34981 [8820] Exited with: ExitStatus(unix_wait_status(9))
34982 [8820][1] reports Exit
34983 Feb 25 20:54:16.366 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/1, method: GET, req_id: 086a5b22-cfb6-457e-97c9-5b89c2640721, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
34984 001/300 Read block 2068 len 40960
34985 002/300 Write block 1404 len 4096 data: 3
34986 003/300 Read block 265 len 24576
34987 004/300 Read block 3062 len 20480
34988 005/300 Write block 2714 len 40960 data: 4 4 3 3 5 6 6 3 3 3
34989 006/300 Write block 223 len 36864 data: 8 7 6 7 8 8 5 3 2
34990 007/300 Write block 3529 len 16384 data: 5 6 7 7
34991 008/300 Write block 413 len 12288 data: 7 9 7
34992 009/300 Write block 1477 len 16384 data: 7 8 8 8
34993 010/300 Write block 900 len 36864 data: 6 6 5 5 6 6 4 4 3
34994 011/300 Write block 3269 len 32768 data: 8 9 5 6 7 6 6 8
34995 012/300 Write block 2563 len 12288 data: 2 2 2
34996 013/300 Read block 3706 len 4096
34997 014/300 Read block 3530 len 32768
34998 015/300 Flush
34999 016/300 Write block 2665 len 32768 data: 7 6 7 8 5 4 3 3
35000 017/300 Write block 3760 len 16384 data: 6 6 6 8
35001 018/300 Read block 3134 len 40960
35002 019/300 Write block 4157 len 32768 data: 5 3 6 8 11 13 12 12
35003 020/300 Read block 3766 len 28672
35004 021/300 Flush
35005 022/300 Read block 3871 len 32768
35006 023/300 Read block 371 len 32768
35007 024/300 Write block 3644 len 24576 data: 11 9 10 6 6 7
35008 025/300 Write block 403 len 20480 data: 4 6 6 8 8
35009 026/300 Read block 3674 len 32768
35010 027/300 Read block 3181 len 28672
35011 028/300 Read block 1471 len 32768
35012 029/300 Read block 3438 len 16384
35013 030/300 Write block 3390 len 16384 data: 6 6 6 6
35014 031/300 Read block 2483 len 4096
35015 032/300 Write block 4215 len 12288 data: 6 6 8
35016 033/300 Read block 1846 len 36864
35017 034/300 Write block 2107 len 20480 data: 8 8 6 4 3
35018 035/300 Read block 1109 len 12288
35019 036/300 Flush
35020 037/300 Read block 3855 len 24576
35021 038/300 Write block 1364 len 12288 data: 8 7 5
35022 039/300 Write block 4497 len 24576 data: 3 4 5 4 4 3
35023 040/300 Read block 2897 len 8192
35024 041/300 Write block 3612 len 28672 data: 8 7 5 5 6 8 7
35025 042/300 Write block 1401 len 12288 data: 3 3 4
35026 043/300 Read block 4382 len 36864
35027 044/300 Write block 2463 len 12288 data: 5 8 6
35028 045/300 Read block 2723 len 28672
35029 046/300 Read block 3818 len 12288
35030 047/300 Write block 2007 len 28672 data: 8 9 10 10 7 7 6
35031 048/300 Read block 4348 len 28672
35032 049/300 Flush
35033 050/300 Read block 4523 len 16384
35034 051/300 Write block 711 len 8192 data: 7 7
35035 052/300 Read block 567 len 4096
35036 053/300 Flush
35037 054/300 Read block 193 len 24576
35038 055/300 Write block 2837 len 16384 data: 5 6 7 7
35039 056/300 Write block 4322 len 24576 data: 5 6 6 6 7 7
35040 057/300 Write block 4435 len 16384 data: 6 6 6 5
35041 058/300 Write block 3375 len 12288 data: 4 5 6
35042 059/300 Write block 1331 len 12288 data: 5 5 5
35043 060/300 Write block 1657 len 32768 data: 12 10 8 9 7 7 6 5
35044 061/300 Read block 116 len 4096
35045 062/300 Read block 2509 len 24576
35046 063/300 Read block 134 len 28672
35047 064/300 Write block 665 len 24576 data: 9 11 10 11 6 4
35048 065/300 Read block 1925 len 8192
35049 066/300 Write block 4933 len 12288 data: 5 4 5
35050 067/300 Write block 3668 len 28672 data: 3 3 2 2 2 2 2
35051 068/300 Read block 2882 len 24576
35052 069/300 Read block 626 len 4096
35053 070/300 Write block 4774 len 28672 data: 7 7 7 8 4 4 5
35054 071/300 Read block 1451 len 8192
35055 072/300 Flush
35056 073/300 Write block 1638 len 40960 data: 6 7 4 4 4 4 3 4 4 4
35057 074/300 Write block 388 len 8192 data: 5 6
35058 075/300 Read block 2388 len 24576
35059 076/300 Write block 1210 len 28672 data: 11 11 11 9 8 6 5
35060 077/300 Read block 3469 len 16384
35061 078/300 Read block 1413 len 32768
35062 079/300 Write block 2940 len 36864 data: 5 5 6 4 4 5 5 4 5
35063 080/300 Read block 1621 len 28672
35064 081/300 Write block 3582 len 28672 data: 3 2 2 3 8 8 7
35065 082/300 Read block 1585 len 36864
35066 083/300 Write block 212 len 4096 data: 7
35067 084/300 Write block 1645 len 12288 data: 5 5 5
35068 085/300 Flush
35069 086/300 Write block 1405 len 40960 data: 3 3 3 4 6 6 6 5 4 5
35070 087/300 Write block 4842 len 16384 data: 5 4 4 4
35071 088/300 Read block 1896 len 4096
35072 089/300 Flush
35073 090/300 Write block 2128 len 20480 data: 7 6 5 6 5
35074 091/300 Read block 1366 len 36864
35075 092/300 Flush
35076 093/300 Read block 3750 len 28672
35077 094/300 Read block 3582 len 28672
35078 095/300 Read block 598 len 28672
35079 096/300 Read block 2527 len 20480
35080 097/300 Write block 3947 len 4096 data: 9
35081 098/300 Write block 4903 len 4096 data: 7
35082 099/300 Read block 4118 len 12288
35083 100/300 Write block 1930 len 32768 data: 3 3 3 3 4 4 4 4
35084 101/300 Write block 2761 len 4096 data: 5
35085 102/300 Read block 719 len 32768
35086 103/300 Read block 3610 len 24576
35087 104/300 Write block 2681 len 12288 data: 11 10 9
35088 105/300 Read block 4417 len 4096
35089 106/300 Write block 4834 len 8192 data: 6 6
35090 107/300 Write block 2740 len 12288 data: 6 6 7
35091 108/300 Write block 2558 len 32768 data: 6 4 3 2 2 3 3 3
35092 109/300 Read block 4305 len 20480
35093 110/300 Read block 3003 len 8192
35094 111/300 Read block 3050 len 4096
35095 112/300 Write block 470 len 20480 data: 4 4 4 4 3
35096 113/300 Write block 155 len 12288 data: 5 4 4
35097 114/300 Write block 3664 len 28672 data: 6 6 6 4 4 4 3
35098 115/300 Read block 24 len 40960
35099 116/300 Read block 483 len 8192
35100 117/300 Read block 4737 len 28672
35101 118/300 Write block 212 len 16384 data: 8 9 9 11
35102 119/300 Read block 2934 len 20480
35103 120/300 Write block 4937 len 28672 data: 6 6 7 8 8 9 9
35104 121/300 Write block 4830 len 32768 data: 3 5 6 6 7 7 6 5
35105 122/300 Flush
35106 123/300 Write block 122 len 4096 data: 4
35107 124/300 Read block 4041 len 20480
35108 125/300 Read block 1684 len 4096
35109 126/300 Read block 1978 len 8192
35110 127/300 Read block 3541 len 20480
35111 128/300 Write block 2675 len 40960 data: 5 5 5 5 5 10 12 11 10 9
35112 129/300 Read block 4542 len 28672
35113 130/300 Read block 4680 len 32768
35114 131/300 Flush
35115 132/300 Write block 4462 len 4096 data: 12
35116 133/300 Read block 551 len 36864
35117 134/300 Read block 4438 len 16384
35118 135/300 Write block 1451 len 4096 data: 5
35119 136/300 Write block 1291 len 20480 data: 4 4 4 2 2
35120 137/300 Read block 2889 len 16384
35121 138/300 Write block 399 len 8192 data: 3 2
35122 139/300 Read block 3294 len 40960
35123 140/300 Read block 576 len 16384
35124 141/300 Read block 3 len 12288
35125 142/300 Read block 4102 len 24576
35126 143/300 Read block 3479 len 4096
35127 144/300 Read block 1471 len 20480
35128 145/300 Write block 2907 len 8192 data: 6 6
35129 146/300 Write block 550 len 16384 data: 7 6 6 8
35130 147/300 Read block 4140 len 28672
35131 148/300 Write block 3222 len 28672 data: 4 5 5 4 4 4 4
35132 149/300 Write block 4424 len 20480 data: 6 7 9 7 7
35133 150/300 Write block 3402 len 12288 data: 5 6 6
35134 151/300 Read block 3169 len 40960
35135 152/300 Read block 2763 len 40960
35136 153/300 Read block 4527 len 20480
35137 154/300 Write block 2111 len 28672 data: 4 6 5 6 6 7 6
35138 155/300 Read block 2783 len 28672
35139 156/300 Write block 1757 len 4096 data: 4
35140 157/300 Read block 4779 len 20480
35141 158/300 Write block 752 len 28672 data: 3 3 2 2 2 2 4
35142 159/300 Write block 908 len 8192 data: 4 2
35143 160/300 Write block 4888 len 24576 data: 6 7 7 5 6 7
35144 161/300 Read block 156 len 40960
35145 162/300 Read block 3524 len 32768
35146 163/300 Read block 2293 len 12288
35147 164/300 Write block 747 len 12288 data: 5 4 4
35148 165/300 Read block 2063 len 32768
35149 166/300 Read block 3499 len 32768
35150 167/300 Read block 1825 len 24576
35151 168/300 Write block 4152 len 8192 data: 7 6
35152 169/300 Write block 2177 len 4096 data: 3
35153 170/300 Read block 2206 len 40960
35154 171/300 Write block 738 len 20480 data: 4 3 2 2 4
35155 172/300 Read block 4278 len 32768
35156 173/300 Flush
35157 174/300 Read block 984 len 36864
35158 175/300 Write block 2782 len 8192 data: 8 9
35159 176/300 Write block 2609 len 24576 data: 7 7 7 7 7 6
35160 177/300 Read block 2788 len 28672
35161 178/300 Read block 1016 len 4096
35162 179/300 Write block 3181 len 24576 data: 8 8 8 7 7 8
35163 180/300 Read block 4079 len 24576
35164 181/300 Write block 1263 len 4096 data: 8
35165 182/300 Read block 3653 len 4096
35166 183/300 Read block 841 len 28672
35167 184/300 Write block 1172 len 8192 data: 5 6
35168 185/300 Write block 4680 len 12288 data: 4 3 3
35169 186/300 Read block 1281 len 4096
35170 187/300 Write block 4870 len 32768 data: 5 6 7 7 8 8 9 9
35171 188/300 Read block 2078 len 28672
35172 189/300 Read block 2019 len 32768
35173 190/300 Read block 3210 len 4096
35174 191/300 Read block 4523 len 28672
35175 192/300 Write block 434 len 40960 data: 6 8 7 10 8 6 8 8 8 8
35176 193/300 Read block 2075 len 24576
35177 194/300 Read block 3091 len 12288
35178 195/300 Flush
35179 196/300 Write block 1327 len 20480 data: 6 7 6 6 6
35180 197/300 Write block 4537 len 4096 data: 4
35181 198/300 Read block 168 len 8192
35182 199/300 Flush
35183 200/300 Flush
35184 201/300 Read block 1257 len 40960
35185 202/300 Write block 1740 len 28672 data: 3 4 3 2 3 4 4
35186 203/300 Write block 338 len 24576 data: 5 4 3 4 4 6
35187 204/300 Write block 417 len 28672 data: 6 5 6 6 10 10 9
35188 205/300 Flush
35189 206/300 Write block 4019 len 40960 data: 4 4 5 6 7 6 6 5 5 5
35190 207/300 Read block 4967 len 40960
35191 208/300 Read block 744 len 4096
35192 209/300 Flush
35193 210/300 Write block 3549 len 4096 data: 5
35194 211/300 Read block 2091 len 8192
35195 212/300 Write block 1885 len 20480 data: 3 6 6 6 6
35196 213/300 Write block 855 len 8192 data: 7 8
35197 214/300 Read block 3053 len 36864
35198 215/300 Read block 3539 len 8192
35199 216/300 Write block 2656 len 12288 data: 4 5 4
35200 217/300 Read block 2491 len 32768
35201 218/300 Read block 975 len 20480
35202 219/300 Write block 2547 len 24576 data: 7 8 8 6 6 8
35203 220/300 Read block 2990 len 28672
35204 221/300 Write block 4815 len 8192 data: 6 4
35205 222/300 Write block 191 len 28672 data: 8 7 7 7 7 7 6
35206 223/300 Write block 2409 len 24576 data: 4 4 4 6 7 7
35207 224/300 Read block 3525 len 36864
35208 225/300 Write block 435 len 32768 data: 9 8 11 9 7 9 9 9
35209 226/300 Read block 2613 len 32768
35210 227/300 Write block 293 len 32768 data: 2 2 2 3 4 3 4 3
35211 228/300 Read block 1194 len 12288
35212 229/300 Read block 926 len 24576
35213 230/300 Write block 4777 len 32768 data: 9 5 5 6 6 6 6 5
35214 231/300 Write block 3501 len 28672 data: 5 5 4 4 3 2 2
35215 232/300 Write block 4955 len 24576 data: 6 6 5 5 5 5
35216 233/300 Read block 4523 len 32768
35217 234/300 Write block 3796 len 12288 data: 6 7 7
35218 235/300 Write block 452 len 8192 data: 7 6
35219 236/300 Read block 4110 len 12288
35220 237/300 Read block 2028 len 8192
35221 238/300 Write block 2874 len 20480 data: 5 4 4 4 5
35222 239/300 Read block 2607 len 40960
35223 240/300 Write block 2889 len 4096 data: 9
35224 241/300 Read block 711 len 8192
35225 242/300 Write block 223 len 36864 data: 9 8 7 8 9 9 6 4 3
35226 243/300 Read block 3025 len 28672
35227 244/300 Write block 19 len 24576 data: 6 5 5 3 3 3
35228 245/300 Write block 1002 len 8192 data: 7 7
35229 246/300 Read block 1885 len 12288
35230 247/300 Flush
35231 248/300 Flush
35232 249/300 Read block 1705 len 28672
35233 250/300 Read block 2135 len 16384
35234 251/300 Write block 4587 len 20480 data: 4 4 6 7 6
35235 252/300 Write block 329 len 28672 data: 7 6 6 9 8 9 9
35236 253/300 Write block 4643 len 36864 data: 7 6 7 7 4 4 3 3 4
35237 254/300 Read block 4351 len 40960
35238 255/300 Write block 4542 len 20480 data: 4 4 4 4 5
35239 256/300 Read block 1216 len 16384
35240 257/300 Flush
35241 258/300 Read block 1756 len 40960
35242 259/300 Write block 1337 len 8192 data: 5 5
35243 260/300 Write block 2109 len 20480 data: 7 5 5 7 6
35244 261/300 Write block 1483 len 12288 data: 3 4 4
35245 262/300 Read block 1597 len 32768
35246 263/300 Read block 4557 len 4096
35247 264/300 Read block 1361 len 28672
35248 265/300 Read block 972 len 24576
35249 266/300 Read block 4687 len 8192
35250 267/300 Write block 2191 len 8192 data: 6 6
35251 268/300 Write block 460 len 28672 data: 5 4 4 4 3 4 3
35252 269/300 Read block 1426 len 24576
35253 270/300 Write block 514 len 32768 data: 6 6 6 5 5 4 4 4
35254 271/300 Read block 3314 len 32768
35255 272/300 Write block 4505 len 16384 data: 4 4 5 4
35256 273/300 Flush
35257 274/300 Write block 4674 len 12288 data: 6 6 6
35258 275/300 Flush
35259 276/300 Write block 4190 len 4096 data: 4
35260 277/300 Read block 3207 len 8192
35261 278/300 Read block 1663 len 32768
35262 279/300 Flush
35263 280/300 Write block 605 len 32768 data: 7 5 5 5 5 5 5 6
35264 281/300 Read block 104 len 12288
35265 282/300 Flush
35266 283/300 Read block 788 len 20480
35267 284/300 Flush
35268 285/300 Flush
35269 286/300 Write block 2887 len 8192 data: 5 6
35270 287/300 Write block 1649 len 8192 data: 7 6
35271 288/300 Read block 1813 len 16384
35272 289/300 Write block 71 len 32768 data: 6 6 7 6 6 6 6 6
35273 290/300 Read block 3321 len 12288
35274 291/300 Read block 3574 len 8192
35275 292/300 Write block 3602 len 36864 data: 4 4 5 5 4 4 4 6 6
35276 293/300 Write block 2190 len 28672 data: 6 7 7 5 6 5 4
35277 294/300 Read block 877 len 12288
35278 295/300 Write block 4285 len 24576 data: 5 5 3 4 5 5
35279 296/300 Read block 1067 len 16384
35280 297/300 Write block 4401 len 8192 data: 5 6
35281 298/300 Write block 2159 len 4096 data: 10
35282 299/300 Write block 1364 len 28672 data: 9 8 6 7 6 7 7
35283 300/300 Read block 2281 len 24576
35284 301/300 Read block 3651 len 28672
35285 Main task has work to do, go find it
35286 got dsc Start(1)
35287 start 1
35288 Feb 25 20:54:16.603 INFO request completed, latency_us: 248, response_code: 204, uri: /start/cid/1, method: POST, req_id: 345ad286-06da-4d9c-93ea-a59d3ba058f2, remote_addr: 127.0.0.1:43930, local_addr: 127.0.0.1:9998
35289 Replay: started 1, returned:Ok(())
35290 ----------------------------------------------------------------
35291 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35292 States: Active Offline Active
35293 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35294 10054 Acked Read 10 Done Sent Done false
35295 10055 Acked Write 1 Done Sent Done false
35296 10056 Acked Read 6 Done Sent Done false
35297 10057 Acked Read 5 Done Sent Done false
35298 10058 Acked Write 10 Done Sent Done false
35299 10059 Acked Write 9 Done Sent Done false
35300 10060 Acked Write 4 Done Sent Done false
35301 10061 Acked Write 3 Done Sent Done false
35302 10062 Acked Write 4 Done Sent Done false
35303 10063 Acked Write 9 Done Sent Done false
35304 10064 Acked Write 8 Done Sent Done false
35305 10065 Acked Write 3 Done Sent Done false
35306 10066 Acked Read 1 Done Sent Done false
35307 10067 Acked Read 8 Done Sent Done false
35308 10068 Acked Flush 0 Done Sent Done false
35309 10069 Acked Write 8 Done Sent Done false
35310 10070 Acked Write 4 Done Sent Done false
35311 10071 Acked Read 10 Done Sent Done false
35312 10072 Acked Write 8 Done Sent Done false
35313 10073 Acked Read 7 Done Sent Done false
35314 10074 Acked Flush 0 Done Sent Done false
35315 10075 Acked Read 8 Done Sent Done false
35316 10076 Acked Read 8 Done Sent Done false
35317 10077 Acked Write 6 Done Sent Done false
35318 10078 Acked Write 5 Done Sent Done false
35319 10079 Acked Read 8 Done Sent Done false
35320 10080 Acked Read 7 Done Sent Done false
35321 10081 Acked Read 8 Done Sent Done false
35322 10082 Acked Read 4 Done Sent Done false
35323 10083 Acked Write 4 Done Sent Done false
35324 10084 Acked Read 1 Done Sent Done false
35325 10085 Acked Write 3 Done Sent Done false
35326 10086 Acked Read 9 Done Sent Done false
35327 10087 Acked Write 5 Done Sent Done false
35328 10088 Acked Read 3 Done Sent Done false
35329 10089 Acked Flush 0 Done Sent Done false
35330 10090 Acked Read 6 Done Sent Done false
35331 10091 Acked Write 3 Done Sent Done false
35332 10092 Acked Write 6 Done Sent Done false
35333 10093 Acked Read 2 Done Sent Done false
35334 10094 Acked Write 7 Done Sent Done false
35335 10095 Acked Write 3 Done Sent Done false
35336 10096 Acked Read 9 Done Sent Done false
35337 10097 Acked Write 3 Done Sent Done false
35338 10098 Acked Read 7 Done Sent Done false
35339 10099 Acked Read 3 Done Sent Done false
35340 10100 Acked Write 7 Done Sent Done false
35341 10101 Acked Read 7 Done Sent Done false
35342 10102 Acked Flush 0 Done Sent Done false
35343 10103 Acked Read 4 Done Sent Done false
35344 10104 Acked Write 2 Done Sent Done false
35345 10105 Acked Read 1 Done Sent Done false
35346 10106 Acked Flush 0 Done Sent Done false
35347 10107 Acked Read 6 Done Sent Done false
35348 10108 Acked Write 4 Done Sent Done false
35349 10109 Acked Write 6 Done Sent Done false
35350 10110 Acked Write 4 Done Sent Done false
35351 10111 Acked Write 3 Done Sent Done false
35352 10112 Acked Write 3 Done Sent Done false
35353 10113 Acked Write 8 Done Sent Done false
35354 10114 Acked Read 1 Done Sent Done false
35355 10115 Acked Read 6 Done Sent Done false
35356 10116 Acked Read 7 Done Sent Done false
35357 10117 Acked Write 6 Done Sent Done false
35358 10118 Acked Read 2 Done Sent Done false
35359 10119 Acked Write 3 Done Sent Done false
35360 10120 Acked Write 7 Done Sent Done false
35361 10121 Acked Read 6 Done Sent Done false
35362 10122 Acked Read 1 Done Sent Done false
35363 10123 Acked Write 7 Done Sent Done false
35364 10124 Acked Read 2 Done Sent Done false
35365 10125 Acked Flush 0 Done Sent Done false
35366 10126 Acked Write 10 Done Sent Done false
35367 10127 Acked Write 2 Done Sent Done false
35368 10128 Acked Read 6 Done Sent Done false
35369 10129 Acked Write 7 Done Sent Done false
35370 10130 Acked Read 4 Done Sent Done false
35371 10131 Acked Read 8 Done Sent Done false
35372 10132 Acked Write 9 Done Sent Done false
35373 10133 Acked Read 7 Done Sent Done false
35374 10134 Acked Write 7 Done Sent Done false
35375 10135 Acked Read 9 Done Sent Done false
35376 10136 Acked Write 1 Done Sent Done false
35377 10137 Acked Write 3 Done Sent Done false
35378 10138 Acked Flush 0 Done Sent Done false
35379 10139 Acked Write 10 Done Sent Done false
35380 10140 Acked Write 4 Done Sent Done false
35381 10141 Acked Read 1 Done Sent Done false
35382 10142 Acked Flush 0 Done Sent Done false
35383 10143 Acked Write 5 Done Sent Done false
35384 10144 Acked Read 9 Done Sent Done false
35385 10145 Acked Flush 0 Done Sent Done false
35386 10146 Acked Read 7 Done Sent Done false
35387 10147 Acked Read 7 Done Sent Done false
35388 10148 Acked Read 7 Done Sent Done false
35389 10149 Acked Read 5 Done Sent Done false
35390 10150 Acked Write 1 Done Sent Done false
35391 10151 Acked Write 1 Done Sent Done false
35392 10152 Acked Read 3 Done Sent Done false
35393 10153 Acked Write 8 Done Sent Done false
35394 10154 Acked Write 1 Done Sent Done false
35395 10155 Acked Read 8 Done Sent Done false
35396 10156 Acked Read 6 Done Sent Done false
35397 10157 Acked Write 3 Done Sent Done false
35398 10158 Acked Read 1 Done Sent Done false
35399 10159 Acked Write 2 Done Sent Done false
35400 10160 Acked Write 3 Done Sent Done false
35401 10161 Acked Write 8 Done Sent Done false
35402 10162 Acked Read 5 Done Sent Done false
35403 10163 Acked Read 2 Done Sent Done false
35404 10164 Acked Read 1 Done Sent Done false
35405 10165 Acked Write 5 Done Sent Done false
35406 10166 Acked Write 3 Done Sent Done false
35407 10167 Acked Write 7 Done Sent Done false
35408 10168 Acked Read 10 Done Sent Done false
35409 10169 Acked Read 2 Done Sent Done false
35410 10170 Acked Read 7 Done Sent Done false
35411 10171 Acked Write 4 Done Sent Done false
35412 10172 Acked Read 5 Done Sent Done false
35413 10173 Acked Write 7 Done Sent Done false
35414 10174 Acked Write 8 Done Sent Done false
35415 10175 Acked Flush 0 Done Sent Done false
35416 10176 Acked Write 1 Done Sent Done false
35417 10177 Acked Read 5 Done Sent Done false
35418 10178 Acked Read 1 Done Sent Done false
35419 10179 Acked Read 2 Done Sent Done false
35420 10180 Acked Read 5 Done Sent Done false
35421 10181 Acked Write 10 Done Sent Done false
35422 10182 Acked Read 7 Done Sent Done false
35423 10183 Acked Read 8 Done Sent Done false
35424 10184 Acked Flush 0 Done Sent Done false
35425 10185 Acked Write 1 Done Sent Done false
35426 10186 Acked Read 9 Done Sent Done false
35427 10187 Acked Read 4 Done Sent Done false
35428 10188 Acked Write 1 Done Sent Done false
35429 10189 Acked Write 5 Done Sent Done false
35430 10190 Acked Read 4 Done Sent Done false
35431 10191 Acked Write 2 Done Sent Done false
35432 10192 Acked Read 10 Done Sent Done false
35433 10193 Acked Read 4 Done Sent Done false
35434 10194 Acked Read 3 Done Sent Done false
35435 10195 Acked Read 6 Done Sent Done false
35436 10196 Acked Read 1 Done Sent Done false
35437 10197 Acked Read 5 Done Sent Done false
35438 10198 Acked Write 2 Done Sent Done false
35439 10199 Acked Write 4 Done Sent Done false
35440 10200 Acked Read 7 Done Sent Done false
35441 10201 Acked Write 7 Done Sent Done false
35442 10202 Acked Write 5 Done Sent Done false
35443 10203 Acked Write 3 Done Sent Done false
35444 10204 Acked Read 10 Done Sent Done false
35445 10205 Acked Read 10 Done Sent Done false
35446 10206 Acked Read 5 Done Sent Done false
35447 10207 Acked Write 7 Done Sent Done false
35448 10208 Acked Read 7 Done Sent Done false
35449 10209 Acked Write 1 Done Sent Done false
35450 10210 Acked Read 5 Done Sent Done false
35451 10211 Acked Write 7 Done Sent Done false
35452 10212 Acked Write 2 Done Sent Done false
35453 10213 Acked Write 6 Done Sent Done false
35454 10214 Acked Read 10 Done Sent Done false
35455 10215 Acked Read 8 Done Sent Done false
35456 10216 Acked Read 3 Done Sent Done false
35457 10217 Acked Write 3 Done Sent Done false
35458 10218 Acked Read 8 Done Sent Done false
35459 10219 Acked Read 8 Done Sent Done false
35460 10220 Acked Read 6 Done Sent Done false
35461 10221 Acked Write 2 Done Sent Done false
35462 10222 Acked Write 1 Done Sent Done false
35463 10223 Acked Read 10 Done Sent Done false
35464 10224 Acked Write 5 Done Sent Done false
35465 10225 Acked Read 8 Done Sent Done false
35466 10226 Acked Flush 0 Done Sent Done false
35467 10227 Acked Read 9 Done Sent Done false
35468 10228 Acked Write 2 Done Sent Done false
35469 10229 Acked Write 6 Done Sent Done false
35470 10230 Acked Read 7 Done Sent Done false
35471 10231 Acked Read 1 Done Sent Done false
35472 10232 Acked Write 6 Done Sent Done false
35473 10233 Acked Read 6 Done Sent Done false
35474 10234 Acked Write 1 Done Sent Done false
35475 10235 Acked Read 1 Done Sent Done false
35476 10236 Acked Read 7 Done Sent Done false
35477 10237 Acked Write 2 Done Sent Done false
35478 10238 Acked Write 3 Done Sent Done false
35479 10239 Acked Read 1 Done Sent Done false
35480 10240 Acked Write 8 Done Sent Done false
35481 10241 Acked Read 7 Done Sent Done false
35482 10242 Acked Read 8 Done Sent Done false
35483 10243 Acked Read 1 Done Sent Done false
35484 10244 Acked Read 7 Done Sent Done false
35485 10245 Acked Write 10 Done Sent Done false
35486 10246 Acked Read 6 Done Sent Done false
35487 10247 Acked Read 3 Done Sent Done false
35488 10248 Acked Flush 0 Done Sent Done false
35489 10249 Acked Write 5 Done Sent Done false
35490 10250 Acked Write 1 Done Sent Done false
35491 10251 Acked Read 2 Done Sent Done false
35492 10252 Acked Flush 0 Done Sent Done false
35493 10253 Acked Flush 0 Done Sent Done false
35494 10254 Acked Read 10 Done Sent Done false
35495 10255 Acked Write 7 Done Sent Done false
35496 10256 Acked Write 6 Done Sent Done false
35497 10257 Acked Write 7 Done Sent Done false
35498 10258 Acked Flush 0 Done Sent Done false
35499 10259 Acked Write 10 Done Sent Done false
35500 10260 Acked Read 10 Done Sent Done false
35501 10261 Acked Read 1 Done Sent Done false
35502 10262 Acked Flush 0 Done Sent Done false
35503 10263 Acked Write 1 Done Sent Done false
35504 10264 Acked Read 2 Done Sent Done false
35505 10265 Acked Write 5 Done Sent Done false
35506 10266 Acked Write 2 Done Sent Done false
35507 10267 Acked Read 9 Done Sent Done false
35508 10268 Acked Read 2 Done Sent Done false
35509 10269 Acked Write 3 Done Sent Done false
35510 10270 Acked Read 8 Done Sent Done false
35511 10271 Acked Read 5 Done Sent Done false
35512 10272 Acked Write 6 Done Sent Done false
35513 10273 Acked Read 7 Done Sent Done false
35514 10274 Acked Write 2 Done Sent Done false
35515 10275 Acked Write 7 Done Sent Done false
35516 10276 Acked Write 6 Done Sent Done false
35517 10277 Acked Read 9 Done Sent Done false
35518 10278 Acked Write 8 Done Sent Done false
35519 10279 Acked Read 8 Done Sent Done false
35520 10280 Acked Write 8 Done Sent Done false
35521 10281 Acked Read 3 Done Sent Done false
35522 10282 Acked Read 6 Done Sent Done false
35523 10283 Acked Write 8 Done Sent Done false
35524 10284 Acked Write 7 Done Sent Done false
35525 10285 Acked Write 6 Done Sent Done false
35526 10286 Acked Read 8 Done Sent Done false
35527 10287 Acked Write 3 Done Sent Done false
35528 10288 Acked Write 2 Done Sent Done false
35529 10289 Acked Read 3 Done Sent Done false
35530 10290 Acked Read 2 Done Sent Done false
35531 10291 Acked Write 5 Done Sent Done false
35532 10292 Acked Read 10 Done Sent Done false
35533 10293 Acked Write 1 Done Sent Done false
35534 10294 Acked Read 2 Done Sent Done false
35535 10295 Acked Write 9 Done Sent Done false
35536 10296 Acked Read 7 Done Sent Done false
35537 10297 Acked Write 6 Done Sent Done false
35538 10298 Acked Write 2 Done Sent Done false
35539 10299 Acked Read 3 Done Sent Done false
35540 10300 Acked Flush 0 Done Sent Done false
35541 10301 Acked Flush 0 Done Sent Done false
35542 10302 Acked Read 7 Done Sent Done false
35543 10303 Acked Read 4 Done Sent Done false
35544 10304 Acked Write 5 Done Sent Done false
35545 10305 Acked Write 7 Done Sent Done false
35546 10306 Acked Write 9 Done Sent Done false
35547 10307 Acked Read 10 Done Sent Done false
35548 10308 Acked Write 5 Done Sent Done false
35549 10309 Acked Read 4 Done Sent Done false
35550 10310 Acked Flush 0 Done Sent Done false
35551 10311 Acked Read 10 Done Sent Done false
35552 10312 Acked Write 2 Done Sent Done false
35553 10313 Acked Write 5 Done Sent Done false
35554 10314 Acked Write 3 Done Sent Done false
35555 10315 Acked Read 8 Done Sent Done false
35556 10316 Acked Read 1 Done Sent Done false
35557 10317 Acked Read 7 Done Sent Done false
35558 10318 Acked Read 6 Done Sent Done false
35559 10319 Acked Read 2 Done Sent Done false
35560 10320 Acked Write 2 Done Sent Done false
35561 10321 Acked Write 7 Done Sent Done false
35562 10322 Acked Read 6 Done Sent Done false
35563 10323 Acked Write 8 Done Sent Done false
35564 10324 Acked Read 8 Done Sent Done false
35565 10325 Acked Write 4 Done Sent Done false
35566 10326 Acked Flush 0 Done Sent Done false
35567 10327 Acked Write 3 Done Sent Done false
35568 10328 Acked Flush 0 Done Sent Done false
35569 10329 Acked Write 1 Done Sent Done false
35570 10330 Acked Read 2 Done Sent Done false
35571 10331 Acked Read 8 Done Sent Done false
35572 10332 Acked Flush 0 Done Sent Done false
35573 10333 Acked Write 8 Done Sent Done false
35574 10334 Acked Read 3 Done Sent Done false
35575 10335 Acked Flush 0 Done Sent Done false
35576 10336 Acked Read 5 Done Sent Done false
35577 10337 Acked Flush 0 Done Sent Done false
35578 10338 Acked Flush 0 Done Sent Done false
35579 10339 Acked Write 2 Done Sent Done false
35580 10340 Acked Write 2 Done Sent Done false
35581 10341 Acked Read 4 Done Sent Done false
35582 10342 Acked Write 8 Done Sent Done false
35583 10343 Acked Read 3 Done Sent Done false
35584 10344 Acked Read 2 Done Sent Done false
35585 10345 Acked Write 9 Done Sent Done false
35586 10346 Acked Write 7 Done Sent Done false
35587 10347 Acked Read 3 Done Sent Done false
35588 10348 Acked Write 6 Done Sent Done false
35589 10349 Acked Read 4 Done Sent Done false
35590 10350 Acked Write 2 Done Sent Done false
35591 10351 Acked Write 1 Done Sent Done false
35592 10352 Acked Write 7 Done Sent Done false
35593 10353 Acked Read 6 Done Sent Done false
35594 10354 Acked Read 7 Done Sent Done false
35595 STATES DS:0 DS:1 DS:2 TOTAL
35596 Sent 0 301 0 301
35597 Done 301 0 301 602
35598 Skipped 0 0 0 0
35599 Error 0 0 0 0
35600 Last Flush: Some(JobId(10338)) Some(JobId(10053)) Some(JobId(10338))
35601 Downstairs last five completed: 10053 10052 10051 10050 10049
35602 Upstairs last five completed: 10354 10353 10352 10351 10350
35603 CLIENT: Up:0 ds:301 act:2
356042025-02-25T20:54:17.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
35605 [8820] got start action
35606 [8820] I am going to restart
35607 Starting downstairs at port 8820
35608 Make output file at "/tmp/dsc/downstairs-8820.txt"
35609 [8820][1] reports Starting
35610 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1237)
35611 [8820][1] reports Running
356122025-02-25T20:54:18.797ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
356132025-02-25T20:54:19.799ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
35614 ----------------------------------------------------------------
35615 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35616 States: Active Offline Active
35617 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35618 10054 Acked Read 10 Done Sent Done false
35619 10055 Acked Write 1 Done Sent Done false
35620 10056 Acked Read 6 Done Sent Done false
35621 10057 Acked Read 5 Done Sent Done false
35622 10058 Acked Write 10 Done Sent Done false
35623 10059 Acked Write 9 Done Sent Done false
35624 10060 Acked Write 4 Done Sent Done false
35625 10061 Acked Write 3 Done Sent Done false
35626 10062 Acked Write 4 Done Sent Done false
35627 10063 Acked Write 9 Done Sent Done false
35628 10064 Acked Write 8 Done Sent Done false
35629 10065 Acked Write 3 Done Sent Done false
35630 10066 Acked Read 1 Done Sent Done false
35631 10067 Acked Read 8 Done Sent Done false
35632 10068 Acked Flush 0 Done Sent Done false
35633 10069 Acked Write 8 Done Sent Done false
35634 10070 Acked Write 4 Done Sent Done false
35635 10071 Acked Read 10 Done Sent Done false
35636 10072 Acked Write 8 Done Sent Done false
35637 10073 Acked Read 7 Done Sent Done false
35638 10074 Acked Flush 0 Done Sent Done false
35639 10075 Acked Read 8 Done Sent Done false
35640 10076 Acked Read 8 Done Sent Done false
35641 10077 Acked Write 6 Done Sent Done false
35642 10078 Acked Write 5 Done Sent Done false
35643 10079 Acked Read 8 Done Sent Done false
35644 10080 Acked Read 7 Done Sent Done false
35645 10081 Acked Read 8 Done Sent Done false
35646 10082 Acked Read 4 Done Sent Done false
35647 10083 Acked Write 4 Done Sent Done false
35648 10084 Acked Read 1 Done Sent Done false
35649 10085 Acked Write 3 Done Sent Done false
35650 10086 Acked Read 9 Done Sent Done false
35651 10087 Acked Write 5 Done Sent Done false
35652 10088 Acked Read 3 Done Sent Done false
35653 10089 Acked Flush 0 Done Sent Done false
35654 10090 Acked Read 6 Done Sent Done false
35655 10091 Acked Write 3 Done Sent Done false
35656 10092 Acked Write 6 Done Sent Done false
35657 10093 Acked Read 2 Done Sent Done false
35658 10094 Acked Write 7 Done Sent Done false
35659 10095 Acked Write 3 Done Sent Done false
35660 10096 Acked Read 9 Done Sent Done false
35661 10097 Acked Write 3 Done Sent Done false
35662 10098 Acked Read 7 Done Sent Done false
35663 10099 Acked Read 3 Done Sent Done false
35664 10100 Acked Write 7 Done Sent Done false
35665 10101 Acked Read 7 Done Sent Done false
35666 10102 Acked Flush 0 Done Sent Done false
35667 10103 Acked Read 4 Done Sent Done false
35668 10104 Acked Write 2 Done Sent Done false
35669 10105 Acked Read 1 Done Sent Done false
35670 10106 Acked Flush 0 Done Sent Done false
35671 10107 Acked Read 6 Done Sent Done false
35672 10108 Acked Write 4 Done Sent Done false
35673 10109 Acked Write 6 Done Sent Done false
35674 10110 Acked Write 4 Done Sent Done false
35675 10111 Acked Write 3 Done Sent Done false
35676 10112 Acked Write 3 Done Sent Done false
35677 10113 Acked Write 8 Done Sent Done false
35678 10114 Acked Read 1 Done Sent Done false
35679 10115 Acked Read 6 Done Sent Done false
35680 10116 Acked Read 7 Done Sent Done false
35681 10117 Acked Write 6 Done Sent Done false
35682 10118 Acked Read 2 Done Sent Done false
35683 10119 Acked Write 3 Done Sent Done false
35684 10120 Acked Write 7 Done Sent Done false
35685 10121 Acked Read 6 Done Sent Done false
35686 10122 Acked Read 1 Done Sent Done false
35687 10123 Acked Write 7 Done Sent Done false
35688 10124 Acked Read 2 Done Sent Done false
35689 10125 Acked Flush 0 Done Sent Done false
35690 10126 Acked Write 10 Done Sent Done false
35691 10127 Acked Write 2 Done Sent Done false
35692 10128 Acked Read 6 Done Sent Done false
35693 10129 Acked Write 7 Done Sent Done false
35694 10130 Acked Read 4 Done Sent Done false
35695 10131 Acked Read 8 Done Sent Done false
35696 10132 Acked Write 9 Done Sent Done false
35697 10133 Acked Read 7 Done Sent Done false
35698 10134 Acked Write 7 Done Sent Done false
35699 10135 Acked Read 9 Done Sent Done false
35700 10136 Acked Write 1 Done Sent Done false
35701 10137 Acked Write 3 Done Sent Done false
35702 10138 Acked Flush 0 Done Sent Done false
35703 10139 Acked Write 10 Done Sent Done false
35704 10140 Acked Write 4 Done Sent Done false
35705 10141 Acked Read 1 Done Sent Done false
35706 10142 Acked Flush 0 Done Sent Done false
35707 10143 Acked Write 5 Done Sent Done false
35708 10144 Acked Read 9 Done Sent Done false
35709 10145 Acked Flush 0 Done Sent Done false
35710 10146 Acked Read 7 Done Sent Done false
35711 10147 Acked Read 7 Done Sent Done false
35712 10148 Acked Read 7 Done Sent Done false
35713 10149 Acked Read 5 Done Sent Done false
35714 10150 Acked Write 1 Done Sent Done false
35715 10151 Acked Write 1 Done Sent Done false
35716 10152 Acked Read 3 Done Sent Done false
35717 10153 Acked Write 8 Done Sent Done false
35718 10154 Acked Write 1 Done Sent Done false
35719 10155 Acked Read 8 Done Sent Done false
35720 10156 Acked Read 6 Done Sent Done false
35721 10157 Acked Write 3 Done Sent Done false
35722 10158 Acked Read 1 Done Sent Done false
35723 10159 Acked Write 2 Done Sent Done false
35724 10160 Acked Write 3 Done Sent Done false
35725 10161 Acked Write 8 Done Sent Done false
35726 10162 Acked Read 5 Done Sent Done false
35727 10163 Acked Read 2 Done Sent Done false
35728 10164 Acked Read 1 Done Sent Done false
35729 10165 Acked Write 5 Done Sent Done false
35730 10166 Acked Write 3 Done Sent Done false
35731 10167 Acked Write 7 Done Sent Done false
35732 10168 Acked Read 10 Done Sent Done false
35733 10169 Acked Read 2 Done Sent Done false
35734 10170 Acked Read 7 Done Sent Done false
35735 10171 Acked Write 4 Done Sent Done false
35736 10172 Acked Read 5 Done Sent Done false
35737 10173 Acked Write 7 Done Sent Done false
35738 10174 Acked Write 8 Done Sent Done false
35739 10175 Acked Flush 0 Done Sent Done false
35740 10176 Acked Write 1 Done Sent Done false
35741 10177 Acked Read 5 Done Sent Done false
35742 10178 Acked Read 1 Done Sent Done false
35743 10179 Acked Read 2 Done Sent Done false
35744 10180 Acked Read 5 Done Sent Done false
35745 10181 Acked Write 10 Done Sent Done false
35746 10182 Acked Read 7 Done Sent Done false
35747 10183 Acked Read 8 Done Sent Done false
35748 10184 Acked Flush 0 Done Sent Done false
35749 10185 Acked Write 1 Done Sent Done false
35750 10186 Acked Read 9 Done Sent Done false
35751 10187 Acked Read 4 Done Sent Done false
35752 10188 Acked Write 1 Done Sent Done false
35753 10189 Acked Write 5 Done Sent Done false
35754 10190 Acked Read 4 Done Sent Done false
35755 10191 Acked Write 2 Done Sent Done false
35756 10192 Acked Read 10 Done Sent Done false
35757 10193 Acked Read 4 Done Sent Done false
35758 10194 Acked Read 3 Done Sent Done false
35759 10195 Acked Read 6 Done Sent Done false
35760 10196 Acked Read 1 Done Sent Done false
35761 10197 Acked Read 5 Done Sent Done false
35762 10198 Acked Write 2 Done Sent Done false
35763 10199 Acked Write 4 Done Sent Done false
35764 10200 Acked Read 7 Done Sent Done false
35765 10201 Acked Write 7 Done Sent Done false
35766 10202 Acked Write 5 Done Sent Done false
35767 10203 Acked Write 3 Done Sent Done false
35768 10204 Acked Read 10 Done Sent Done false
35769 10205 Acked Read 10 Done Sent Done false
35770 10206 Acked Read 5 Done Sent Done false
35771 10207 Acked Write 7 Done Sent Done false
35772 10208 Acked Read 7 Done Sent Done false
35773 10209 Acked Write 1 Done Sent Done false
35774 10210 Acked Read 5 Done Sent Done false
35775 10211 Acked Write 7 Done Sent Done false
35776 10212 Acked Write 2 Done Sent Done false
35777 10213 Acked Write 6 Done Sent Done false
35778 10214 Acked Read 10 Done Sent Done false
35779 10215 Acked Read 8 Done Sent Done false
35780 10216 Acked Read 3 Done Sent Done false
35781 10217 Acked Write 3 Done Sent Done false
35782 10218 Acked Read 8 Done Sent Done false
35783 10219 Acked Read 8 Done Sent Done false
35784 10220 Acked Read 6 Done Sent Done false
35785 10221 Acked Write 2 Done Sent Done false
35786 10222 Acked Write 1 Done Sent Done false
35787 10223 Acked Read 10 Done Sent Done false
35788 10224 Acked Write 5 Done Sent Done false
35789 10225 Acked Read 8 Done Sent Done false
35790 10226 Acked Flush 0 Done Sent Done false
35791 10227 Acked Read 9 Done Sent Done false
35792 10228 Acked Write 2 Done Sent Done false
35793 10229 Acked Write 6 Done Sent Done false
35794 10230 Acked Read 7 Done Sent Done false
35795 10231 Acked Read 1 Done Sent Done false
35796 10232 Acked Write 6 Done Sent Done false
35797 10233 Acked Read 6 Done Sent Done false
35798 10234 Acked Write 1 Done Sent Done false
35799 10235 Acked Read 1 Done Sent Done false
35800 10236 Acked Read 7 Done Sent Done false
35801 10237 Acked Write 2 Done Sent Done false
35802 10238 Acked Write 3 Done Sent Done false
35803 10239 Acked Read 1 Done Sent Done false
35804 10240 Acked Write 8 Done Sent Done false
35805 10241 Acked Read 7 Done Sent Done false
35806 10242 Acked Read 8 Done Sent Done false
35807 10243 Acked Read 1 Done Sent Done false
35808 10244 Acked Read 7 Done Sent Done false
35809 10245 Acked Write 10 Done Sent Done false
35810 10246 Acked Read 6 Done Sent Done false
35811 10247 Acked Read 3 Done Sent Done false
35812 10248 Acked Flush 0 Done Sent Done false
35813 10249 Acked Write 5 Done Sent Done false
35814 10250 Acked Write 1 Done Sent Done false
35815 10251 Acked Read 2 Done Sent Done false
35816 10252 Acked Flush 0 Done Sent Done false
35817 10253 Acked Flush 0 Done Sent Done false
35818 10254 Acked Read 10 Done Sent Done false
35819 10255 Acked Write 7 Done Sent Done false
35820 10256 Acked Write 6 Done Sent Done false
35821 10257 Acked Write 7 Done Sent Done false
35822 10258 Acked Flush 0 Done Sent Done false
35823 10259 Acked Write 10 Done Sent Done false
35824 10260 Acked Read 10 Done Sent Done false
35825 10261 Acked Read 1 Done Sent Done false
35826 10262 Acked Flush 0 Done Sent Done false
35827 10263 Acked Write 1 Done Sent Done false
35828 10264 Acked Read 2 Done Sent Done false
35829 10265 Acked Write 5 Done Sent Done false
35830 10266 Acked Write 2 Done Sent Done false
35831 10267 Acked Read 9 Done Sent Done false
35832 10268 Acked Read 2 Done Sent Done false
35833 10269 Acked Write 3 Done Sent Done false
35834 10270 Acked Read 8 Done Sent Done false
35835 10271 Acked Read 5 Done Sent Done false
35836 10272 Acked Write 6 Done Sent Done false
35837 10273 Acked Read 7 Done Sent Done false
35838 10274 Acked Write 2 Done Sent Done false
35839 10275 Acked Write 7 Done Sent Done false
35840 10276 Acked Write 6 Done Sent Done false
35841 10277 Acked Read 9 Done Sent Done false
35842 10278 Acked Write 8 Done Sent Done false
35843 10279 Acked Read 8 Done Sent Done false
35844 10280 Acked Write 8 Done Sent Done false
35845 10281 Acked Read 3 Done Sent Done false
35846 10282 Acked Read 6 Done Sent Done false
35847 10283 Acked Write 8 Done Sent Done false
35848 10284 Acked Write 7 Done Sent Done false
35849 10285 Acked Write 6 Done Sent Done false
35850 10286 Acked Read 8 Done Sent Done false
35851 10287 Acked Write 3 Done Sent Done false
35852 10288 Acked Write 2 Done Sent Done false
35853 10289 Acked Read 3 Done Sent Done false
35854 10290 Acked Read 2 Done Sent Done false
35855 10291 Acked Write 5 Done Sent Done false
35856 10292 Acked Read 10 Done Sent Done false
35857 10293 Acked Write 1 Done Sent Done false
35858 10294 Acked Read 2 Done Sent Done false
35859 10295 Acked Write 9 Done Sent Done false
35860 10296 Acked Read 7 Done Sent Done false
35861 10297 Acked Write 6 Done Sent Done false
35862 10298 Acked Write 2 Done Sent Done false
35863 10299 Acked Read 3 Done Sent Done false
35864 10300 Acked Flush 0 Done Sent Done false
35865 10301 Acked Flush 0 Done Sent Done false
35866 10302 Acked Read 7 Done Sent Done false
35867 10303 Acked Read 4 Done Sent Done false
35868 10304 Acked Write 5 Done Sent Done false
35869 10305 Acked Write 7 Done Sent Done false
35870 10306 Acked Write 9 Done Sent Done false
35871 10307 Acked Read 10 Done Sent Done false
35872 10308 Acked Write 5 Done Sent Done false
35873 10309 Acked Read 4 Done Sent Done false
35874 10310 Acked Flush 0 Done Sent Done false
35875 10311 Acked Read 10 Done Sent Done false
35876 10312 Acked Write 2 Done Sent Done false
35877 10313 Acked Write 5 Done Sent Done false
35878 10314 Acked Write 3 Done Sent Done false
35879 10315 Acked Read 8 Done Sent Done false
35880 10316 Acked Read 1 Done Sent Done false
35881 10317 Acked Read 7 Done Sent Done false
35882 10318 Acked Read 6 Done Sent Done false
35883 10319 Acked Read 2 Done Sent Done false
35884 10320 Acked Write 2 Done Sent Done false
35885 10321 Acked Write 7 Done Sent Done false
35886 10322 Acked Read 6 Done Sent Done false
35887 10323 Acked Write 8 Done Sent Done false
35888 10324 Acked Read 8 Done Sent Done false
35889 10325 Acked Write 4 Done Sent Done false
35890 10326 Acked Flush 0 Done Sent Done false
35891 10327 Acked Write 3 Done Sent Done false
35892 10328 Acked Flush 0 Done Sent Done false
35893 10329 Acked Write 1 Done Sent Done false
35894 10330 Acked Read 2 Done Sent Done false
35895 10331 Acked Read 8 Done Sent Done false
35896 10332 Acked Flush 0 Done Sent Done false
35897 10333 Acked Write 8 Done Sent Done false
35898 10334 Acked Read 3 Done Sent Done false
35899 10335 Acked Flush 0 Done Sent Done false
35900 10336 Acked Read 5 Done Sent Done false
35901 10337 Acked Flush 0 Done Sent Done false
35902 10338 Acked Flush 0 Done Sent Done false
35903 10339 Acked Write 2 Done Sent Done false
35904 10340 Acked Write 2 Done Sent Done false
35905 10341 Acked Read 4 Done Sent Done false
35906 10342 Acked Write 8 Done Sent Done false
35907 10343 Acked Read 3 Done Sent Done false
35908 10344 Acked Read 2 Done Sent Done false
35909 10345 Acked Write 9 Done Sent Done false
35910 10346 Acked Write 7 Done Sent Done false
35911 10347 Acked Read 3 Done Sent Done false
35912 10348 Acked Write 6 Done Sent Done false
35913 10349 Acked Read 4 Done Sent Done false
35914 10350 Acked Write 2 Done Sent Done false
35915 10351 Acked Write 1 Done Sent Done false
35916 10352 Acked Write 7 Done Sent Done false
35917 10353 Acked Read 6 Done Sent Done false
35918 10354 Acked Read 7 Done Sent Done false
35919 STATES DS:0 DS:1 DS:2 TOTAL
35920 Sent 0 301 0 301
35921 Done 301 0 301 602
35922 Skipped 0 0 0 0
35923 Error 0 0 0 0
35924 Last Flush: Some(JobId(10338)) Some(JobId(10053)) Some(JobId(10338))
35925 Downstairs last five completed: 10053 10052 10051 10050 10049
35926 Upstairs last five completed: 10354 10353 10352 10351 10350
35927 CLIENT: Up:0 ds:301 act:2
359282025-02-25T20:54:21.107ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359292025-02-25T20:54:21.804ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359302025-02-25T20:54:22.366ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359312025-02-25T20:54:22.366ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62227) = io task client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359322025-02-25T20:54:22.367ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359332025-02-25T20:54:22.368ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359342025-02-25T20:54:22.368ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359352025-02-25T20:54:22.368ZINFOcrucible: send last flush ID to this DS: Some(JobId(10053)) = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359362025-02-25T20:54:22.368ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359372025-02-25T20:54:22.368ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10053)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
359382025-02-25T20:54:22.368ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10053)) = downstairs session_id = 0093020f-8db1-4b17-8f9d-499d3d7b91d5
35939 ----------------------------------------------------------------
35940 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35941 Downstairs last five completed: 10355 10354 10353 10352 10351
35942 Upstairs last five completed: 10355 10354 10353 10352 10351
35943 CLIENT: Up:0 ds:0 act:3
35944 Replay: All jobs finished
35945 Test replay has completed
35946 Wrote out file "/tmp/test_replay_verify.log"
35947 CLIENT: Tests done. All submitted work has been ACK'd
35948 ----------------------------------------------------------------
35949 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35950 Downstairs last five completed: 10355 10354 10353 10352 10351
35951 Upstairs last five completed: 10355 10354 10353 10352 10351
35952 CLIENT: Up:0 ds:0 act:3
35953 CLIENT: All jobs finished, all DS active.
359542025-02-25T20:54:24.611ZWARNcrucible: Guest handle has been dropped
35955 Replays:30 time: 7:40
35956 Do final verify
35957 Feb 25 20:54:24.682 INFO accepted connection, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
35958 Feb 25 20:54:24.683 INFO request completed, latency_us: 226, response_code: 200, uri: /regioninfo, method: GET, req_id: 2b748295-674d-4570-9882-997cb1ae7f6b, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
359592025-02-25T20:54:24.684ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
35960 Feb 25 20:54:24.684 INFO request completed, latency_us: 108, response_code: 200, uri: /regioncount, method: GET, req_id: 25133c98-2296-4c67-9be3-be49c846daa1, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
359612025-02-25T20:54:24.685ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
35962 Feb 25 20:54:24.685 INFO request completed, latency_us: 109, response_code: 200, uri: /port/cid/0, method: GET, req_id: 26d150ce-ddca-4d5a-9a5f-59e34e7235af, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
35963 Feb 25 20:54:24.685 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: f83f814d-f24d-467a-8b57-1888ef89c5e9, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
35964 Feb 25 20:54:24.686 INFO request completed, latency_us: 95, response_code: 200, uri: /port/cid/2, method: GET, req_id: 71e01b60-0660-4e69-ba68-6b0f0389ef09, remote_addr: 127.0.0.1:53175, local_addr: 127.0.0.1:9998
359652025-02-25T20:54:24.686ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
359662025-02-25T20:54:24.686ZINFOcrucible: Upstairs starts
359672025-02-25T20:54:24.686ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "cdb6c7ae221cade941351dba5e763db1f87ef907", git_commit_timestamp: "2025-02-25T18:15:07.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
359682025-02-25T20:54:24.686ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
359692025-02-25T20:54:24.686ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
359702025-02-25T20:54:24.687ZINFOcrucible: Crucible 21f23c19-909e-415f-8182-4aa08264a1e0 has session id: 544cbdcf-dc49-4084-9948-0c2f514f055e session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359712025-02-25T20:54:24.687ZINFOcrucible: Upstairs opts: Upstairs UUID: 21f23c19-909e-415f-8182-4aa08264a1e0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359722025-02-25T20:54:24.687ZINFOcrucible: Crucible stats registered with UUID: 21f23c19-909e-415f-8182-4aa08264a1e0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359732025-02-25T20:54:24.687ZWARNcrucible: could not find Downstairs address for Nexus session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359742025-02-25T20:54:24.688ZINFOcrucible: Downstairs in volume = 3 task = crutest
359752025-02-25T20:54:24.688ZINFOcrucible: The guest has requested activation with gen:3
359762025-02-25T20:54:24.688ZINFOcrucible: 21f23c19-909e-415f-8182-4aa08264a1e0 active request set session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359772025-02-25T20:54:24.688ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359782025-02-25T20:54:24.688ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359792025-02-25T20:54:24.688ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359802025-02-25T20:54:24.688ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359812025-02-25T20:54:24.688ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359822025-02-25T20:54:24.688ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359832025-02-25T20:54:24.688ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359842025-02-25T20:54:24.688ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359852025-02-25T20:54:24.688ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359862025-02-25T20:54:24.688ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37913) = io task client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359872025-02-25T20:54:24.688ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51356) = io task client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359882025-02-25T20:54:24.688ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44095) = io task client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359892025-02-25T20:54:24.689ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359902025-02-25T20:54:24.689ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359912025-02-25T20:54:24.689ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359922025-02-25T20:54:24.690ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359932025-02-25T20:54:24.690ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359942025-02-25T20:54:24.690ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359952025-02-25T20:54:24.691ZINFOcrucible: Waiting for more clients to be ready session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359962025-02-25T20:54:24.691ZINFOcrucible: Waiting for more clients to be ready session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359972025-02-25T20:54:24.691ZINFOcrucible: [0]R flush_numbers[0..12]: [929, 920, 920, 922, 923, 923, 926, 914, 920, 914, 920, 915] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359982025-02-25T20:54:24.691ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
359992025-02-25T20:54:24.691ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360002025-02-25T20:54:24.691ZINFOcrucible: [1]R flush_numbers[0..12]: [929, 920, 920, 922, 923, 923, 926, 914, 920, 914, 920, 915] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360012025-02-25T20:54:24.691ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360022025-02-25T20:54:24.691ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360032025-02-25T20:54:24.691ZINFOcrucible: [2]R flush_numbers[0..12]: [929, 920, 920, 922, 923, 923, 926, 914, 920, 914, 920, 915] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360042025-02-25T20:54:24.691ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360052025-02-25T20:54:24.691ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360062025-02-25T20:54:24.691ZINFOcrucible: Max found gen is 3 = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360072025-02-25T20:54:24.691ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360082025-02-25T20:54:24.691ZINFOcrucible: Next flush: 930 = downstairs session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
36009 {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
36010 ,"time":"2025-02-25T20:54:24.691357674Z","hostname":"w-01JMZDPGE8Y12NMX8R0HG7VCSW","pid":1242,"":"downstairs","":"downstairs","session_id":"544cbdcf-dc49-4084-9948-0c2f514f055e"}
360112025-02-25T20:54:24.691ZINFOcrucible: No downstairs reconciliation required session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360122025-02-25T20:54:24.691ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360132025-02-25T20:54:24.691ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360142025-02-25T20:54:24.691ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360152025-02-25T20:54:24.691ZINFOcrucible: All required reconciliation work is completed session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360162025-02-25T20:54:24.691ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360172025-02-25T20:54:24.691ZINFOcrucible: 21f23c19-909e-415f-8182-4aa08264a1e0 is now active with session: 544cbdcf-dc49-4084-9948-0c2f514f055e session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360182025-02-25T20:54:24.691ZINFOcrucible: Set Active after no reconciliation session_id = 544cbdcf-dc49-4084-9948-0c2f514f055e
360192025-02-25T20:54:24.691ZINFOcrucible: The guest has finished waiting for activation with:3
36020 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36021 Loading write count information from file "/tmp/test_replay_verify.log"
36022 Read and Verify all blocks (0..5000 range:false)
36023 Wrote out file "/tmp/test_replay_verify.log"
36024 Verify test completed
36025 Wrote out file "/tmp/test_replay_verify.log"
36026 CLIENT: Tests done. All submitted work has been ACK'd
36027 ----------------------------------------------------------------
36028 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36029 States: Active Active Active
36030 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36031 1000 Acked Read 256 Done Done Done false
36032 1001 Acked Read 256 Done Done Done false
36033 1002 Acked Read 256 Done Done Done false
36034 1003 Acked Read 256 Done Done Done false
36035 1004 Acked Read 256 Done Done Done false
36036 1005 Acked Read 256 Done Done Done false
36037 1006 Acked Read 256 Done Done Done false
36038 1007 Acked Read 256 Done Done Done false
36039 1008 Acked Read 256 Done Done Done false
36040 1009 Acked Read 256 Done Done Done false
36041 1010 Acked Read 256 Done Done Done false
36042 1011 Acked Read 256 Done Done Done false
36043 1012 Acked Read 256 Done Done Done false
36044 1013 Acked Read 256 Done Done Done false
36045 1014 Acked Read 256 Done Done Done false
36046 1015 Acked Read 256 Done Done Done false
36047 1016 Acked Read 256 Done Done Done false
36048 1017 Acked Read 136 Done Done Done false
36049 1018 Acked Read 256 Done Done Done false
36050 1019 Acked Read 256 Done Done Done false
36051 STATES DS:0 DS:1 DS:2 TOTAL
36052 Sent 0 0 0 0
36053 Done 20 20 20 60
36054 Skipped 0 0 0 0
36055 Error 0 0 0 0
36056 Last Flush: None None None
36057 Downstairs last five completed:
36058 Upstairs last five completed: 1019 1018 1017 1016 1015
36059 CLIENT: Up:0 ds:20 act:3
36060 ----------------------------------------------------------------
36061 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36062 Downstairs last five completed: 1020 1019 1018 1017 1016
36063 Upstairs last five completed: 1020 1019 1018 1017 1016
36064 CLIENT: Up:0 ds:0 act:3
36065 CLIENT: All crucible jobs finished, exiting program
360662025-02-25T20:54:29.091ZWARNcrucible: Guest handle has been dropped
36067 Feb 25 20:54:29.160 INFO accepted connection, remote_addr: 127.0.0.1:40891, local_addr: 127.0.0.1:9998
36068 Main task has work to do, go find it
36069 got dsc Shutdown
36070 Shutdown
36071 Shut it down
36072 Feb 25 20:54:29.161 INFO request completed, latency_us: 165, response_code: 204, uri: /shutdown, method: POST, req_id: 9cc0de3f-2acb-457c-968f-fadf2d062b6d, remote_addr: 127.0.0.1:40891, local_addr: 127.0.[8810] Disable keep_running
36073 0[8810] Got stop action so:false kr:false
36074 .1:9998
36075 [8820] Disable keep_running
36076 [8820] Got stop action so:false kr:false
36077 [8830] Disable keep_running
36078 [8830] Got stop action so:false kr:false
36079 [8810] Exited with: ExitStatus(unix_wait_status(9))
36080 [8810][0] reports Exit
36081 [8820] Exited with: ExitStatus(unix_wait_status(9))
36082 [8820][1] reports Exit
36083 [8830] Exited with: ExitStatus(unix_wait_status(9))
36084 [8830][2] reports Exit
36085 February 25, 2025 at 08:54:34 PM UTC Test ends with 0